Main

The phenomenon of entanglement is one of the fundamental results of quantum mechanics. Among its most extraordinary manifestations are quantum spin liquids, in which a macroscopic number of spins are entangled but do not show conventional magnetic order1. The earliest examples of QSLs were found in quasi-one-dimensional systems such as Heisenberg spin chains16 and in the distorted triangular-lattice antiferromagnet Cs2CuCl4 (ref. 17). The search for QSLs in two- and three-dimensional systems has focused on frustrated magnets, in which the lattice geometry prevents all magnetic interactions from being satisfied simultaneously1,2 or in which spin–orbit coupling leads to interactions that depend on bond directions18. In two dimensions, the prototype of a QSL is Anderson’s ‘resonating valence bond’ model19. Its ground state is a superposition of all possible tilings of dimers on the triangular lattice, where each dimer is built from an entangled pair of spin-1/2; its excitations are unpaired spin-1/2, which are delocalized. The presence of delocalized excitations with fractional quantum numbers leads to a necessary experimental signature of a QSL: a magnetic excitation spectrum that is continuous in energy but structured in momentum space7. Such a spectrum has indeed been observed using neutron-scattering measurements of ZnCu3(OD)6Cl2 (‘herbertsmithite’)7,20, in which spins occupy a kagome lattice. However, the stabilization of a continuous excitation spectrum by quantum fluctuations in a structurally perfect triangular-lattice magnet—the scenario originally proposed by Anderson19—has remained an open question.

Recent experiments have identified the triangular-lattice magnet YbMgGaO4 as an exciting QSL candidate3,4. The crystal structure (space group ) contains undistorted triangular planes of magnetic Yb3+ (Fig. 1a), separated by two triangular planes occupied in a disordered manner by Mg2+ and Ga3+ (Fig. 1b)3. Thermodynamic and muon-spin relaxation (μSR)5 measurements show the absence of conventional magnetic order and spin freezing to T < 0.1 K, far below the Weiss temperature θW ≈ −4 K; moreover, the apparent absence of zero-point entropy indicates that the system essentially occupies a single quantum state at 0.06 K (ref. 3). The crystal-field ground state of Yb3+ is a Kramers doublet4, separated by a large energy gap of 38 meV from the first excited state (Fig. 1c and Supplementary Fig. 1); hence, an effective spin-1/2 description is appropriate at low temperature, as for ‘quantum spin ice’ Yb2Ti2O7 (ref. 21). Electron-spin resonance (ESR) measurements indicate that the nearest-neighbour magnetic interaction is anisotropic and depends on the bond orientation4. This provides one potential mechanism to stabilize a QSL ground state on the triangular lattice13,14. However, several alternative mechanisms for QSL behaviour are known, including further-neighbour interactions9,10,11,12, disorder-induced entanglement15 and multi-spin interactions22.

Figure 1: Structure and magnetic properties of YbMgGaO4.
figure 1

a, Partial crystal structure, showing a triangular layer of Yb3+ ions (large cyan spheres) and their coordination by oxygen (small red spheres). A nearest-neighbour interaction pathway J1 and a next-nearest-neighbour interaction pathway J2 are shown by yellow and blue lines, respectively. b, Stacking of the Yb3+ planes along the c axis and their separation by two layers of disordered Mg2+ and Ga3+ ions. c, High-energy neutron-scattering measurements on a powder sample at T = 4 K. Filled (respectively, open) symbols correspond to an incident neutron energy of 160 meV (respectively, 300 meV). Solid lines are Gaussian fits to the observed magnetic excitations. Light (respectively, dark) horizontal bars correspond to the observed (respectively, calculated) peak’s full-width at half-maximum. d, Magnetic component of the specific heat Cm(T), showing data measured on a powder sample (open squares) and a single-crystal sample (filled circles). Single-crystal data are shown for applied magnetic fields along the c axis of 0, 4, 7.8 and 14 T (labelled above each curve). The orange line shows a fit of the zero-field single-crystal data to a power law, Cm(T) T0.70(1). e, Magnetic entropy change ΔSm, showing data measured in zero field and in a 7.8 T field (labelled on each curve). f, Dependence of the magnetization M on applied field μ0H, showing data measured at temperatures of 1.7 K for two different field directions (labelled on each curve). In df the temperatures and applied fields at which we performed neutron-scattering measurements (Figs 2 and 3) are indicated by grey arrowheads. Error bars correspond to 1 standard error.

Experiments to reveal the nature of the potential triangular-lattice QSL in YbMgGaO4 are therefore crucial. To enable such experiments, we have grown a large single crystal of YbMgGaO4 using the floating-zone technique (Supplementary Fig. 2). We characterized our crystal using specific heat, magnetization, and single-crystal neutron-diffraction measurements. In addition, we compared specific-heat and X-ray diffraction measurements of crushed crystal pieces with the same measurements of a powder sample prepared by the solid-state method of ref. 3. Single-crystal neutron-diffraction measurements (Supplementary Fig. 3) reveal that our single crystal is predominantly a single grain and is therefore suitable for inelastic neutron-scattering measurements. The magnetic specific heat shows a broad peak at T ≈ 3 K in zero magnetic field, and a Tγ dependence with γ = 0.70(1) below the peak (Fig. 1d), consistent with previous experimental results3. Our magnetic entropy (Fig. 1e) and magnetization (Fig. 1f) measurements fully support the effective spin-1/2 scenario (with g ≈ 3.7) for Yb3+ at low temperatures. Rietveld refinements to our X-ray diffraction data (Supplementary Fig. 4) yield a good fit for the published structural model3 with no Yb3+/Ga3+ site mixing observed within our experimental accuracy and with disordered occupancy on the interstitial layers by Mg2+ and Ga3+ (Supplementary Table 1). Such disorder can affect magnetism by modifying the charge environment around Yb3+ ions. Our broadband neutron-scattering experiment (Fig. 1c and Supplementary Fig. 1) reveals three intense crystal electric-field excitations at 38(1), 61(1) and 97(1) meV, consistent with the four Kramers doublets expected for Yb3+. These excitations are broader than the instrumental resolution and an additional weak mode is observed around 134(8) meV, hinting at a distribution of local environments across Yb3+ ions. In all our measurements, we observe no significant differences between powder and single-crystal samples, suggesting that the sample dependence that has afflicted QSL candidates such as Yb2Ti2O7 (ref. 23) is not evident in YbMgGaO4.

Neutron-scattering experiments measure spin correlations as a function of energy and reciprocal space, yielding direct information about correlated quantum states. Single-crystal neutron-scattering data measured in zero field are shown in Fig. 2, and provide strong evidence for QSL behaviour. Throughout, we plot neutron-scattering intensity as (1 + eβE) I(Q, E), where I(Q, E) is the measured intensity as a function of scattering wavevector Q = h a + k b + l c and energy transfer E, and the factor 1 + eβE corrects for detailed balance24. Figure 2a shows the energy dependence of the scattering intensity along high-symmetry reciprocal-space directions. The scattering at E 0.2 meV is magnetic, as shown by its dependence on applied magnetic field (discussed below). At both 0.06 K and 14 K, our data reveal a broad continuum of excitations, in contrast to the spin-wave scattering observed in the spin-1/2 triangular-lattice compound Ba3CoSb2O9 (ref. 25). The excitation continuum has a bandwidth of 1.3 meV and is gapless within the experimental energy resolution of approximately 0.1 meV. This scattering spans a much smaller energy scale but is otherwise qualitatively similar to herbertsmithite7,20—a surprising result, because the kagome lattice of herbertsmithite is considered more highly frustrated than the triangular lattice of YbMgGaO4 (refs 8,26).

Figure 2: Neutron-scattering data for YbMgGaO4 measured in zero applied field.
figure 2

a, Energy dependence of magnetic excitations along high-symmetry directions in reciprocal space, showing data at 0.06 K (upper panel) and 14 K (lower panel). Reciprocal-space points are labelled in b and scattering intensity in arbitrary units is shown as a colour scale. The temperature of 0.06 K was measured at the mixing chamber of our dilution refrigerator. b, Wavevector dependence of magnetic excitations at 0.06 K integrated for different energy transfers: scattering within the elastic instrumental resolution (upper left panel), 0.25 to 0.5 meV (upper right panel) and 0.5 to 1.0 meV (lower panel). c, Energy dependence of the scattering at the M point. The red line corresponds to the amount of scattering within the 0.11 meV elastic resolution of the instrument. d, One-dimensional cuts along high-symmetry directions at 0.06 K (upper panel) and 14 K (lower panel). Each panel shows energy transfers of 0.25, 0.45 and 0.65 meV (labelled on the graph). For a,b upper panels, c and d, the incident neutron energy Ei = 3.32 meV and the data have been integrated over the vertical range ; for b lower panel, Ei = 12 meV and −1 ≤l≤ 1. For bd, the 7.8 T data (see below) are used as background. The scattering intensity in the top panels of b is multiplied by a factor 2/3 to share the same colour scale as a. Error bars correspond to 1 standard error.

The wavevector dependence of the 0.06 K scattering intensity is shown in Fig. 2b (see also Supplementary Fig. 5). The elastic scattering is much broader than the instrumental resolution (δQ/Q ≈ 2%) and reveals no sharp magnetic Bragg peaks, confirming the absence of long-range magnetic order at 0.06 K. The low-energy scattering (E ≤ 0.5 meV) shows a broad maximum at the point of the hexagonal Brillouin zone, whereas at higher energies (0.5 ≤ E ≤ 1.0 meV) the scattering is isotropic around the zone boundary. The energy dependence of the magnetic signal at the M point (Fig. 2c) indicates that at most 16(3)% of the total spectral weight is elastic. Correspondingly, the inelastic contribution (≥84(3)%) is large compared with the 66% expected for a spin-1/2 glass27. This rules out a totally frozen state for YbMgGaO4, in agreement with μSR5. Further insight into the M-point scattering maximum is obtained from line plots at different energies (Fig. 2d). At 0.06 K, these plots reveal a weak dispersive excitation originating from the M point, superimposed on the continuum shown in Fig. 2a. Strikingly, some structure persists in the excitation spectrum at 14 K (3θW), but the location of the intensity maximum at low energy (0.25 meV) shifts to between Γ ≡ (000) and points (Fig. 2c).

What is the probable origin of the QSL behaviour of YbMgGaO4? To answer this question, we measured its spectrum with a large magnetic field applied along the c axis. The field polarizes the spins, yielding excitations that can be modelled by conventional linear spin-wave theory to determine the values of the exchange interactions21. The starting point for our analysis is the effective spin-1/2 Hamiltonian4,21,28,

where spin operators S± = Sx ± iSy, g and g are the components of the g-tensor parallel and perpendicular to the applied field, and the complex numbers γij are defined in ref. 4. Brackets 〈〉 and 〈〈〉〉 denote nearest- and next-nearest-neighbour pairs, respectively. The exchange interactions Jzz and J± define an XXZ model29, where we include nearest- and next-nearest-neighbour interactions (denoted by subscripts 1 and 2, respectively). We also include a symmetry-allowed bond-dependent (also known as pseudo-dipolar) interaction, J1±±; however, we neglect the other bond-dependent interaction (J1z± in ref. 4), because our current experimental data are insensitive to this term and ESR measurements indicate that it takes a very small value, 0.003(9) meV (ref. 4). For a large applied field along c, it follows from equation (1) that the spin-wave dispersion is given by13,14

where r1, i and r2, i label the nearest and next-nearest-neighbour vectors, respectively.

Our experimental data measured in a 7.8 T field close to the ≈ 8 T saturation are shown in Fig. 3a. As anticipated from equation (2) and from the field dependence of the specific heat (Fig. 1b), the applied field induces a spin gap of 1.0 meV, and a single dispersive signal with a bandwidth of 1.1 meV dominates the spectrum. A continuum of excitations is also visible and carries a significant spectral weight. This is a surprising observation that requires further investigation; we postulate that the continuum originates from exchange disorder induced by a distribution of local environments across Yb3+ ions. Nevertheless, two observations confirm that the system is essentially polarized: the integrated intensity of the inelastic scattering at 7.8 T is relatively constant throughout the Brillouin zone (Fig. 3b), and the overall intensity is significantly reduced compared with 0 T, as expected for a field-induced ferromagnetic state30.

Figure 3: Field-polarized neutron-scattering data and evidence for next-nearest-neighbour interactions in YbMgGaO4.
figure 3

a, Energy dependence of magnetic excitations along high-symmetry directions, measured at 0.06 K in an applied field of 7.8 T. The white circles show the location of the maximum intensity at each wavevector (see text). The red and blue lines show a fit to the spin-wave dispersion relation, equation (2). The labelling of reciprocal-space positions is given in Fig. 2b. The rightmost panel, for which , demonstrates the absence of dispersion along l (perpendicular to the triangular planes). b, One-dimensional plots of the 0.06 K E-integrated magnetic signal, showing that the intensity is roughly constant along the chosen path, with the vicinity of the Y point as a notable exception. c, Representative fit to the E-dependence of the data shown in a, used to determine the position of the intensity maximum at a given wavevector (white circles in a). d, Dependence of the goodness-of-fit parameter χ2 on the values of the exchange interactions |J1±±| and J2±, and two sets of exchanges discussed in the main text. e, Experimental diffuse-scattering data at 0.06 K obtained by integrating over energy transfer (top panel); calculation from classical Monte Carlo simulations of our |J1±±| = 0 model with XXZ nearest-neighbour interactions only (bottom left panel); and calculation including next-nearest-neighbour interactions (bottom right panel). f, Calculated diffuse scattering for models with nearest-neighbour interactions only and |J1±±| /J1± = 1.36 with J1±± > 0 (right panel) and J1±± < 0 (left panel) at a temperature of 5 K. The data in ac were measured with Ei = 3.32 meV, and the data in e with Ei = 12 meV . Error bars correspond to 1 standard error.

To model the dispersion, we first fit the E-dependence at each Q with a double (that is, asymmetric) Lorentzian to determine the peak position; a representative fit is shown in Fig. 3c. We then fit equation (2) to the extracted dispersion curve by varying the exchange parameters J1zz + J2zz, J1±, |J1±±|, and J2±. The choice of a purely two-dimensional model is justified by the absence of visible dispersion along l (Fig. 3a). Throughout, we fix g = 3.721 from magnetization data4. Initially, we also fix |J1±±| = 0.013 meV from ESR measurements4. We obtain an excellent fit with J1zz + J2zz = 0.154(3), J1± = 0.109(4) and J2± = 0.024(1) meV . The ratio Jzz/J± = 1.16(4) is comparable to the result from magnetization measurements (Jzz/J± = 1.09(13); ref. 4), locating the interactions of YbMgGaO4 between isotropic and planar limits (Jzz/J± equal to 2 and 0, respectively). In the limit of planar nearest-neighbour interactions, a gapless Dirac QSL may be stabilized31; however, this phase does not persist for J1zz/J1± > 0 (ref. 29). The parameters J1±± and J2± are therefore of particular interest, because both may in principle stabilize a QSL12,13,14. Figure 3d shows the dependence of the goodness-of-fit on these parameters. Our dispersion curve is relatively insensitive to |J1±±| : the best fit is for |J1±±| = 0, but the previously reported value of 0.013 meV (ref. 4) yields visually indistinguishable results. Crucially, however, our data are highly sensitive to J2±. Matching the shallow dispersion minimum at the M point yields a best-fit ratio J2±/J1± ≈ 0.22 (Fig. 3a, solid red line). It is possible to reproduce this minimum without J2± (ref. 13,14), but for fixed planar anisotropy Jzz/J± = 1.1 this yields a large and dominant |J1±±| = 0.15(1) meV (Fig. 3a, blue dashed line), contradicting the ESR result (|J1±±| = 0.013(1) meV) (ref. 4). Remarkably, the ratio J2±/J1± lies close to the QSL regime of the spin-1/2 J1-J2 Heisenberg triangular-lattice antiferromagnet, which is predicted to occur for 0.06 J2/J1 0.19 (refs 9,10,11,12). Taken together with the published ESR results4, our 7.8 T results thus point towards moderate planar anisotropy with small antiferromagnetic next-nearest-neighbour J2± as a minimal model for YbMgGaO4.

Instantaneous spin-pair correlations in zero applied field provide an independent check on our results. We obtain the experimental diffuse intensity I(Q) = ∫ 0E(1 + eβE)I(Q, E) dE, where E′ = 1.6 meV, and compare these data with classical Monte Carlo simulations in Fig. 3e, f. Our calculations, driven by equation (1), use our fitted values of the XXZ exchange interactions and assume J1zz/J1± = J2zz/J2±, where relevant. At a simulation temperature of 1.3 K, the calculated diffuse intensity for J2± = J1±± = 0 has its maximum at the K point, contrary to experiment. However, including J2± = 0.22J1± reproduces the M-point maximum observed experimentally (Fig. 3e)—a result also observed in quantum calculations12. A dominant J1±± also produces M-point scattering4 but introduces an additional intensity modulation that depends on the sign of J1±±, and is not observed in our experimental data (Fig. 3f and Supplementary Fig. 6). Crucially, after cooling our Monte Carlo simulations below 1.3 K, the calculated diffuse scattering shows much sharper features than the experimental data (Supplementary Fig. 7). Quantum fluctuations are thus a possible contender to explain the suppressed spin correlation length of 2.8 Å we observe at 0.06 K.

Our observations in YbMgGaO4 set it apart from other inorganic triangular-lattice magnets with quantum spins, in which magnetic order or spin freezing typically occurs at a temperature θW/10 (see, for example, refs 25,32). In contrast, YbMgGaO4 shows a continuous excitation spectrum without magnetic order to temperatures below θW/60. Our results identify magnetic scattering at the M point of the hexagonal Brillouin zone and provide strong constraints on spin-space anisotropies and next-nearest-neighbour interactions to guide further theoretical investigations. Our work also calls for further studies to elucidate the effect of interstitial Mg2+/Ga3+ disorder on exchange interactions. Compared with molecule-based triangular QSL candidates33,34,35, the strength of the magnetic signal and the availability of large single-crystal samples make YbMgGaO4 an exceptional candidate for neutron-scattering experiments, suggesting that mapping the response of a two-dimensional QSL candidate to temperature, applied magnetic field, and chemical perturbations is now a practical prospect.

Methods

Sample preparation.

A polycrystalline sample of YbMgGaO4 was synthesized by a solid-state method. Stoichiometric ratios of Yb2O3, MgO and Ga2O3 fine powder were carefully ground and reacted at a temperature of 1,450 °C for 3 days with several intermediate grindings. The single-crystal sample of YbMgGaO4 (Supplementary Fig. 2) was grown using the optical floating-zone method under a 5 atm oxygen atmosphere4. The best single crystal was obtained with a pulling speed of 1.5 mm h−1, and showed cliffed [001] surfaces after several hours of growth.

X-ray diffraction measurements and refinements.

Room-temperature powder X-ray diffraction were carried out on powder and crushed single-crystal samples using a Panalytical X’pert Pro Alpha-1 diffractometer with monochromatic Cu-Kα radiation (λ = 1.540598 Å). Preliminary measurements in flat-plate geometry on a loose powder using a Huber X-ray diffractometer showed preferred orientation, especially for the crushed-crystal sample. To minimize the extent of preferred orientation, we loaded our samples into a glass capillary that was rotated at 30 r.p.m. Due to the large absorption cross-section of Yb, these measurements were limited to small sample sizes. Measurements were taken between 5 ≤ 2θ ≤ 140° with Δ2θ = 0.016°. Rietveld refinement was carried out using the FULLPROF program36. Peak shapes were modelled by pseudo-Voigt functions, and the remaining preferred orientation was treated within the March model37. Fits to data are shown in Supplementary Fig. 4, and refined values of structural parameters are given in Supplementary Table 1.

Thermomagnetic measurements.

Heat-capacity measurements were carried out on a Quantum Design physical property measurement system (PPMS) instrument using dilution fridge (0.06 ≤ T ≤ 4 K) and standard (1.6 ≤ T ≤ 100 K) probes in a range of measuring magnetic fields, 0 ≤ μ0H ≤ 14 T. Single-crystal measurements were made on a flat thin piece polished to ≈ 1 mg and oriented with the c axis parallel to the applied magnetic field. To ensure sample thermalization at low temperatures, powder measurements were made on pellets of YbMgGaO4 mixed with an approximately equal mass of silver powder, the contribution of which was measured separately and subtracted to obtain the specific heat Cp. The magnetic specific heat Cm was obtained by subtracting a modelled lattice contribution CL with two Debye temperatures, 480 K and 142 K. The change in magnetic entropy ΔS(T) was subsequently obtained by integrating Cm/T from 0.06 K to T. Isothermal magnetization measurements M(H) were performed on the above single-crystal piece using a PPMS vibrating sample magnetometer in a range of magnetic fields 0 ≤ μ0H ≤ 14 T and temperatures 1.7 ≤ T ≤ 10 K.

Low-energy inelastic neutron-scattering measurements.

Inelastic neutron-scattering24 experiments were performed on the cold neutron chopper spectrometer (CNCS) at the Spallation Neutron Source (SNS), Oak Ridge National Laboratory38. The sample was a 2.2 g rod-shaped crystal cut into two shorter pieces to fit in the bore of a cryomagnet. The two pieces (total dimensions 16 × 16 × 4 mm3) were co-aligned to within 1.5° using a Multiwire X-ray Laue backscattering machine, and mounted in the (hk0) scattering plane on a oxygen-free copper holder (Supplementary Fig. 2). The mount was attached to the bottom of a dilution refrigerator reaching a base temperature of 0.06 K at the mixing chamber, indicating a sample temperature 0.1 K. The sample stick was inserted in an 8 T superconducting cryomagnet, and measurements were performed in zero field and in a field of 7.8 T applied along the c axis. Two incident neutron energies were used, Ei = 3.32 and 12.0 meV, yielding elastic energy resolutions (full-width at half-maximum) of 0.11 and 0.75 meV, respectively. The sample was rotated in steps of 1°, with a range of 270° for Ei = 12 meV and 180° for Ei = 3.32 meV . For Ei = 3.32 meV, the background scattering from the sample environment was measured and subtracted from the data. For Ei = 12 meV, the background and non-magnetic scattering at low energy (E ≤ 0.9 meV) was subtracted using the 7.8 T measurement, taking advantage of the spin gap induced by applied field.

Broadband neutron-scattering measurements.

Broadband neutron-scattering experiments were performed on a sintered powder sample and on one of the above single crystals using the SEQUOIA spectrometer at ORNL’s SNS39. The powder sample was mounted in a cylindrical aluminium can sealed under one atmosphere of 4He at room temperature. The single crystal was mounted with its ab plane horizontal and rotated continuously, yielding an effective two-dimensional powder averaging. Both samples were cooled using a closed-cycle refrigerator reaching a base temperature of T = 4 K. The spectrometer was operated with three distinct configurations to cover a large range of momentum and energy transfers: Ei = 300 meV with a Fermi chopper frequency of f = 420 Hz, Ei = 160 meV with f = 600 Hz, and Ei = 22 meV with f = 300 Hz. Background and sample holder contributions were subtracted from the data using empty can measurements. The contribution from the lattice was subtracted by extracting the phonon density of states at |Q| = 6.0 Å−1 (160 meV) and |Q| = 9.0 Å−1 (300 meV) and extrapolating to other momenta using the known |Q|2 intensity dependence for single-phonon scattering40. The position and width of the observed excitations were obtained through fits to Gaussian profiles. The energy resolution of the spectrometer was estimated using simplified geometrical considerations yielding a correct elastic energy resolution but known to under-estimate the energy resolution at finite energy transfer. Thirteen electrons reside in the 4f shell of the Kramers ion Yb3+. Following ref. 41, the crystal electric field can maximally split the local electronic manifold into four Kramers doublets yielding three excited doublets above the doublet ground state.

Elastic neutron-scattering measurements.

Elastic neutron-scattering experiments were performed on the CORELLI spectrometer at ORNL’s SNS. One of the two above crystal pieces was attached to a copper pin at the bottom of a 3He cryostat reaching a base temperature of 0.3 K. The sample was aligned in the (h0l) scattering plane to assess crystal quality and stacking of the triangular-lattice planes along c. Neutron-absorbing Cd was used to shield the sample holder and an empty cryostat measurement was used to remove the background contribution. The sample was rotated in steps of 6° over a 360° range, and measurements were taken at temperatures of 0.3, 4.0 and 40 K. Elastic neutron-scattering data measured at 0.3 K are shown in Supplementary Fig. 3.

Data analysis.

Initial data reduction was performed in MANTID42 for CNCS, SEQUOIA and CORELLI data sets. For the CNCS data, subsequent analysis was performed in HORACE43 on a dedicated node within Georgia Tech’s Partnership for Advanced Computing infrastructure. To increase counting statistics, inelastic-scattering data were symmetrized into an irreducible 60° wedge of the hexagonal reciprocal lattice. The CORELLI data were normalized to absolute units in MANTID44.

Monte Carlo simulations.

To perform classical Monte Carlo simulations, we rewrite the spin Hamiltonian, equation (1), for zero field in terms of spin components Sx, Sy and Sz:

where the phase factors for nearest-neighbour bonds along the directions a, b and a + b, respectively (where a and b are shown in Fig. 1a). In Fig. 3e, we keep J1zz = 0.126 and J1± = 0.109 meV with either J2zz = 0 and J2± = 0 (bottom left) or J2zz = 0.027 and J2± = 0.024 meV (bottom right). In Fig. 3f, we take J1±± = ± 0.15 meV with J1zz = 0.12 and J1± = 0.11 meV . The effect of small non-zero J1±± = ± 0.013 meV is discussed in the Supplementary Information. We take the length of spin vectors as . Simulations were performed on a three-dimensional spin configuration consisting of nine triangular layers, each containing 504 spins. A proposed spin move consists of rotating a spin by a small amount (chosen so that 50–70% of proposed moves were accepted). The simulations were initialized at a high temperature and cooled in gradual increments. At each temperature, the simulation was run for at least 10t0 proposed moves, where t0 is the number of proposed moves required to decorrelate the system. The single-crystal diffuse-scattering intensity was calculated as40

where f(Q) is the Yb3+ magnetic form factor45, and Si = SiQ(Si Q)/Q2 is the component of spin Si perpendicular to Q. The calculated pattern was slightly smoothed to allow calculation on an arbitrary Q-grid. To increase statistical averaging, the calculated I(Q) was averaged over 100 spin configurations and diffraction symmetry was applied.

Data availability.

The data that support the plots within this paper and other findings of this study are available from the corresponding author on request.