## Abstract

Thermal frequency fluctuations in optical cavities limit the sensitivity of precision experiments ranging from gravitational wave observatories to optical atomic clocks. Conventional modeling of these noises assumes a linear response of the optical field to the fluctuations of cavity frequency. Fundamentally, however, this response is nonlinear. Here we show that nonlinearly transduced thermal fluctuations of cavity frequency can dominate the broadband noise in photodetection, even when the magnitude of fluctuations is much smaller than the cavity linewidth. We term this noise “thermal intermodulation noise” and show that for a resonant laser probe it manifests as intensity fluctuations. We report and characterize thermal intermodulation noise in an optomechanical cavity, where the frequency fluctuations are caused by mechanical Brownian motion, and find excellent agreement with our developed theoretical model. We demonstrate that the effect is particularly relevant to quantum optomechanics: using a phononic crystal ${{\rm Si}_3}{{\rm N}_4}$ membrane with a low-mass, soft-clamped mechanical mode, we are able to operate in the regime where measurement quantum backaction contributes as much force noise as the thermal environment does. However, in the presence of intermodulation noise, quantum signatures of measurement are not revealed in direct photodetection. The reported noise mechanism, while studied for an optomechanical system, can exist in any optical cavity.

© 2020 Optical Society of America under the terms of the OSA Open Access Publishing Agreement

## 1. INTRODUCTION

Optical cavities are an enabling technology for precision experiments, including gravitational wave detection [1], ultrastable lasers [2], cavity quantum electrodynamics [3], and cavity optomechanics [4]. Depending on the application, they can be exquisite detectors for performing quantum-limited measurements [5] or highly stable frequency references for clock lasers [2]. In both cases, the performance is limited by fundamental thermodynamic frequency fluctuations that exist in any cavity at finite temperature, be it as a result of the Brownian motion of mirror surfaces, fluctuations of the refractive index, or the thermoelastic effect [6,7]. Minimizing these fluctuations has played a key role in the design of interferometric gravitational wave detectors [8], motivated the development of crystalline mirror coatings [9] and cryogenic operation of reference cavities for optical frequency metrology [10].

Thermal noises are particularly strong in optical cavities at the micro- and nanoscale. High-finesse optical microcavities are increasingly employed in a variety of precision measurements, ranging from compact reference cavities using crystalline microresonators [11 –13] to photonic integrated microresonator-based quantum optics experiments [14 –16]. In optical frequency metrology, driven Kerr nonlinear microresonators produce octave-spanning combs [17] whose carrier envelope frequency is thermal noise limited [18]. In such microcavities thermal fluctuations are particularly prominent because of the small mode volume [19 –21].

The conventional framework in which optical measurements are described assumes linear transduction of cavity frequency fluctuations into the optical field, justified by the frequency excursions being small compared to the cavity linewidth. However, the nonlinearity of transduction is inherently present in any cavity. It gives rise to qualitatively new effects and results in the conversion of Gaussian fluctuations of cavity frequency into non-Gaussian fluctuations of the optical field. When a cavity is coupled to a quantum system, this phenomenon has been proposed for performing nonlinear quantum measurements [22 –24], which cannot be described within the leading-order perturbation theory [5]. At the same time, the nonlinear conversion of thermal frequency fluctuations can impose qualitatively new constraints on a broad range of precision experiments, which to date have not been analyzed.

Here we report for the first time to our knowledge that the nonlinear modulation of the optical field by thermal frequency fluctuations can manifest as a broadband added noise in detection, the bandwidth of which is limited by the cavity decay rate. We refer to this noise as *thermal intermodulation noise* (TIN) since it mixes different Fourier components of cavity frequency fluctuations. This noise dominates when the linearly transduced thermal fluctuations are small, such as when detecting the intensity of a near-resonant optical probe. As it is the leading-order contribution, TIN it is not necessarily negligible, even when the nonlinearity of cavity transduction is small.

We experimentally observe and study TIN in a membrane-in-the-middle (MIM) optomechanical system [25,26]—a promising platform for room-temperature quantum optomechanical experiments [27,28]—and find excellent agreement with our developed theoretical model. Using a ${{\rm Si}_3}{{\rm N}_4}$ membrane resonator hosting a high-$Q$ and low-mass soft-clamped mode [29,30], we operate at a nominal quantum cooperativity of unity, i.e., in the regime where the linear measurement quantum backaction (arising from radiation pressure quantum fluctuations) is expected to overwhelm the thermal motion. This regime is required for a range of quantum-enhanced measurement protocols [31 –33] and for generation of optical squeezed states [34,35]. Yet the nonlinearity of our cavity prevents the observation of quantum correlations between the field quadratures and manifests itself in TIN significantly above the shot noise (i.e., quantum noise) level. Surprisingly, we find that TIN dominates the fluctuations of the intensity of the optical field even when the thermally induced frequency fluctuations are substantially smaller than the cavity linewidth. Since TIN is a coherent effect, it only requires the knowledge of spectrum of cavity frequency fluctuations to be modeled, and our experimental data is well matched by a model with no free parameters.

We show that for a particular “magic” detuning from the cavity, TIN is fully cancelled in direct detection, and we propose a more general cancellation scheme suitable for arbitrary detuning. Our observations, while made for an optomechanical system, are broadly applicable, irrespective of the underlying thermal noise source. Thermal intermodulation noise can be of relevance to any cavity-based measurement scheme at finite temperature.

## 2. THEORY OF THERMAL INTERMODULATION NOISE

We begin by presenting the theory of thermal intermodulation noise in the classical regime with the assumption that the cavity frequency fluctuations are slow compared to the optical decay rate. We concentrate on the lowest-order, i.e., quadratic, nonlinearity of the cavity detuning transduction. We consider (as in our experimental setup) an optical cavity with two ports, which is driven by a laser coupled to port 1. The output from port 2 is directly detected on a photodiode. In the classical regime, i.e., neglecting vacuum fluctuations, the complex amplitude of the intracavity optical field $a$ and the output field ${s_{\!{\rm out,2}}}$ can be found from the input-output relations

Notice that $\delta \nu (t)$ and $\delta \nu {(t)^2}$ can be distinguished by their detuning dependence [36]. The linearly transduced fluctuations vanish on resonance (${\nu _0} = 0$), where $\partial |L{|^2}/\partial \nu = 0$. Similarly, when ${\partial ^2}|L{|^2}/\partial {\nu ^2} = 0$, the quadratic frequency fluctuations vanish, and thus so does the thermal intermodulation noise. We refer to the corresponding detuning values

as “magic.” In the following experiments, we will make measurements at ${\nu _0} = - 1/\sqrt 3$ and ${\nu _0} = 0$ to independently characterize the spectra of $\delta \nu (t)$ and $\delta \nu {(t)^2}$, respectively.The total spectrum [37] of the detected signal $I(t)$ is an incoherent sum of the linear term given by

## 3. BROWNIAN INTERMODULATION NOISE

In an optomechanical cavity, the dominant source of cavity frequency fluctuations is the Brownian motion of mechanical modes coupled to the cavity:

where $G = - \partial {\omega _c}/\partial x$ is the linear optomechanical coupling constant and $x$ is the total resonator displacement, i.e., the sum of independent contributions ${x_n}$ of different mechanical modes (the effect of the finite cavity mode waist is treated in the Supplement 1). The spectrum of the Brownian frequency noise is then found to beA reservation needs to be made: the theory presented in Section 2 is only strictly applicable to an optomechanical cavity when the input power is sufficiently low, such that the driving of mechanical motion by radiation pressure fluctuations created by the intermodulation noise is negligible; otherwise the fluctuations of $x(t)$ and $\delta \nu (t)$ may deviate from purely Gaussian, and correlations exist between $\delta \nu (t)$ and $\delta \nu {(t)^2}$. On a practical level, this reservation has minor significance for our experiment. Also, the presence of linear dynamical backaction of radiation pressure does not change the results of Section 2 but does modify ${S_{\textit{xx}}}$.

Thermal intermodulation noise can preclude the observation of linear quantum correlations, which are induced by the vacuum fluctuations of radiation pressure between the quadratures of light and manifest as ponderomotive squeezing [34,35], Raman sideband asymmetry [39], and the cancellation of shot noise in force measurements [32,33]. The observation of quantum correlations typically requires selecting a mechanical mode with high quality factor $Q$, a spectral neighbourhood free from other modes, and a high optomechanical coupling rate. If TIN is taken into account, simply increasing the quantum cooperativity is not sufficient, and the following condition also needs to be satisfied:

The nonlinearity of the cavity-laser detuning response, which produces TIN, modulates the optical field proportional to ${x^2}$ in a way analogous to, but not equivalent to, quadratic optomechanical coupling ${\partial ^2}{\omega _c}/\partial {x^2}$. It was noticed that the cavity transduction commonly results in an effective quadratic coupling that is orders of magnitude stronger than the highest experimentally reported ${\partial ^2}{\omega _c}/\partial {x^2}$ (in terms of the optical signal proportional to ${x^2}$ [22,23]). In the Supplement 1, it is shown that the same is true in the MIM system. Here the quadratic signal originating from nonlinear transduction, which creates the intermodulation noise, is larger than the signals due to nonlinear optomechanical coupling ${\partial ^2}{\omega _c}/\partial {x^2}$ by a factor of $r{\cal F}$, where $r$ is the membrane reflectivity and ${\cal F}$ is the optical finesse.

## 4. EXPERIMENTAL OBSERVATION OF THERMAL INTERMODULATION NOISE

Our experimental setup, shown in Fig. 1(c), comprises a MIM cavity consisting of two high-reflectivity mirrors and a chip with high-stress stoichiometric ${{\rm Si}_3}{{\rm N}_4}$ membrane sandwiched directly between them. The MIM cavity is situated in a vacuum chamber at room temperature and probed in transmission. See the Supplement 1 for more details.

#### A. Intermodulation Noise in a Cavity with a Uniform Membrane

We first characterize the TIN in cavities with 20 nm thick uniform square membranes of different sizes. The optomechanical cooperativity was kept low in order to eliminate dynamical backaction of the light; this was achieved by increasing the vacuum pressure and keeping the mechanical modes gas damped to $Q \sim {10^3}$. The reflection signals of two resonances of a MIM cavity with a 2 mm membrane are presented in Fig. 2(a). The resonances have similar optical linewidths (about 15 MHz), but their optomechanical coupling is different by a factor of 10. The resonance with high coupling (${g_0}/2\pi = 150\;{\rm Hz} $) shows clear signatures of thermal noise. For this resonance, the total rms thermal frequency fluctuations are expected to be around 2 MHz, which is still well below the cavity linewidth $\kappa /2\pi = 16\;{\rm MHz} $. The detuning scan rates are approximately 1 THz/s for both plots in Fig. 2(a).

Thermal fluctuations of the reflection signal are clearly observed in the right panel of Fig. 2(a), even when the laser is resonant with the cavity. This is not expected in linear optomechanics, where the mechanical motion only modulates the phase of a resonant laser probe. Typical spectra of the detected noise are shown in Fig. 2(d) for a cavity with a different, 1 mm, square membrane. With the laser detuned from the cavity resonance (close to the “magic” detuning ${\nu _0} \approx - 1/\sqrt 3$), the transmission signal is dominated by the Brownian motion of membrane modes transduced by the cavity [shown in Fig. 2(d)] and by the extraneous thermal noise from the mirrors, in agreement with the prediction of linear optomechanics. The magnitude of thermomechanical noise is gradually reduced at high frequencies due to the averaging of membrane mode profiles [40,41] over the cavity waist, until it meets shot noise at around 15 MHz. With the laser on resonance, from linear optomechanics it is expected that the output signal is shot-noise limited. However, the experimental signal [shown in Fig. 2(e)] contains a large amount of thermal noise—at an input power of 5 µW the classical relative intensity noise (RIN) exceeds the shot noise level by about 25 dB at MHz frequencies. The spectrum of the resonant RIN is different from the spectrum of detuning fluctuations, owing to the nonlinear origin of the noise. At high frequency, the RIN level approaches shot noise, as verified by the optical power dependence (see the Supplement 1).

An unambiguous proof of the intermodulation origin of the resonant intensity noise is obtained by examining the scaling of the noise level with $G/\kappa$. In thermal equilibrium, the spectral density of frequency fluctuations $\delta \nu (t)$ created by a particular membrane is proportional to ${(G/\kappa)^2}$, and therefore the spectral density of intermodulation noise is expected to be proportional to ${(G/\kappa)^4}$. We confirm this scaling by measuring the resonant intensity noise for different optical resonances of a cavity with a 2 mm membrane and present in Fig. 2(c) the average noise magnitude as a function of ${g_0}/\kappa$, where ${g_0}$ is the optomechanical coupling of the fundamental mode. By performing a sweep of the input laser power on one of the resonances of the same cavity, we show [see Fig. 2(b)] that the resonant intensity noise level is power independent, and therefore the noise is not related to radiation pressure effects.

The TIN observed in our experiments agrees well with our theoretical model. By calculating the spectrum of total membrane fluctuations according to Eq. (13) and applying the convolution formula from Eq. (11) (see the Supplement 1 for full details), we can accurately reproduce the observed noise. In Fig. 3, we compare the measured detuning and intensity noise spectra with the theoretical model. Here we assume that the damping rates of all the membrane modes are identical, as the experiment is operated in the gas-damping-dominated regime. While this model is not detailed enough to reproduce all the noise features, it accurately reproduces the overall magnitude and the broadband envelope of the intermodulation noise observed in the experiment.

We would like to address two potential confounding effects: *laser frequency noise* and *dissipative coupling*. The intensity noise of the laser was below ${10^{- 12}}\;{{\rm Hz}^{- 1}}$ for frequencies above 100 kHz and therefore negligible in all resonant RIN measurements. In the same frequency range, the frequency noise of the laser is below $1\;{{\rm Hz}^2}/{\rm Hz}$, which is, again, much lower than the thermomechanical noise. For more details on the extraneous noises, see the Supplement 1. As dissipative coupling leads to the modulation of optical linewidth by mechanical position, it could also potentially explain intensity noise in a resonant optical field. Although dissipative coupling is generally present in MIM cavities [41], the magnitude of this noise is expected to be orders of magnitude below that measured in our experiments (see the Supplement 1). Moreover, dissipative coupling cannot explain the observed scaling of resonant RIN ($\propto {(G/\kappa)^4}$) and the absence of correlation between the RIN level and the excess optical loss added by the membrane.

#### B. Thermal Intermodulation Noise Caused by a Soft-Clamped Phononic Crystal Membrane

Localized (“soft-clamped”) defect modes in stressed phononic crystal (PnC) resonators can have quality factors in excess of ${10^8}$ at room temperature due to enhanced dissipation dilution [29,42]. Owing to their high $Q$ and low effective mass, which result in low thermal force noise ${S_{{\rm FF,th}}} = 2{k_B}T{m_{{\rm eff}}}{\Gamma _m}$, these modes are promising for quantum optomechanics experiments [43].

In Figs. 4(a) and 4(b) we present ${{\rm Si}_3}{{\rm N}_4}$ PnC membranes with soft-clamped modes optimized for low effective mass and high $Q$. The phononic crystals are formed by the hexagonal pattern of circular holes introduced in Ref. [29], which creates a band gap for flexural modes. The phononic crystal is terminated to the silicon frame at half the hole radii in order to prevent mode localization at the membrane edges—such modes have low $Q$ and can have frequencies within the phononic band gap, contaminating the spectrum. Figure 4(a) shows a microscope image of a resonator with a trampoline defect, featuring ${m_{{\rm eff}}} = 3.8$ ng and $Q = 1.65 \times {10^8}$ at 0.853 MHz, corresponding to a thermal force noise ${S_{{\rm FF,th}}} = 13\;{\rm aN/}\sqrt {{\rm Hz}}$. Another resonator, shown in Fig. 4(b), is a 2 mm phononic crystal membrane with a defect engineered to create a single mode localized in the middle of the phononic band gap. The displayed sample has $Q = 7.4 \times {10^7}$ at 1.46 MHz and ${m_{{\rm eff}}} = 1.1\;{\rm ng}$, corresponding to ${S_{{\rm FF,th}}} = 34\;{\rm aN/}\sqrt {{\rm Hz}}$.

The phononic band gap spectrally isolates soft-clamped modes from the thermomechanical noise created by the rest of the membrane spectrum. Nevertheless, when a PnC membrane is incorporated in a MIM cavity the entire multitude of membrane modes contributes to the TIN *even within band-gap frequencies*, as TIN is produced by a nonlinear process. Figure 5(a) shows the spectrum of light transmitted through a resonance of MIM cavity with ${g_0}/2\pi = 0.9\;{\rm kHz} $ for the soft-clamped mode, $\kappa /2\pi = 34\;{\rm MHz} $, and ${C_0} = 2.5$. The noise at band-gap frequencies is dominated by TIN, which exceeds the shot noise by 4 orders of magnitude. The spectrum also shows a dispersive feature in the middle of the band gap, which is a signature of classical correlations due to the intracavity TIN exciting the localized mechanical mode. The mechanical resonator in this case is a 2 mm square PnC membrane with the patterning shown in Fig. 4(b), but made of 40 nm thick ${{\rm Si}_3}{{\rm N}_4}$. The membrane has a single soft-clamped mode with $Q = 4.1 \times {10^7}$ at 1.55 MHz, as characterized immediately before inserting the membrane in the cavity assembly. The input power in the measurement was 100 µW after correcting for spatial mode matching, which corresponds to a nominal ${C_q} \approx 1$. The shot noise level was calibrated in a separate measurement by directing an independent laser beam on the detector.

We next present in Fig. 5(b) the dependence of the band-gap noise level on the laser detuning, measured on a different optical resonance of the same MIM cavity and at lower input power. The measurement shows that the in-band-gap excess noise is dominated by TIN at all detunings except for the immediate vicinity of the “magic” detuning ${\nu _0} = - 1/\sqrt 3$. Around ${\nu _0} = - 1/\sqrt 3$ the excess noise is consistent with the substrate noise of an empty cavity (see the Supplement 1). The total noise level is well fitted by our model, which includes both ${S_{\nu \nu}}$ and $S_{\nu \nu}^{(2)}$ contributions to the detected signal and accounts for the radiation pressure cooling (full details are given in the Supplement 1). In the measurement in Fig. 5(b), ${g_0}/2\pi = 360\;{\rm Hz} $ for the localized mode, $\kappa /2\pi = 24.8\;{\rm MHz} $, and the input power was 30 µW. Note that the relation between the special densities of the linear and quadratic fluctuations in the band gap do not reflect the relation between the rms magnitudes of their contributions. The linear position fluctuations are low at band-gap frequencies, such that the quadratic contribution dominates the output spectrum even at large detunings from resonance, where the linear and quadratic transduction factors are of the same order. At the same time, the integral rms magnitude of the linear signal is always larger in our experiments, such that the second-order approximation of the transduction nonlinearity is valid.

The intensity of the detected light in our measurement is proportional to the intensity of the intracavity field. Therefore, the suppression of TIN at the magic detuning necessarily implies the suppression of the corresponding radiation pressure noise, which otherwise can lead to classical heating of the mechanical oscillator and thereby limit the true quantum cooperativity. Classical radiation pressure fluctuations due to linearly transduced mirror noise remain when operating at the magic detuning, which can also lead to oscillator heating.

## 5. A SCHEME TO CANCEL INTERMODULATION NOISE IN DETECTION

The existence of a “magic” detuning at which TIN is canceled in direct detection hints at a more general scheme. We show next that TIN can be removed from the measurement of an arbitrary optical quadrature at arbitrary cavity detuning by an appropriate choice of local oscillator in single-port homodyne detection. Single-port homodyning is a quantum-limited detection scheme if the signal beam is combined with a local oscillator on a highly asymmetric beam splitter.

Keeping terms up to the second order in $\delta \nu$, the complex amplitude of the optical field $s$ in front of the photodetector of a single-port homodyne is given by

where ${s_1} \propto L^\prime ({\nu _0})$ and ${s_2} \propto L^{\prime \prime} ({\nu _0})$. The carrier ${s_0} = r{e^{{i\theta}}}$ can be set arbitrarily by an appropriate choice of the local oscillator phase and amplitude. The photocurrent is found asThe cancellation of intermodulation noise allows cavity-based measurements limited by quantum vacuum fluctuations or extraneous thermal noises of the cavity, as in the ideal linear case. One caveat, however, is that if the cavity has a nonlinearity other than the transduction nonlinearity, the effect of intermodulation on intracavity dynamics, in general, will not be eliminated by the proposed cancellation scheme. Optomechanical cavities are nonlinear due to the radiation pressure interaction, and in this case intermodulation noise can lead to classical fluctuations of radiation pressure that heat the mechanical oscillator.

## 6. RELATION TO NONLINEAR QUANTUM MEASUREMENTS

When a cavity is used as a meter to perform measurements on quantum systems [5], the intracavity photon number ${\hat n_c}$ is coupled to an observable of the system $\hat x$ by means of the interaction Hamiltonian

where ${\omega _c}$ is the cavity frequency. The detected variable of the meter is the propagating optical field. If the coupling between the system and the meter is weak, the only way of performing nonlinear measurements, i.e., measurements of ${\hat x^2}$, is to create a nonlinear coupling ${\omega _c} \propto {x^2}$ [44,45]. In quantum optomechanics, where the microscopic system is a harmonic oscillator and $x$ is its position, quadratic measurements allow quantum nondemolition measurements of the oscillator energy [46]. Such measurements can be used for the observation of phononic jumps [44,45], phononic shot noise [47], and the creation of mechanical squeezed states [48] if the effects of linear measurement backaction are kept small [23,44]. While considerable efforts have been dedicated to realizing nonlinear optomechanical coupling, achievable coupling rates remain modest [25,49], and the corresponding experiments so far have been deeply in the classical regime.A different path towards nonlinear measurements [22] opens as the sensitivity of the meter is increased, for example, by increasing the finesse of the optical cavity. Eventually measurements enter the regime when the information is still obtained gradually, but the first-order perturbation theory [50] is not sufficient to describe the measurement process [51]. In this regime, nonlinear measurements are possible with linear coupling ${\omega _c} \propto x$, which was experimentally demonstrated [23,24,36], yet in the classical domain. At the same time it was shown in [23], and as we also show in this work (see Supplement 1), that under quite typical experimental conditions the nonlinearity of a cavity as a meter is orders of magnitude stronger than the nonlinearity due to quadratic coupling. Notably, it crucially requires the cavity being coupled to the propagating field, and it does not have an equivalent in a closed system of a mechanical oscillator coupled to an isolated optical mode [52].

## 7. CONCLUSIONS AND OUTLOOK

We have presented the observation and characterization of a previously unreported broadband thermal noise in optical cavities, TIN. It originates from the quadratic transduction of cavity frequency fluctuations within the optical cavity linewidth. The key qualitative feature of TIN is that it creates classical intensity fluctuations in an optical field resonant with the cavity, which is otherwise shot-noise limited. To the lowest order, the TIN magnitude grows quadratically with the ratio of rms thermal frequency fluctuations by the optical linewidth, and therefore it strongly affects high-finesse optical cavities with large frequency fluctuations, such as microcavities at room temperature.

Thermal intermodulation noise in optomechanical experiments can be avoided by using cavities with low finesse (equivalently, low ${g_0}/\kappa$) and by coupling them to mechanical resonators with lower total thermal fluctuations, i.e., those that have fewer mechanical modes, higher frequency, and higher $Q$ for all modes. The latter consideration could make the fundamental modes of mechanical resonators (e.g., low-mass trampolines [53]) seem preferable compared to high-$Q$ but high-order PnC defect soft-clamped modes. In this context, a newly proposed method of exploiting self-similar structures as mechanical resonators with soft-clamped fundamental modes [54] could potentially be fruitful for overcoming TIN. Another way of reducing the TIN is laser cooling of mechanical motion, either by dynamical backaction of a red-detuned beam or by active feedback. In this case, however, all mechanical modes that contribute to the total cavity noise must be efficiently cooled, which could be technically challenging.

The raw measurement data, analysis scripts, and membrane designs are available in [55].

## Funding

Schweizerischer Nationalfonds zur Förderung der Wissenschaftlichen Forschung (182103, 185870 (Ambizione)); Horizon 2020 Framework Programme (722923 (OMT)); Defense Advanced Research Projects Agency (D19AP00016 (QUORT)).

## Acknowledgment

The authors thank Ryan Schilling for fabrication advice, Ramin Lalezari at FiveNine Optics for his assistance with the development of the cavity mirrors, and Itay Shomroni for a careful reading of the paper. All samples were fabricated and grown in the Center of MicroNanoTechnology (CMi) at EPFL.

## Disclosures

The authors declare no conflicts of interest.

See Supplement 1 for supporting content.

## REFERENCES AND NOTES

**1. **LIGO Scientific Collaboration, and Virgo Collaboration, “Observation of gravitational waves from a binary black hole merger,” Phys. Rev. Lett. **116**, 061102 (2016). [CrossRef]

**2. **U. Sterr, T. Legero, T. Kessler, H. Schnatz, G. Grosche, O. Terra, and F. Riehle, “Ultrastable lasers: new developments and applications,” Proc. SPIE **7431**, 74310A (2009). [CrossRef]

**3. **J. Ye, H. J. Kimble, and H. Katori, “Quantum state engineering and precision metrology using state-insensitive light traps,” Science **320**, 1734–1738 (2008). [CrossRef]

**4. **M. Aspelmeyer, T. J. Kippenberg, and F. Marquardt, “Cavity optomechanics,” Rev. Mod. Phys. **86**, 1391 (2014). [CrossRef]

**5. **A. A. Clerk, M. H. Devoret, S. M. Girvin, F. Marquardt, and R. J. Schoelkopf, “Introduction to quantum noise, measurement, and amplification,” Rev. Mod. Phys. **82**, 1155–1208 (2010). [CrossRef]

**6. **V. B. Braginsky, M. L. Gorodetsky, and S. P. Vyatchanin, “Thermo-refractive noise in gravitational wave antennae,” Phys. Lett. A **271**, 303–307 (2000). [CrossRef]

**7. **M. L. Gorodetsky, “Thermal noises and noise compensation in high-reflection multilayer coating,” Phys. Lett. A **372**, 6813–6822 (2008). [CrossRef]

**8. **V. B. Braginsky, M. L. Gorodetsky, and S. P. Vyatchanin, “Thermodynamical fluctuations and photo-thermal shot noise in gravitational wave antennae,” Phys. Lett. A **264**, 1–10 (1999). [CrossRef]

**9. **G. D. Cole, W. Zhang, M. J. Martin, J. Ye, and M. Aspelmeyer, “Tenfold reduction of Brownian noise in high-reflectivity optical coatings,” Nat. Photonics **7**, 644–650 (2013). [CrossRef]

**10. **J. M. Robinson, E. Oelker, W. R. Milner, W. Zhang, T. Legero, D. G. Matei, F. Riehle, U. Sterr, and J. Ye, “Crystalline optical cavity at 4 k with thermal-noise-limited instability and ultralow drift,” Optica **6**, 240–243 (2019). [CrossRef]

**11. **A. B. Matsko, A. A. Savchenkov, N. Yu, and L. Maleki, “Whispering-gallery-mode resonators as frequency references. I. Fundamental limitations,” J. Opt. Soc. Am. B **24**, 1324–1335 (2007). [CrossRef]

**12. **J. Alnis, A. Schliesser, C. Y. Wang, J. Hofer, T. J. Kippenberg, and T. W. Hänsch, “Thermal-noise-limited crystalline whispering-gallery-mode resonator for laser stabilization,” Phys. Rev. A **84**, 011804 (2011). [CrossRef]

**13. **H. Lee, M.-G. Suh, T. Chen, J. Li, S. A. Diddams, and K. J. Vahala, “Spiral resonators for on-chip laser frequency stabilization,” Nat. Commun. **4**, 2468 (2013). [CrossRef]

**14. **A. Dutt, K. Luke, S. Manipatruni, A. L. Gaeta, P. Nussenzveig, and M. Lipson, “On-chip optical squeezing,” Phys. Rev. Appl. **3**, 044005 (2015). [CrossRef]

**15. **U. B. Hoff, B. M. Nielsen, and U. L. Andersen, “Integrated source of broadband quadrature squeezed light,” Opt. Express **23**, 12013–12036 (2015). [CrossRef]

**16. **A. Otterpohl, A. Otterpohl, A. Otterpohl, F. Sedlmeir, F. Sedlmeir, U. Vogl, U. Vogl, T. Dirmeier, T. Dirmeier, G. Shafiee, G. Shafiee, G. Schunk, G. Schunk, G. Schunk, D. V. Strekalov, H. G. L. Schwefel, H. G. L. Schwefel, T. Gehring, U. L. Andersen, U. L. Andersen, G. Leuchs, G. Leuchs, C. Marquardt, and C. Marquardt, “Squeezed vacuum states from a whispering gallery mode resonator,” Optica **6**, 1375–1380 (2019). [CrossRef]

**17. **T. J. Kippenberg, A. L. Gaeta, M. Lipson, and M. L. Gorodetsky, “Dissipative Kerr solitons in optical microresonators,” Science **361**, eaan8083 (2018). [CrossRef]

**18. **T. E. Drake, J. R. Stone, T. C. Briles, and S. B. Papp, “Thermal decoherence and laser cooling of Kerr microresonator solitons,” Nat. Photon. **14**, 480–485 (2020) [CrossRef] .

**19. **M. L. Gorodetsky and I. S. Grudinin, “Fundamental thermal fluctuations in microspheres,” J. Opt. Soc. Am. B **21**, 697–705 (2004). [CrossRef]

**20. **G. Anetsberger, O. Arcizet, Q. P. Unterreithmeier, R. Rivière, A. Schliesser, E. M. Weig, J. P. Kotthaus, and T. J. Kippenberg, “Near-field cavity optomechanics with nanomechanical oscillators,” Nat. Phys. **5**, 909–914 (2009). [CrossRef]

**21. **G. Huang, E. Lucas, J. Liu, A. S. Raja, G. Lihachev, M. L. Gorodetsky, N. J. Engelsen, and T. J. Kippenberg, “Thermorefractive noise in silicon-nitride microresonators,” Phys. Rev. A **99**, 061801 (2019). [CrossRef]

**22. **M. R. Vanner, “Selective linear or quadratic optomechanical coupling via measurement,” Phys. Rev. X **1**, 021011 (2011). [CrossRef]

**23. **G. A. Brawley, M. R. Vanner, P. E. Larsen, S. Schmid, A. Boisen, and W. P. Bowen, “Nonlinear optomechanical measurement of mechanical motion,” Nat. Commun. **7**, 10988 (2016). [CrossRef]

**24. **R. Leijssen, G. R. L. Gala, L. Freisem, J. T. Muhonen, and E. Verhagen, “Nonlinear cavity optomechanics with nanomechanical thermal fluctuations,” Nat. Commun. **8**, 16024 (2017). [CrossRef]

**25. **J. D. Thompson, B. M. Zwickl, A. M. Jayich, F. Marquardt, S. M. Girvin, and J. G. E. Harris, “Strong dispersive coupling of a high-finesse cavity to a micromechanical membrane,” Nature **452**, 72–75 (2008). [CrossRef]

**26. **D. J. Wilson, C. A. Regal, S. B. Papp, and H. J. Kimble, “Cavity optomechanics with stoichiometric SiN films,” Phys. Rev. Lett. **103**, 207204 (2009). [CrossRef]

**27. **J. Cripe, N. Aggarwal, R. Lanza, A. Libson, R. Singh, P. Heu, D. Follman, G. D. Cole, N. Mavalvala, and T. Corbitt, “Measurement of quantum back action in the audio band at room temperature,” Nature **568**, 364–367 (2019). [CrossRef]

**28. **M. J. Yap, J. Cripe, G. L. Mansell, T. G. McRae, R. L. Ward, B. J. J. Slagmolen, P. Heu, D. Follman, G. D. Cole, T. Corbitt, and D. E. McClelland, “Broadband reduction of quantum radiation pressure noise via squeezed light injection,” Nat. Photonics **14**, 19–23 (2019). [CrossRef]

**29. **Y. Tsaturyan, A. Barg, E. S. Polzik, and A. Schliesser, “Ultracoherent nanomechanical resonators via soft clamping and dissipation dilution,” Nat. Nanotechnol. **12**, 776 (2017). [CrossRef]

**30. **C. Reetz, R. Fischer, G. Assumpção, D. McNally, P. Burns, J. Sankey, and C. Regal, “Analysis of membrane phononic crystals with wide band gaps and low-mass defects,” Phys. Rev. Appl. **12**, 044027 (2019). [CrossRef]

**31. **S. P. Vyatchanin and E. A. Zubova, “Quantum variation measurement of a force,” Phys. Lett. A **201**, 269–274 (1995). [CrossRef]

**32. **V. Sudhir, R. Schilling, S. Fedorov, H. Schütz, D. Wilson, and T. J. Kippenberg, “Quantum correlations of light from a room-temperature mechanical oscillator,” Phys. Rev. X **7**, 031055 (2017). [CrossRef]

**33. **N. Kampel, R. Peterson, R. Fischer, P.-L. Yu, K. Cicak, R. Simmonds, K. Lehnert, and C. Regal, “Improving broadband displacement detection with quantum correlations,” Phys. Rev. X **7**, 021008 (2017). [CrossRef]

**34. **A. H. Safavi-Naeini, S. Gröblacher, J. T. Hill, J. Chan, M. Aspelmeyer, and O. Painter, “Squeezed light from a silicon micromechanical resonator,” Nature **500**, 185–189 (2013). [CrossRef]

**35. **T. P. Purdy, P.-L. Yu, R. W. Peterson, N. S. Kampel, and C. A. Regal, “Strong optomechanical squeezing of light,” Phys. Rev. X **3**, 031012 (2013). [CrossRef]

**36. **C. Doolin, B. D. Hauer, P. H. Kim, A. J. R. MacDonald, H. Ramp, and J. P. Davis, “Nonlinear optomechanics in the stationary regime,” Phys. Rev. A **89**, 053838 (2014). [CrossRef]

**37. **We use two-sided spectral densities, denoted as ${{\rm S}_{\textit{xx}}}[\omega]$, in theoretical derivations and one-sided spectral densities, denoted as ${{\rm S}_x}[\omega] = {{2S}_{\textit{xx}}}[\omega]$ for $\omega \gt {0}$, for the presentation of experimental data.

**38. **C. W. Gardiner, *Handbook of Stochastic Methods*, 2nd ed. (Springer, 1985), Section 2.8.1.

**39. **V. Sudhir, D. Wilson, R. Schilling, H. Schütz, S. Fedorov, A. Ghadimi, A. Nunnenkamp, and T. J. Kippenberg, “Appearance and disappearance of quantum correlations in measurement-based feedback control of a mechanical oscillator,” Phys. Rev. X **7**, 011001 (2017). [CrossRef]

**40. **Y. Zhao, D. J. Wilson, K.-K. Ni, and H. J. Kimble, “Suppression of extraneous thermal noise in cavity optomechanics,” Opt. Express **20**, 3586–3612 (2012). [CrossRef]

**41. **D. J. Wilson, “Cavity optomechanics with high-stress silicon nitride films,” Ph.D. thesis (California Institute of Technology, 2012).

**42. **A. H. Ghadimi, S. A. Fedorov, N. J. Engelsen, M. J. Bereyhi, R. Schilling, D. J. Wilson, and T. J. Kippenberg, “Elastic strain engineering for ultralow mechanical dissipation,” Science **360**, 764–768 (2018). [CrossRef]

**43. **M. Rossi, D. Mason, J. Chen, Y. Tsaturyan, and A. Schliesser, “Measurement-based quantum control of mechanical motion,” Nature **563**, 53–58 (2018). [CrossRef]

**44. **I. Martin and W. H. Zurek, “Measurement of energy eigenstates by a slow detector,” Phys. Rev. Lett. **98**, 120401 (2007). [CrossRef]

**45. **A. A. Gangat, T. M. Stace, and G. J. Milburn, “Phonon number quantum jumps in an optomechanical system,” New J. Phys. **13**, 043024 (2011). [CrossRef]

**46. **V. B. Braginsky, F. Y. Khalili, and K. S. Thorne, *Quantum Measurement*, 1st ed. (Cambridge University, 1992).

**47. **A. A. Clerk, F. Marquardt, and J. G. E. Harris, “Quantum measurement of phonon shot noise,” Phys. Rev. Lett. **104**, 213603 (2010). [CrossRef]

**48. **A. Nunnenkamp, K. Børkje, J. G. E. Harris, and S. M. Girvin, “Cooling and squeezing via quadratic optomechanical coupling,” Phys. Rev. A **82**, 021806 (2010). [CrossRef]

**49. **T. K. Paraïso, M. Kalaee, L. Zang, H. Pfeifer, F. Marquardt, and O. Painter, “Position-squared coupling in a tunable photonic crystal optomechanical cavity,” Phys. Rev. X **5**, 041024 (2015). [CrossRef]

**50. **A. A. Clerk, “Quantum-limited position detection and amplification: a linear response perspective,” Phys. Rev. B **70**, 245306 (2004). [CrossRef]

**51. **M.-A. Lemonde, N. Didier, and A. A. Clerk, “Nonlinear interaction effects in a strongly driven optomechanical cavity,” Phys. Rev. Lett. **111**, 053602 (2013). [CrossRef]

**52. **A. B. Matsko and S. P. Vyatchanin, “Electromagnetic-continuum-induced nonlinearity,” Phys. Rev. A **97**, 053824 (2018). [CrossRef]

**53. **C. Reinhardt, T. Müller, A. Bourassa, and J. C. Sankey, “Ultralow-noise SiN trampoline resonators for sensing and optomechanics,” Phys. Rev. X **6**, 021001 (2016). [CrossRef]

**54. **S. Fedorov, A. Beccari, N. Engelsen, and T. J. Kippenberg, “Fractal-like mechanical resonators with a soft-clamped fundamental mode,” Phys. Rev. Lett. **124**, 025502 (2020). [CrossRef]

**55. **Raw measurement data, analysis code to reproduce the manuscript figures, and the designs of PnC membranes are available on zenodo.org, doi: 10.5281/zenodo.3747301.