## Abstract

The study of light propagation has been a cornerstone of progress in physics and technology. Recently, advances in control and shaping of light have created significant interest in the propagation of complex structures of light – particularly under realistic terrestrial conditions. While theoretical understanding of this research question has significantly grown over the last two decades, outdoor experiments with complex light structures are rare, and comparisons with theory have been nearly lacking. Such situations show a significant gap between theoretical models of atmospheric light behaviour and current experimental effort. Here, in an attempt to reduce this gap, we describe an interesting result of atmospheric models that are feasible for empirical observation. We analyze in detail light propagation in different spatial bases and present results of the theory that the influence of atmospheric turbulence is basis-dependent. Concretely, light propagating as eigenstate in one complete basis is more strongly influenced by atmosphere than light propagating in a different, complete basis. We obtain these results by exploiting a family of the continuously adjustable, complete basis of spatial modes—the Ince-Gauss modes. Our concrete numerical results will hopefully inspire experimental efforts and bring the theoretical and empirical study of complex light patterns in realistic scenarios closer together.

Published by The Optical Society under the terms of the Creative Commons Attribution 4.0 License. Further distribution of this work must maintain attribution to the author(s) and the published article's title, journal citation, and DOI.

## 1. Introduction

Complex, higher-order spatial modes of light have come under the spotlight due to its inherent orthogonality, and discrete infinite state spaces [1–3]. Its well-developed technology of generation [4–8] and manipulation [9–13] allows the increasing applications of free-space optical (FSO) communication links using spatial modes. The first experiment using spatial modes to encode information in an FSO channel was implemented by Gibson et al. in 2004 [14]. Since these early demonstrations, the transmission rate has been increased to 100 Tbit/sec [15,16]. However, realistic FSO links involve atmospheric turbulence which causes random fluctuations of the intensity and distorts the phase front of the transmitted light beam [17].

A natural question that arises is, "How does atmospheric turbulence influence complex spatially modulated beams of light?". The answer to this question would not only be practically useful for long-distance communication schemes but would also provide scientific insights into the interaction of light with realistic, turbulent air – and thus potentially allow for novel measurement techniques of atmospheric effects.

Due to the importance of the question, much effort has been made to study it. To simulate atmospheric turbulence and its effect on complex spatial modes in laboratories, researchers have exploited heat pipes [18,19], random phase screens generated by spatial light modulators (SLMs) [20–23], static diffractive plates [24] and rotating random phase plates [25]. Those efforts were based on models of the turbulent atmosphere that date back to the Kolmogorov’s seminal work from 1941 [26–28], and further extension and advances of it [29–31].

Kolmogorov’s models have since been applied to spatial modes of light (with single phase-screen approximations) [32,33]. Those early methods have been advanced by theories of multi-phase planes developed for complex spatial modes of light [34] (which have been used in beam propagation simulations already in the late 1980s [35,36]). These models allow for the understanding of spatial light propagation in strong scintillation conditions [37] and comparison of infinitesimal propagation with multi-phase screen propagation [38].

While the theoretical study of atmospheric turbulence effects of spatial modes of light is flourishing, experimental results in real outdoor conditions are lacking. Only in 2014, the first outdoor experiments with spatial modes were conducted, with long-distance transmissions up to 143 km [39,40], with high-speed data rates up to 400Gbit/sec [41,42], and in the quantum regime with entangled photons [40] and for quantum communication [43]. These results establish the feasibility of long-distance transmission of spatial modes of light but left open the question about the predictive power of current models. It was only in 2017 when Lavery et al. performed an experiment transmitting spatially modulated light in an urban environment and compared their results with theoretical models [44].

One reason for lack of experimental studies in real-world spatial mode propagation is the uncontrollable environment and the additional noise sources which do not exist in laboratories, such as background stray light, humidity and light absorption. Those are particularly hazardous for quantum experiments. Therefore, as steps to overcome the theoretical-experimental gap, experiments with small complexity are favourable.

Here we explain a detailed numerical study of higher-order spatial modes in different bases propagating through turbulent atmosphere, relying on the method of [44,45]. We find an at first sight effect – the basis-dependence of atmospheric influence, which could be detected with already existing optical setups. That means information transmitted in one basis (for instance, the famous Laguerre-Gauss (LG) basis carrying orbital angular momentum) is stronger degraded as if the information would have been encoded in another complete basis (such as the Hermite-Gauss (HG) basis). This is interesting to us, as those bases span the same space of modes, and each element of the first basis can be seen as a superposition of elements of the other basis. We find this effect by studying a continuous space of states, the so-called Ince-Gauss (IG) modes [46], which have both the LG as well as the HG basis as a special case. The predicted effect can be up to 7% of the total transmission quality; thus, it should be observable in already existing transmission links. If such an effect cannot be observed, it would raise serious questions about some of the best models to describe spatially modulated light in the atmosphere. If the effect is indeed physical, it will shine light on a curious property of the atmosphere and could indicate a novel technique to measure atmospheric properties.

Generally speaking, we have made several contributions in the manuscript: 1). For the first time, we detailed the investigation of Ince-Gaussian modes in turbulence (as a continuously tunable, discrete, complete orthogonal basis which has LG and HG as special cases). 2). As a result of the continuous tunable basis, we identify very concrete basis-dependent effects that have not been reported in the literature before. 3). We present an experimental implementation that will potentially allow for real-world investigations. Our numerically discovered effect could be measured using strong lasers and thus, does not require single-photon or entanglement investigations which are highly elaborate. 4). The number of numerical predictions, due to Ince-Gauss basis’s continuous parameter $\epsilon$ is large, and the number of fitting parameters are very small, which results in an ideal target for testing theoretical models accuracy and potential limits. 5). Motivated by the results presented from Lavery et al. [44,45], we provide potentially interesting experimental observations in the language that requires limited technical vocabulary perfectly suited for experimentalists.

The article is structured in the following way. We start by explaining what we mean by basis-dependent effect and why this is interesting – especially from a quantum mechanical standpoint in Sec. 2. Then we continue showing how it occurs using light propagation. Details about spatial modes of light are given in Sec. 3. In Sec. 4, we describe the atmospheric turbulence model used for our investigation. In Sec. 5, the numerical results and discussions are shown to illustrate the effect found in our simulation, and we explain a feasible experiment to test this prediction. The conclusions of the paper are given in Sec. 6.

## 2. Intuitive description of basis-dependence

First, we will give a simplified, intuitive explanation of what we call by basis-dependent effects. Our description will be independent of any atmospheric effects and purely motivated by quantum information considerations.

Let us consider a simple two-dimensional sub-basis in a large Hilbert space, which we use to encode information. Let’s assume these basis modes undergo a transformation which introduces unbiased loss (for example, scattering into other modes):

However, we could have chosen to encode the information in a different basis, such as the X-basis (with ${|\pm \rangle }=1/\sqrt {2}\left ({|0\rangle }\pm {|1\rangle }\right )$). Using Eq. (1), the transformation of the ${|\pm \rangle }$ leads to

*neutral*density filter acts in this way.

Now we give two simple examples of basis-dependence. Consider the transformation, which is a basis-dependent rotation, such as performed by a half-wave plate in polarization optics,

Let’s consider a third example, a mode-dependent loss.

We will see that the atmospheric influence on spatial modes is basis-dependent. As we perform a numerical study, we have to restrict ourselves to a subset of modes. We will see that the influence is mode-dependent when we consider a complete orthogonal *subspace* of modes with the same mode-order. This is particularly interesting, as the mode-order is defined independent of the basis, and a specific mode of one basis (with order $M$) can be decomposed in a superposition of modes of order $M$ in arbitrary other bases. Of course, the considerations here are vast simplifications of influence of the atmosphere on spatial modes, and can only be considered as a simple, intuitive picture of how to understand such effects in general.

## 3. Complex spatial modes of light

The well-known solutions to the paraxial wave equation consist of the HG and LG beams, which are derived from cartesian and circular cylindrical coordinates, respectively [47]. They both provide in principle an infinite state space and form a complete orthogonal basis, such that one can describe HG states in terms of LG modes and vice versa [48,49]. The HG modes are denoted as $HG_{n_{x},n_{y}}$ with indices $n_{x}$ and $n_{y}$ and LG modes are described as $LG_{n,\ell }$ with orbital angular momentum (OAM) index $\ell$ [50] and radial number $n$ [51–53].

In 2004, Bandres and Gutiérrez-Vega introduced another complete and orthogonal family of modes, which interpolates between the LG and HG modes [46]. These modes are exact solutions to the paraxial wave equation in elliptic coordinates, and of which the even IG modes are described as

The helical IG modes (which, for simplicity, we call IG modes for the rest of the paper) are described as superposition of even and odd IG modes [56,57], which are given by

When the elliptic coordinates tend to the circular cylindrical coordinates, namely $\epsilon \rightarrow 0$, the IG modes will be transferred into LG modes. In this case, the indices of states $LG_{n,\ell }$ and $IG_{p,m,\epsilon }$ are related as: $|\ell |=m$ and $n=(p-m)/2$. Additionally, when the elliptic coordinates tend to the cartesian coordinates, namely $\epsilon \rightarrow \infty$, the IG modes will be transferred into helical Hermite-Gaussian (HG) modes [56,58,59]. In this case, the indices of modes $HG_{n_{x},n_{y}}$ and $IG_{p,m,\epsilon }$ are related as: $n_{x}=m$ and $n_{y}=(p-m)$. This puts the IG modes in a special position between LG and HG, and therefore makes them an ideal workhorse for investigating basis-dependent effects.

For a fixed ellipticity $\epsilon$, the IG modes $IG_{p,m,\epsilon }$ with orthogonal mode indices $(p,m)\in \mathbb {N}$ form a complete basis. Changing the value of the ellipticity $\epsilon$ gives another complete orthogonal basis. In Fig. 1, we show the intensity and phase distributions of different IG modes $IG_{p,m,\epsilon }$ varying their ellipticity $\epsilon$.

There are also other well-investigated types of complete sets of transverse spatial modes. One example is the family of Bessel modes which posses intrinsic self-healing properties that could be of interest in out-door turbulence experiments [60], or their elliptic generalizations called Mathieu beams [61,62]. However, here we focus on the Ince-Gaussian basis.

## 4. Turbulence model

The inhomogeneity and anisotropy in the temperature and pressure of atmosphere results in random fluctuations of the refractive index along the propagation path of light beam [17]. Those variations of refractive index introduce the distortions of the spatially structured light fields and increase intermodal crosstalk, which dramatically affects the quality of spatial modes over long-distance link [63]. Thus understanding the turbulent behaviour of the atmosphere is very crucial.

In 1941, the Russian mathematician Andrey Kolmogorov published three seminal articles, which established the foundations of statistical turbulence theory [26–28]. The statistical properties of the atmospheric turbulence are homogeneous and isotropic within scale $L_0$ and $l_0$. He found that the random statistical behaviour of turbulence can be described by refractive index power density spectrum, which describes how the kinetic energy of atmospheric turbulence distributes with respect to frequencies. This directly relates to the phase fluctuations of light along propagation path and later is represented by the phase power density spectrum $\Phi _{\varphi }^{mvK}(\kappa )$ [17]. In our study, we apply the modified von Kármán spectrum [31], which avoids the singularity that represents energy per unit volume and becomes unbounded as the eddy size increases in Kolmogorov spectrum model. Therefore it is numerically more stable. The turbulent phase screens are generated using the modified von Kármán spectrum, which is described as [64,65]

There $r_{0}$ is the Fried parameter [66], which can be used to describe the strength of the atmospheric turbulence along the propagation distance. $\kappa$ is the angular spatial frequency. $\kappa _{m}$ and $\kappa _{0}$ are constant model-dependent parameters [17,64,65].

In order to simulate the atmospheric turbulence along a long propagation distance, we adopt a multi-phase plane model [34,45]. In the model shown in Fig. 2, the atmosphere along 1.5 km link is split into 150 planes of weakly turbulence separated by 10 m. The total atmospheric turbulence $r_{0}$ is approximately the accumulative effect of the turbulence of each plane $r_{0_{plane}}$. In our numerical simulation, we satisfy geometric and aliasing constraints [64,67,68].

#### 4.1 Turbulence strength and model

In the presence of a turbulent atmosphere, spatial modes of light experience atmospheric refractive index variations caused by fluctuations in temperature and pressure. These atmospheric refractive index variations distort the wavefronts of the propagated light beams [17]. A measure of the strength of random fluctuations is the refractive index structure parameter $C_{n}^2(z)$, which is used to quantify the strength of the atmospheric turbulence along the propagation path. Typical values of $C_{n}^2(z)$ range from $10^{-17}$ $m^{-2/3}$ in weak scintillation up to $10^{-13}$ $m^{-2/3}$ in strong turbulence [17,65].

Another parameter often used to estimate the integrated strength of turbulence, especially in connection with astronomical imaging, is the Fried parameter $r_{0}$ [69]. Stronger turbulence corresponds to a smaller $r_{0}$. For a known refractive index structure parameter $C_{n}^2(z)$ along the propagation path, the Fried parameter $r_{0}$ is given by [70–72]

There $k=2\pi /\lambda$ is the wavenumber and $\lambda$ is optical beam wavelength (in our simulations, a wavelength of $809nm$ is adopted). $\alpha _{1}=0.423$ is a constant number which derived in the case of the phase variance is approximately one [65,70]. The integral is taken over the overall propagation path from the transmitter plane to the receiver plane.

We rely on the experimentally demonstrated model from Lavery et al. [44,45] to simulate the atmosphere over long-distance. There the total 1.5 km turbulent link is decomposed into many short weakly perturbing planes separated by 10 m. In general, $C_{n}^2(z)$ is assumed to be roughly a constant over short time intervals or propagation distance. Using Eq. (8), we could approximately describe the overall strength of atmosphere $r_{0}$ as an accumulation of strength in every plane $r_{0_{plane}}$, which is described as

The values of different turbulence conditions used for studying the propagation of spatial modes of light are listed in Table 1.For the purpose of numerically modeling turbulence, we use the modified von Kármán phase power spectrum $\Phi _{\varphi }^{mvK}(\kappa )$ to generate random phase screens [31,65], which is described as

#### 4.2 Numerical simulation

In our simulation, we adopted a collimated light beam of beam waist $w_{0}=25mm$ and wavelength $\lambda =809nm$. All parameters used in our numerical simulation are presented in Table 2.

Then we investigate the propagation quality of IG modes $IG_{p,m,\epsilon }$ with different ellipticity under different turbulence conditions. The transmission fidelity describes the "closeness" of the received propagated light field ${|\Psi _{tur}\rangle }$ and the undisturbed light field in the observed planes ${|\Psi _{vac}\rangle}$, which is given by the squared overlapping the two light fields as

The average fidelity is computed by averaging all individual transmissions. For the average fidelity of certain mode order $M$, it is given by over all the modes in the same order and basis. Furthermore, we investigate our finding by analyzing the mode cross-talk matrix for complex spatial modes $IG_{p,m,\epsilon }$ with different ellipticity $\epsilon$. In the limit of $\epsilon \rightarrow 0$, a chosen basis with IG indices set $(p,m)$ can be rewritten as a LG indices $(n,\ell )$. Thus, in the cross-talk matrix labeled columns represent the modes with index $\ell$, meanwhile labeled rows represent the modes with index $n$. Each element in the matrix is given by an inner product measurement of the transmitted field ${|\Psi _{tur}\rangle }$ and each undisturbed mode (individually) from the set of basis ${|vac_{j}\rangle }$, which is given by## 5. Results and discussion

What is the quality of spatial modes of light propagating through atmospheric turbulence over long-distance? We start with transmitting an IG mode $IG_{4,0,\epsilon }$ over a 1.5 km turbulent channel. We call radial-like modes for IG states $IG_{p,m,\epsilon }$ when they are equivalent to radial modes $LG_{n,0}$ in the limit case of $\epsilon \rightarrow 0$. Analogously, we call OAM-like modes for IG states $IG_{p,m,\epsilon }$ when they are equivalent to OAM modes $LG_{0,\ell }$ in the limit case of $\epsilon \rightarrow 0$. Therefore, the IG mode $IG_{4,0,\epsilon }$ used in our simulation corresponds to a radial-like mode, with two intensity rings with ellipticity $\epsilon =0$.

Here a question naturally arises: What role does the ellipticity play on the transmission quality of IG modes under different turbulent conditions? For simplicity, we use two different ellipticity $\epsilon \rightarrow 0$ and $\epsilon =4$ and propagate the IG modes through the atmosphere of different turbulent strength $r_{0}$. The average fidelity of $IG_{4,0,\epsilon }$ through different turbulent strengths are described in Table 3 and Fig. 3 (a). There the fidelity is in percentage and the error is given by the standard deviation of the mean.

The result clearly shows the fact that under weak scintillation (equivalent to the case of large $r_{0}$), the quality is better than that under strong turbulence. Interestingly, we find that the ellipticity plays an interesting role in the transmission quality of IG modes along the turbulent path. To our surprise, there is a significant increase for the radial-like modes with a large ellipticity propagating through strong turbulence. In Fig. 3 (b), we show the fidelity difference versus various turbulent strengths. We find that there is a large fidelity difference when the turbulent strength is $r_{0}\approx 0.05$ m, which is a realistic turbulence condition (we use this $r_{0}$ for the rest of our simulations). With this observation, we would expect that the difference will continue to increase by enlarging the ellipticity of the radial-like mode $IG_{4,0,\epsilon }$. This indicates that helical HG modes, corresponding to radial-like modes in the case of $\epsilon \rightarrow \infty$, perform better under strong turbulence than that by LG radial modes.

The transmission quality for radial-like modes increases when we increase the ellipticity $\epsilon$. We would therefor expect that this increase is compensated by OAM-like modes whose quality decreases when we increase $\epsilon$, such that the transmission quality for a complete order of modes stay constant. However, the results in Table 4 and Fig. 4 show that the transmission quality of all other modes with order $M=4$ remains constant (within significant uncertainty). This means that the average propagation quality of order $M=4$ increases when we change to a basis with a large ellipticity. This means that the atmosphere influences bases with small $\epsilon$ in a stronger way than bases with large $\epsilon$. Every mode with order $M$ of a specific basis can be decomposed into a coherent sum of modes of order $M$ in another basis. For that reason, observing the effect in order $M=4$ is sufficient to conclude a basis-dependent effect. This is the basis-dependent effect of atmospheric turbulence we present here, and its investigation is the main results of our paper.

Furthermore, we find that this effect consistently exists for order $M=2$ up to $M=6$ (which contains 3 and 7 modes, respectively). For mode order $M=0$ and $M=1$, the ellipticity does not change the modes. Therefore, mode order $M=2$ is the smallest that we investigate here. We show the results in Table 5. An interesting insight into this effect is a cross-talk matrix of the radial-like mode $IG_{4,0,\epsilon }$ with small and large ellipticity. Indeed, in Fig. 5 we observe that small ellipticity leads to larger cross-talk with other modes of this basis, whereas larger ellipticity reduces the cross-talk. In addition, when we rotate the HG modes and $IG_{4,0,\epsilon }$ modes with higher $\epsilon$, such mode-dependence effect still exit, see Appendix A for more details. We have already explained an intuitive way of understanding this mode-dependence in Sec. 2, and a fully mathematical description of this effect would be exciting, but that is out of the scope of this manuscript.

There is now one important question that remains: Is this a physical effect or an artefact of a currently well-trusted model for the propagation of complex spatial modes? This question can only be solved by experiments. An experimental investigation would require a long-distance outdoor link for transmitting spatial modes of light, which already exists in several places worldwide. At the sender, one requires a high-quality construction of complex spatial modes, for instance, using the technique presented in [73]. At the receiver, the mode needs to be measured – for example by transforming it to a Gaussian mode with high-quality [74] and using a Single-Mode Fiber as a filter [75,76]. Atmospheric conditions (in particular the Fried parameter $r_0$) are stable for long enough to perform measurements in the form of Figs. 4 and 5 successfully. The experiment is well within reach of current technology. It could be one interesting target for detailed investigations of turbulence models using a complex, continuously class of complete, orthogonal spatial mode families. In particular, with merely one free fit-parameters $r_0$ (which could be measured by other means), the experiments can test many predictions, such as fidelities of different modes, such as shown in Figs. 4 and 5, and in particular, the important quantity of mode-dependence $\Delta F$ in Table 5 . Other tests could involve varying of beam radii (which we didn’t explicitly calculate), but which can be performed in the same setup with only adjusting the computer-generated holograms.

## 6. Related work

Several recent studies have compared properties between Hermite-Gauss modes and OAM modes (a subset of Laguerre-Gaussian modes) in terms of their classical communication quality. In 2016, Restuccia et al. have shown that in real-world communication, the finite-size apertures of sending and detection systems degrade the quality of the communication channel. The authors have shown that in such cases, Laguerre-Gaussian beams are more suitable than Hermite-Gauss modes [77]. We do not take finite-size effects of the measurement systems into account as we are interested in the fundamental physical properties of light propagation (and we are – for the moment – agnostic to technical implementations).

An alternative result has been obtained by the authors of [78], which has been recently experimentally simulated in the lab [79]. The authors show that a subset of Hermite-Gauss modes is less perturbed than doughnut modes (a subset of Laguerre-Gaussian modes with the radial mode index $n$=0) by single-phase screens for approximating turbulence effects. The effect stems from larger robustness against lateral translation because Hermite-Gauss modes are *azimuthally asymmetric*. Their work, which is essential for applications in classical communications (for instance, see [80,81]) differs in several ways from our findings: First, we investigate the full two-dimensional spatial basis of light, which is defined by two-mode numbers of Laguerre-Gaussian mode. This involves the azimuthal part (which the authors examine) and the radial part (which is not investigated, and which is known to have different physical properties, for instance, see [52]). Only the combination of all spatial mode numbers lets us make conclusions about the physical effect of the basis-dependent phenomena. Second, the results obtained by the authors are qualitatively similar (but quantitative different) compared to our findings, and it is not immediately clear why some asymmetric Ince-Gaussian modes (as shown by the orange and green line in Fig. 4) do not improve the fidelity when the asymmetry (ellipticity) is increased. Furthermore, we use an empirically suggested model of light propagation through turbulence because single-phase approximations cannot cover all experimentally observed effects [44]. We would argue that the best way to understand this effect is actually to perform the real-world experiment.

A different, fascinating recent work is conceptually related to our investigation: The authors investigate the propagation of doughnut modes (Laguerre-Gaussian beam with radial mode number $n$=0) through simulated turbulence. They present that for an OAM mode under realistic turbulence conditions, the spiral spectrum develops two maxima. One of the maxima is on the negative axis, and one is on the positive axis of the spectrum, while OAM=0 is very unlikely to appear [82]. It would be exciting to observe the predicted effect in a real-world experiment.

The mode-dependent influence of turbulence has been seen in theoretical calculations several times, for example, [32,83]. However, those studies did not investigate general full-field sets of modes (but mainly the subset of OAM modes) or didn’t make concrete predictions about the basis-dependence that only Ince-Gaussian modes allows.

## 7. Conclusion

We employ a state-of-the-art numerical simulation model and use it to investigate the propagation of very general spatial modes of light in realistic conditions. Therewith, we find the interesting effects that the atmosphere is basis-dependent. That means information encoded in one complete basis of spatially-modulated beams can be transmitted with different quality depending on which basis one uses. This is interesting as all complete spatial bases cover the same space, and modes of one basis can be expressed as a linear combination of modes in another basis. What is more, the influence of this effect can be up to 7% of the total transmission quality, which makes it suitable for experimental observation. To observe this result, we describe an experiment which is feasible with today’s technology and communication channels that have already been established in several locations worldwide.

Finally, in order to reduce the gap between theoretical understanding of atmospheric models and its empirical investigations, we would like to advocate a scientific program, which we call *the phenomenology of complex atmospheric light propagation*: Here, the goal is to distill new, empirically observable predictions of light propagation from the best existing atmospheric turbulence models. Those results will allow physicists to perform interesting real-world, outdoor experiments to observe novel, potentially unintuitive phenomena.

In particular phenomena that distinguish between different models, or show the effect of (the absents of) certain theoretical approximations could inspire real-world experimental campaigns. Especially proposals for the strong-light regime could be accessible to state-of-the-art technology.

An interesting question that follows from our study is “What set of mode is most robust against influence of atmospheric turbulence". The behavior of controlled propagation through scattering material [84] and eigenmodes of multi-mode fibers [85] are well understood. However, the time-dependence of turbulence (in the order of 1/100sec) makes it a challenging dynamic (and computational expensive) optimization problem.

Likewise, publically sharing of systematically recorded experimental outdoor data, which is not available at all today, could further help the modeling of real-world phenomena on a theoretical level. For the sharing of data, well-established scientific data-sharing platforms such as *datadryad* or scientific data journals such as *Nature: Scientific Data* are adequate resources [86].

We believe that this *program* could lead to flourishing experimental endeavours, which – apart from its pure scientific purpose – might have an impact in practical questions such as classical and quantum communication, or potentially novel methods to measure properties of atmosphere and thereby weather dynamics.

## Appendix

## A. Additional details on numerical simulation

We show some values of the mode cross-talk matrix of IG modes $IG_{4,0,\epsilon }$ in Tables 6 and 7. We can see that the mode cross-talk becomes less for radial-like modes by increasing $\epsilon$ and the fidelity of the propagated mode (highlight in green) is significantly larger than that with small $\epsilon$, as described in the main text in Fig. 5.

We show each mode fidelity for HG modes in different order $M$ in Table 8. We can clearly see that when one of the index of HG mode is zero, the mode performs better than other modes in the same order. The average fidelity for each modes are over 2000 iterations of propagations. The unit for fidelity is percentage and the error is described by standard deviation of the mean.

In order to verify the mode-dependent effect described in the article is not a numerical artefact, we perform sanity checks involving the analyzation of the normalization, decomposition and orthogonality of different spatial modes in different basis and particularly perform a calculation of pure LG modes comparing to pure HG modes for the same order $M=2$ with a even large grid number N=2048 in Table 9. There the average fidelity is over all the modes from the same basis in the same order, such as order $M=2$ contains 3 different modes (each modes with more than 3000 iterations). We see those values are within the statistical uncertainty to the values in the main text and show the effect described in the article. Our chosen discrete parameter fulfills the geometrical and aliasing constraints for numerical beam propagation [64,67,68]. Therefore we are confident that the results presented in the paper are due to the model instead of the numerical inaccuracy.

Furthermore, we rotate the HG modes and IG modes with higher ellipticity $\epsilon =4$ by the angle $\phi =45$ degree and propagate these modes through atmospheric turbulence. Then we compare with the LG modes and un-rotated HG or IG modes to further indicate our result is a basis-dependence effect. There we only consider the modes in the same order $M=2$ and the fidelity is over all the modes in the same order from one basis. Due to the expensive computation resource, we only calculate around 1400 samples for rotated HG modes with N=1024 and 1000 samples for rotated IG modes with N=2048 (for reducing the numerical error introduced by rotating matrix) . The results are described in Table 10 and 11. Interestingly, our results confirm that the HG and IG modes with higher ellipticity perform better than the LG modes through turbulence, and these values are within the statistical uncertainty. Our results indicate that rotating these HG and IG modes has no influences to our presented results in the main text, which further predicts the basis-dependence effect is not a numerical artefact but a real result of the model.

## Funding

National Key Research and Development Program of China (2017YFA0303700); National Natural Science Foundation of China (11690030, 11690032, 61771236); China Scholarship Council; Österreichischen Akademie der Wissenschaften; Austrian Science Fund (SFB F40 (FOQUS), Erwin Schrödinger fellowship No. J4309).

## Acknowledgments

The authors thank Mitchell A. Cox, Giovanni Milione, Giacomo Sorelli, Anton Zeilinger and Johannes Handsteiner for useful discussions and valuable comments on the manuscript. The authors thank Dominik Leitner for providing computational resources.

## Disclosures

The authors declare no conflicts of interest.

## References

**1. **A. M. Yao and M. J. Padgett, “Orbital angular momentum: origins, behavior and applications,” Adv. Opt. Photonics **3**(2), 161–204 (2011). [CrossRef]

**2. **M. J. Padgett, “Orbital angular momentum 25 years on,” Opt. Express **25**(10), 11265–11274 (2017). [CrossRef]

**3. **M. Erhard, R. Fickler, M. Krenn, and A. Zeilinger, “Twisted photons: new quantum perspectives in high dimensions,” Light: Sci. Appl. **7**(3), 17146 (2018). [CrossRef]

**4. **M. Beijersbergen, R. Coerwinkel, M. Kristensen, and J. Woerdman, “Helical-wavefront laser beams produced with a spiral phaseplate,” Opt. Commun. **112**(5-6), 321–327 (1994). [CrossRef]

**5. **N. Heckenberg, R. McDuff, C. Smith, and A. White, “Generation of optical phase singularities by computer-generated holograms,” Opt. Lett. **17**(3), 221–223 (1992). [CrossRef]

**6. **A. Vijayakumar, C. Rosales-Guzmán, M. Rai, J. Rosen, O. Minin, I. Minin, and A. Forbes, “Generation of structured light by multilevel orbital angular momentum holograms,” Opt. Express **27**(5), 6459–6470 (2019). [CrossRef]

**7. **L. Marrucci, C. Manzo, and D. Paparo, “Optical spin-to-orbital angular momentum conversion in inhomogeneous anisotropic media,” Phys. Rev. Lett. **96**(16), 163905 (2006). [CrossRef]

**8. **M. Mirhosseini, O. S. Magana-Loaiza, C. Chen, B. Rodenburg, M. Malik, and R. W. Boyd, “Rapid generation of light beams carrying orbital angular momentum,” Opt. Express **21**(25), 30196–30203 (2013). [CrossRef]

**9. **J. Leach, M. J. Padgett, S. M. Barnett, S. Franke-Arnold, and J. Courtial, “Measuring the orbital angular momentum of a single photon,” Phys. Rev. Lett. **88**(25), 257901 (2002). [CrossRef]

**10. **G. C. Berkhout, M. P. Lavery, J. Courtial, M. W. Beijersbergen, and M. J. Padgett, “Efficient sorting of orbital angular momentum states of light,” Phys. Rev. Lett. **105**(15), 153601 (2010). [CrossRef]

**11. **M. P. Lavery, D. J. Robertson, G. C. Berkhout, G. D. Love, M. J. Padgett, and J. Courtial, “Refractive elements for the measurement of the orbital angular momentum of a single photon,” Opt. Express **20**(3), 2110–2115 (2012). [CrossRef]

**12. **Y. Zhou, M. Mirhosseini, D. Fu, J. Zhao, S. M. H. Rafsanjani, A. E. Willner, and R. W. Boyd, “Sorting photons by radial quantum number,” Phys. Rev. Lett. **119**(26), 263602 (2017). [CrossRef]

**13. **X. Gu, M. Krenn, M. Erhard, and A. Zeilinger, “Gouy phase radial mode sorter for light: Concepts and experiments,” Phys. Rev. Lett. **120**(10), 103601 (2018). [CrossRef]

**14. **G. Gibson, J. Courtial, M. J. Padgett, M. Vasnetsov, V. Pas’ko, S. M. Barnett, and S. Franke-Arnold, “Free-space information transfer using light beams carrying orbital angular momentum,” Opt. Express **12**(22), 5448–5456 (2004). [CrossRef]

**15. **J. Wang, J.-Y. Yang, I. M. Fazal, N. Ahmed, Y. Yan, H. Huang, Y. Ren, Y. Yue, S. Dolinar, M. Tur, and A. E. Willner, “Terabit free-space data transmission employing orbital angular momentum multiplexing,” Nat. Photonics **6**(7), 488–496 (2012). [CrossRef]

**16. **H. Huang, G. Xie, Y. Yan, N. Ahmed, Y. Ren, Y. Yue, D. Rogawski, M. J. Willner, B. I. Erkmen, K. M. Birnbaum, S. J. Dolinar, M. P. J. Lavery, M. J. Padgett, M. Tur, and A. E. Willner, “100 tbit/s free-space data link enabled by three-dimensional multiplexing of orbital angular momentum, polarization, and wavelength,” Opt. Lett. **39**(2), 197–200 (2014). [CrossRef]

**17. **L. C. Andrews and R. L. Phillips, * Laser beam propagation through random media, vol. 152* (SPIE press Bellingham, WA, 2005).

**18. **B.-J. Pors, C. Monken, E. R. Eliel, and J. Woerdman, “Transport of orbital-angular-momentum entanglement through a turbulent atmosphere,” Opt. Express **19**(7), 6671–6683 (2011). [CrossRef]

**19. **A. Chaibi, C. Mafusire, and A. Forbes, “Propagation of orbital angular momentum carrying beams through a perturbing medium,” J. Opt. **15**(10), 105706 (2013). [CrossRef]

**20. **B. Rodenburg, M. P. Lavery, M. Malik, M. N. O’Sullivan, M. Mirhosseini, D. J. Robertson, M. Padgett, and R. W. Boyd, “Influence of atmospheric turbulence on states of light carrying orbital angular momentum,” Opt. Lett. **37**(17), 3735–3737 (2012). [CrossRef]

**21. **M. Malik, M. O’Sullivan, B. Rodenburg, M. Mirhosseini, J. Leach, M. P. Lavery, M. J. Padgett, and R. W. Boyd, “Influence of atmospheric turbulence on optical communications using orbital angular momentum for encoding,” Opt. Express **20**(12), 13195–13200 (2012). [CrossRef]

**22. **A. H. Ibrahim, F. S. Roux, M. McLaren, T. Konrad, and A. Forbes, “Orbital-angular-momentum entanglement in turbulence,” Phys. Rev. A **88**(1), 012312 (2013). [CrossRef]

**23. **Y. Zhang, S. Prabhakar, A. H. Ibrahim, F. S. Roux, A. Forbes, and T. Konrad, “Experimentally observed decay of high-dimensional entanglement through turbulence,” Phys. Rev. A **94**(3), 032310 (2016). [CrossRef]

**24. **A. Trichili, A. B. Salem, A. Dudley, M. Zghal, and A. Forbes, “Encoding information using laguerre gaussian modes over free space turbulence media,” Opt. Lett. **41**(13), 3086–3089 (2016). [CrossRef]

**25. **Y. Ren, H. Huang, G. Xie, N. Ahmed, Y. Yan, B. I. Erkmen, N. Chandrasekaran, M. P. Lavery, N. K. Steinhoff, M. Tur, S. Dolinar, M. Neifeld, M. J. Padgett, R. W. Boyd, J. H. Shapiro, and A. E. Willner, “Atmospheric turbulence effects on the performance of a free space optical link employing orbital angular momentum multiplexing,” Opt. Lett. **38**(20), 4062–4065 (2013). [CrossRef]

**26. **A. N. Kolmogorov, “The local structure of turbulence in incompressible viscous fluid for very large reynolds numbers,” Proc. R. Soc. Lond. A **434**(1890), 9–13 (1991). [CrossRef]

**27. **A. N. Kolmogorov, *On degeneration (decay) of isotropic turbulence in an incompressible viscous liquid*, vol. 31 (1941).

**28. **A. N. Kolmogorov, *Dissipation of energy in locally isotropic turbulence*, vol. 32 (1941).

**29. **A. Oboukhov, “Structure of the temperature field in turbulent flows,” Isv. Geogr. Geophys. Ser. **13**, 58–69 (1949).

**30. **S. Corrsin, “On the spectrum of isotropic temperature fluctuations in an isotropic turbulence,” J. Appl. Phys. **22**(4), 469–473 (1951). [CrossRef]

**31. **T. Von Karman, “Progress in the statistical theory of turbulence,” Proc. Natl. Acad. Sci. U. S. A. **34**(11), 530–539 (1948). [CrossRef]

**32. **C. Paterson, “Atmospheric turbulence and orbital angular momentum of single photons for optical communication,” Phys. Rev. Lett. **94**(15), 153901 (2005). [CrossRef]

**33. **G. A. Tyler and R. W. Boyd, “Influence of atmospheric turbulence on the propagation of quantum states of light carrying orbital angular momentum,” Opt. Lett. **34**(2), 142–144 (2009). [CrossRef]

**34. **F. S. Roux, T. Wellens, and V. N. Shatokhin, “Entanglement evolution of twisted photons in strong atmospheric turbulence,” Phys. Rev. A **92**(1), 012326 (2015). [CrossRef]

**35. **J. Martin and S. M. Flatté, “Intensity images and statistics from numerical simulation of wave propagation in 3-d random media,” Appl. Opt. **27**(11), 2111–2126 (1988). [CrossRef]

**36. **J. Martin and S. M. Flatté, “Simulation of point-source scintillation through three-dimensional random media,” J. Opt. Soc. Am. A **7**(5), 838–847 (1990). [CrossRef]

**37. **C. M. Mabena and F. S. Roux, “Optical orbital angular momentum under strong scintillation,” Phys. Rev. A **99**(1), 013828 (2019). [CrossRef]

**38. **C. M. Mabena, T. Bell, and F. S. Roux, “Corroboration of a multi-phase screen model,” Free. Laser Commun. XXXI **10910**, 109101T (2019). [CrossRef]

**39. **M. Krenn, R. Fickler, M. Fink, J. Handsteiner, M. Malik, T. Scheidl, R. Ursin, and A. Zeilinger, “Communication with spatially modulated light through turbulent air across vienna,” New J. Phys. **16**(11), 113028 (2014). [CrossRef]

**40. **M. Krenn, J. Handsteiner, M. Fink, R. Fickler, R. Ursin, M. Malik, and A. Zeilinger, “Twisted light transmission over 143 km,” Proc. Natl. Acad. Sci. **113**(48), 13648–13653 (2016). [CrossRef]

**41. **Y. Ren, Z. Wang, P. Liao, L. Li, G. Xie, H. Huang, Z. Zhao, Y. Yan, N. Ahmed, A. Willner, M. P. J. Lavery, N. Ashrafi, S. Ashrafi, R. Bock, M. Tur, I. B. Djordjevic, M. A. Neifeld, and A. E. Willner, “Experimental characterization of a 400 gbit/s orbital angular momentum multiplexed free-space optical link over 120 m,” Opt. Lett. **41**(3), 622–625 (2016). [CrossRef]

**42. **L. Li, R. Zhang, Z. Zhao, G. Xie, P. Liao, K. Pang, H. Song, C. Liu, Y. Ren, G. Labroille, P. Jian, D. Starodubov, B. Lynn, R. Bock, M. Tur, and A. E. Willner, “High-capacity free-space optical communications between a ground transmitter and a ground receiver via a uav using multiplexing of multiple orbital-angular-momentum beams,” Sci. Rep. **7**(1), 17427 (2017). [CrossRef]

**43. **A. Sit, F. Bouchard, R. Fickler, J. Gagnon-Bischoff, H. Larocque, K. Heshami, D. Elser, C. Peuntinger, K. Günthner, B. Heim, C. Marquardt, G. Leuchs, R. W. Boyd, and E. Karimi, “High-dimensional intracity quantum cryptography with structured photons,” Optica **4**(9), 1006–1010 (2017). [CrossRef]

**44. **M. P. Lavery, C. Peuntinger, K. Günthner, P. Banzer, D. Elser, R. W. Boyd, M. J. Padgett, C. Marquardt, and G. Leuchs, “Free-space propagation of high-dimensional structured optical fields in an urban environment,” Sci. Adv. **3**(10), e1700552 (2017). [CrossRef]

**45. **M. P. Lavery, “Vortex instability in turbulent free-space propagation,” New J. Phys. **20**(4), 043023 (2018). [CrossRef]

**46. **M. A. Bandres and J. C. Gutiérrez-Vega, “Ince–gaussian beams,” Opt. Lett. **29**(2), 144–146 (2004). [CrossRef]

**47. **A. E. Siegman, “Lasers. 1986,” P492 P **1017** (1986).

**48. **I. Kimel and L. R. Elias, “Relations between hermite and laguerre gaussian modes,” IEEE J. Quantum Electron. **29**(9), 2562–2567 (1993). [CrossRef]

**49. **M. W. Beijersbergen, L. Allen, H. Van der Veen, and J. Woerdman, “Astigmatic laser mode converters and transfer of orbital angular momentum,” Opt. Commun. **96**(1-3), 123–132 (1993). [CrossRef]

**50. **L. Allen, M. W. Beijersbergen, R. Spreeuw, and J. Woerdman, “Orbital angular momentum of light and the transformation of laguerre-gaussian laser modes,” Phys. Rev. A **45**(11), 8185–8189 (1992). [CrossRef]

**51. **E. Karimi and E. Santamato, “Radial coherent and intelligent states of paraxial wave equation,” Opt. Lett. **37**(13), 2484–2486 (2012). [CrossRef]

**52. **E. Karimi, R. Boyd, P. De La Hoz, H. De Guise, J. Řeháček, Z. Hradil, A. Aiello, G. Leuchs, and L. L. Sánchez-Soto, “Radial quantum number of laguerre-gauss modes,” Phys. Rev. A **89**(6), 063813 (2014). [CrossRef]

**53. **W. N. Plick and M. Krenn, “Physical meaning of the radial index of laguerre-gauss beams,” Phys. Rev. A **92**(6), 063841 (2015). [CrossRef]

**54. **F. M. Arscott, * Periodic differential equations: an introduction to Mathieu, Lamé, and allied functions*, vol. 66 (Elsevier, 2014).

**55. **M. A. Bandres, “Ince gaussian beam,” https://www.mathworks.com/matlabcentral/fileexchange/46222-ince-gaussian-beam (2014).

**56. **M. A. Bandres and J. C. Gutiérrez-Vega, “Ince–gaussian modes of the paraxial wave equation and stable resonators,” J. Opt. Soc. Am. A **21**(5), 873–880 (2004). [CrossRef]

**57. **J. B. Bentley, J. A. Davis, M. A. Bandres, and J. C. Gutiérrez-Vega, “Generation of helical ince-gaussian beams with a liquid-crystal display,” Opt. Lett. **31**(5), 649–651 (2006). [CrossRef]

**58. **C. López-Mariscal and J. C. Gutiérrez-Vega, “Propagation dynamics of helical hermite-gaussian beams,” Proc. SPIE **6663**, 666307 (2007). [CrossRef]

**59. **W. N. Plick, M. Krenn, R. Fickler, S. Ramelow, and A. Zeilinger, “Quantum orbital angular momentum of elliptically symmetric light,” Phys. Rev. A **87**(3), 033806 (2013). [CrossRef]

**60. **M. McLaren, T. Mhlanga, M. J. Padgett, F. S. Roux, and A. Forbes, “Self-healing of quantum entanglement after an obstruction,” Nat. Commun. **5**(1), 3248 (2014). [CrossRef]

**61. **J. C. Gutiérrez-Vega, M. Iturbe-Castillo, and S. Chávez-Cerda, “Alternative formulation for invariant optical fields: Mathieu beams,” Opt. Lett. **25**(20), 1493–1495 (2000). [CrossRef]

**62. **J. Gutiérrez-Vega, M. Iturbe-Castillo, G. Ramırez, E. Tepichın, R. Rodrıguez-Dagnino, S. Chávez-Cerda, and G. New, “Experimental demonstration of optical mathieu beams,” Opt. Commun. **195**(1-4), 35–40 (2001). [CrossRef]

**63. **S. Li, S. Chen, C. Gao, A. E. Willner, and J. Wang, “Atmospheric turbulence compensation in orbital angular momentum communications: Advances and perspectives,” Opt. Commun. **408**, 68–81 (2018). [CrossRef]

**64. **J. D. Schmidt, * Numerical simulation of optical wave propagation with examples in MATLAB* (SPIE Bellingham, Washington, USA, 2010).

**65. **J. Osborn, * Profiling the turbulent atmosphere and novel correction techniques for imaging and photometry in astronomy* (Durham University, 2010).

**66. **D. L. Fried, “Statistics of a geometric representation of wavefront distortion,” J. Opt. Soc. Am. **55**(11), 1427–1435 (1965). [CrossRef]

**67. **S. Coy, * Choosing mesh spacings and mesh dimensions for wave optics simulation*, vol. 5894 (International Society for Optics and Photonics, 2005).

**68. **Q. Tang, J. Wang, P. Lv, and Q. Sun, * A propagation method with adaptive mesh grid based on wave characteristics for wave optics simulation*, vol. 9674 (International Society for Optics and Photonics, 2015).

**69. **D. L. Fried, “Optical resolution through a randomly inhomogeneous medium for very long and very short exposures,” J. Opt. Soc. Am. **56**(10), 1372–1379 (1966). [CrossRef]

**70. **F. Roddier, * V the effects of atmospheric turbulence in optical astronomy, vol. 19* (Elsevier, 1981).

**71. **R. J. Sasiela, * Electromagnetic wave propagation in turbulence: evaluation and application of Mellin transforms* (SPIE, 2007).

**72. **T. Schmitt-Manderbach, * Long distance free-space quantum key distribution* (Universitat Munchen, 2007).

**73. **E. Bolduc, N. Bent, E. Santamato, E. Karimi, and R. W. Boyd, “Exact solution to simultaneous intensity and phase encryption with a single phase-only hologram,” Opt. Lett. **38**(18), 3546–3549 (2013). [CrossRef]

**74. **F. Bouchard, N. H. Valencia, F. Brandt, R. Fickler, M. Huber, and M. Malik, “Measuring azimuthal and radial modes of photons,” Opt. Express **26**(24), 31925–31941 (2018). [CrossRef]

**75. **A. Mair, A. Vaziri, G. Weihs, and A. Zeilinger, “Entanglement of the orbital angular momentum states of photons,” Nature **412**(6844), 313–316 (2001). [CrossRef]

**76. **M. Krenn, R. Fickler, M. Huber, R. Lapkiewicz, W. Plick, S. Ramelow, and A. Zeilinger, “Entangled singularity patterns of photons in ince-gauss modes,” Phys. Rev. A **87**(1), 012326 (2013). [CrossRef]

**77. **S. Restuccia, D. Giovannini, G. Gibson, and M. Padgett, “Comparing the information capacity of laguerre–gaussian and hermite–gaussian modal sets in a finite-aperture system,” Opt. Express **24**(24), 27127–27136 (2016). [CrossRef]

**78. **B. Ndagano, N. Mphuthi, G. Milione, and A. Forbes, “Comparing mode-crosstalk and mode-dependent loss of laterally displaced orbital angular momentum and hermite–gaussian modes for free-space optical communication,” Opt. Lett. **42**(20), 4175–4178 (2017). [CrossRef]

**79. **M. A. Cox, L. Maqondo, R. Kara, G. Milione, L. Cheng, and A. Forbes, “The resilience of hermite-and laguerre-gaussian modes in turbulence,” J. Lightwave Technol. **37**(16), 3911–3917 (2019). [CrossRef]

**80. **G. Milione, B. Ndagano, N. Mphuthi, and A. Forbes, * Using Hermite-Gaussian modes for free-space optical communication (Conference Presentation*), vol. 10549 (International Society for Optics and Photonics, 2018).

**81. **G. Milione, “Free-space optical communication using hermite-gaussian modes,” Google Patents (2019). US Patent App. 16/048,155.

**82. **G. Sorelli, N. Leonhard, V. Shatokhin, C. Reinlein, and A. Buchleitner, “Entanglement protection of high-dimensional states by adaptive optics,” New J. Phys. **21**(2), 023003 (2019). [CrossRef]

**83. **B. J. Smith and M. Raymer, “Two-photon wave mechanics,” Phys. Rev. A **74**(6), 062104 (2006). [CrossRef]

**84. **S. Rotter and S. Gigan, “Light fields in complex media: Mesoscopic scattering meets wave control,” Rev. Mod. Phys. **89**(1), 015005 (2017). [CrossRef]

**85. **J. Carpenter, B. J. Eggleton, and J. Schröder, “Comparison of principal modes and spatial eigenmodes in multimode optical fibre,” Laser Photonics Rev. **11**(1), 1600259 (2017). [CrossRef]

**86. **L. Candela, D. Castelli, P. Manghi, and A. Tani, “Data journals: A survey,” J. Assn. Inf. Sci. Tec. **66**(9), 1747–1762 (2015). [CrossRef]