## Abstract

In a free space optical communication system based on vortex beams, the effects of spread and crosstalk caused by atmospheric turbulence should not be ignored. The orbital angular momentum (OAM) spectrum of the signal based on elliptic Gaussian beam (EGB) after propagation through non-Kolmogorov turbulent atmosphere are deduced, and a theoretical model of the spiral spectrum of EGB propagating through turbulent atmosphere is obtained. Numerically calculated OAM modes detection and crosstalk probability under different ellipticity parameters. The results show that the ellipticity parameter has a significant impact on the OAM spectral distribution of EGB and the transmission characteristics after turbulent atmosphere. The selection of appropriate ellipticity parameter can correspondingly reduce the degradation and crosstalk caused by turbulent atmosphere. We also compared a Laguerre-Gaussian beam (LGB) with EGB and pointed out the advantages and limitations of these two kinds of beams. The research results may be useful in the field of short distance optical communication and OAM-based multiplex communication.

© 2021 Optical Society of America under the terms of the OSA Open Access Publishing Agreement

## 1. Introduction

The vortex optical field with spiral phase factor exp(*inφ*) was proved by Allen in 1992 to carry orbital angular momentum (OAM), where *n* is an arbitrary integer or fraction, represents the topological charge, and *φ* is the azimuthal coordinate. Each photon in a vortex beam carries *lħ* OAM (*ħ* is Planck's constant divided by 2*π*) [1]. As a new degree of freedom, OAM has been widely used in optical tweezers [2,3], large capacity data transmission and exchange [4,5], remote material imaging [6], high performance optical communication in free space and turbulent environment and [7–12] so on. Owing to the orthogonality and completeness of OAM, it has been proved that OAM has infinite dimensional Hilbert space and can carry infinite dimensional information. In optical communication system, a mode-division multiplexing technique has been introduced which means that OAM beams can expand the capacity of optical communications [13–17].

In a free-space communication system based on OAM, the atmosphere turbulence effects cannot be ignored. The wavefront and OAM mode of vortex optical field will be affected by turbulence environment, resulting in distortion and crosstalk, and increase the bit error rate. This is because the launched OAM mode of the optical field will degenerate and redistributed into its adjacent modes [18–21]. In previous studies, the spiral spectrum and OAM crosstalk characteristics of some vortex optical fields in turbulent atmosphere have been reported, such as LGB [22–24], Bessel-Gaussian beam (BGB) [25–27], Hypergeometric-Gaussian beam [28], Whittaker-Gaussian beam [29], Airy beam [30], etc. It is worth noting that the non-diffraction beam represented by Airy beam has new characteristics such as self-focusing, self-recovery, and self-bending. It has unique advantages and broad prospects in the field of optical communication in free space and turbulent environment.

The concept of EGB was proposed by Kotlyar *et al*. in 2017, who embedded an elliptical optical vortex into Gaussian beam, pointing out that EGB is the finite sum of LGB with different topological charges. It is also discussed that the major axis of the intensity ellipse rotates by 90 degrees during the propagation of EGB from the initial plane to the focal plane of a spherical lens [31]. In earlier studies, Dennis described an optical vortex (phase singularity) with a high topological strength resides on the axis of a high-order beam that splits into a straight row of unit-strength vortices under elliptic perturbation [32]. Kotlyar derived and analyzed the analytical expression for the diffraction of an elliptic Laguerre-Gaussian beam (ELGB), and showed that a beam with even singularity order have nonzero axial intensity for any ellipticity and at any finite distance *z* from the initial plane, whereas *z*=0 and *z*=∞ the axial intensity is zero [33]. In recent years, some complicated elliptical Gaussian beams with fractional order OAM have also been studied, including Ince-Gaussian beams (IGB) [34], Hermite-Laguerre-Gaussian beams (HLGB) [35], Lommel modes [36]. However, the EGB considered here is different from the complex elliptical vortex laser mode mentioned above. The main difference is that the characteristics of EGB is closest to those of noncanonical vortices [37]. Nonetheless, only the density distribution and evolution of OAM in the propagation process are given in Ref. [31]. In general, LGB are used as the OAM beams in free-space optical communication system, while EGB is a finite superposition of LGB. As the representative of the noncanonical vortex, the spiral spectrum distribution of EGB presents a multiplex OAM states. High-order EGB has more channels than low-order EGB and LGB, and the capacity is also greatly increased. Thus, there are greater application prospects in communication systems. However, as far as we know, there is no research on the OAM spiral spectrum and crosstalk characteristics of this type of noncanonical vortex elliptic Gaussian beam when propagating in turbulent atmosphere.

This paper is organized as follows: In Section 2, the theoretical model of the OAM spectrum calculation of EGB in non-Kolmogorov turbulence are derived based on the Rytov approximation and Huygens-Fresnel diffraction integral; In Section 3, the effects of turbulence parameters and beam parameters on mode and crosstalk probability are discussed in detail; In Section 4, we compare and analyze EGB and LGB; In Section 5, compare the transmission performance of EGB from 3-order to 7-order; In Section 6, we give the optimal value of the ellipticity parameter. Section 7 presents the conclusion.

## 2. Theoretical model

In the cylindrical coordinate system, the field distribution of the EGB in the source plane can be written as follows [31]:

*r*

_{0}and

*θ*

_{0}are the radial and azimuthal coordinates.

*a*is a dimensionless parameter that defines the ellipticity of the beam.

*n*is an integer and is the topological charge of the EGB.

*w*is the waist width of the Gaussian beam. If

*a*<0 here, the spiral phase rotates clockwise, if

*a*>0 - anticlockwise. When

*a*=±1, the beam degenerates into a doughnut beam. Therefore, the sign of

*a*determines the direction of the spiral phase rotation and directly affects the distribution of OAM states.

Figure 1 shows the intensity and phase distribution of EGB at the source plane for different values of ellipticity parameter *a*. The calculation parameters are *n*=3, *w*=0.05 m. It can be observed from Fig. 1 that if *n* is given, *a*>0 and the increase of *a* leads to enhanced phase noncanonical. When *a*<0, the phase rotation direction of the beam changes from anticlockwise to clockwise. Therefore, considering the positive and negative values of *a* will lead to a change in the phase, which in turn will lead to a change in the distribution of the OAM mode.

Using the Euler's formula [38]:

and Newton's binomial expansion:The field distribution of the EGB at the source plane in the cylindrical coordinate system can be obtained as

It can be observed from Eq. (4) that when *n*≥1, the EGB is a multiplexed OAM beam, and the number of OAM states is *n*+1. Therefore, as *n* increases, the EGB will have more OAM states, that is, more channels.

According to the Huygens-Fresnel diffraction integral formula, the field of the EGB propagating in free space is [39]

*z*,

*λ*is the wavelength of the beam and the wave number

*k*=2

*π*/

*λ*. Substituting Eq. (4) into Eq. (5) and using the following formula [38],

*J*(·) denotes the Bessel function, and

_{n}_{1}

*F*

_{1}(

*a*;

*b*;

*x*) denotes the confluent hypergeometric function. Through the integration operation, the field of the EGB propagating in free space is

According to the discussion in Ref. [6,40], to elucidate the OAM content, or spiral spectrum, of a field distribution *E (r, θ, z)* one has to compute its projection into the spiral harmonics exp(*imθ*). Thus, we can write the function *E* (*r*, *θ*, *z*) as a superposition of the spiral harmonics exp(*imθ*)

*a*(

_{m}*r*,

*z*) is given by the integral

*a*(

_{m}*r*,

*z*) takes this form

*T*(

*α*,

*z*) is the physical quantity describing the intensity of turbulence, Φ

*(*

_{n}*κ*,

*α*) is the spatial power spectrum of the refractive index in the turbulent medium,

*α*is the power spectrum index and 3<

*α*<4. Since the statistical characteristics of atmospheric turbulence follow the more general non-Kolmogorov power spectrum, and when the beam travels along a vertical path, atmospheric turbulence exhibits strong non-Kolmogorov characteristics, this paper uses the non-Kolmogorov power spectrum to simulate atmospheric turbulence, and

*T*(

*α*,

*z*) is expressed as [42]

*l*

_{0}and

*L*

_{0}are the inner and outer scales of atmospheric turbulence, respectively. $\widetilde C_n^2$ is the generalized structural parameter of turbulence.

According to Eq. (10)–Eq. (16), and using the following integral formula

*I*is the modified Bessel function of the first kind with order

_{j}*j*. After integration, we can obtain

The total energy received by the detector can be written as [24]

where [6,43,44] represents the energy content of each OAM mode, and*R*denotes the receiving aperture radius.

The energy fraction possessed by the *m*-th spiral harmonic of the EGB after turbulent transmission is determined by the following expression

When the designated OAM mode *n* propagates through atmospheric turbulence, the value of ${P_m}$ for $m = n$ will decrease at the receiving plane due to the random phase disturbance caused by turbulence. In this case, ${P_{m = n}}$ can be regarded as the detection probability of signal OAM mode *n*, which shows the transmission efficiency of the transmitted OAM mode. As the high-order EGB studied in this paper is a multiplexed OAM mode beam, it has multiple OAM states at the source plane. When transmitted in the turbulent atmosphere, crosstalk between different channels presents different distributions than the beam with a single OAM state. For $m = n \pm \Delta n$, define ${P_m}$ as the crosstalk probability, which denotes the probability of a photon changing its OAM mode, and $\Delta n$ is the difference between the transmitted and received OAM mode.

## 3. Numerical results

Since the conclusion of high-order EGB can be extended from the 3-order EGB, hence, here we take 3-order EGB as an example to study its transmission characteristics.

Figure 2 shows the OAM spiral spectrum distribution of the EGB at the source plane, and the calculation parameters are *λ*=1550 nm, *w*=0.05 m, *n*=3. Unless otherwise specified, the following calculation parameters are the same. According to Fig. 2, as mentioned above, the OAM of the EGB when *n*=3 is the superposition of four (3 + 1) OAM modes, which are −3, −1, 1, 3. The sum of the detection probabilities of these OAM modes is 1. For a single-mode OAM beam, its OAM is equal to the topological charge. However, in EGB, its OAM distribution is closely related to the value of *a*. As the value of *a* increases, ${P_{m = 3}}$ will decrease, and its energy will be redistributed into the other three OAM states. It can be observed from the OAM spectrum distribution at the source plane that when *a*<0, the greater the value of *a* and the closer to −1, the greater the probability of its center detection, that is, the probability of ${P_{m ={-} 3}}$, and the higher the purity; This is the opposite of the situation when *a*>0, the smaller the value of *a*, the closer to +1, the greater the probability of detection of ${P_{m = 3}}$. In the Ref. [31], only the difference in the spiral direction of the phase is given when *a* is positive and negative, so we give a detailed discussion here. The positive and negative orientations of *a* also change the distribution of OAM modes inside the beam. In this way, we believe that the security of quantum communication lines can be achieved by using this kind of EGB, and due to its multi-channel characteristics, theoretically, the larger the order *n*, the more OAM modes and the stronger the multiplexing. Because when *a*=±1, EGB degenerates into a common doughnut beam, and its phase distribution is canonical. Therefore, this paper studies the transmission characteristics of the EGB spiral spectrum when *a*≠±1.

Figure 3 shows the variation trend of OAM detection probability with the transmission distance *z* under different values of *a*. The calculation parameters are *l*_{0}=0.01 m, *L*_{0}=1 m, *α*=11/3, $\widetilde C_n^2$=10^{−14}m^{3-α}, *R*=0.05 m.

It can be observed from Fig. 3 that as the transmission distance *z* increases, the OAM modes detection probability decreases, and its energy distribution spreads into adjacent modes, resulting in an increase in the detection probability of other states and increased crosstalk. This is because the EGB is continuously affected by turbulence in the atmospheric channel propagation, the detection probability of signal OAM mode will decrease and the crosstalk probability of adjacent modes will increase. Due to the different value of *a*, the OAM detection probability of an EGB is not like traditional single-mode beams (such as LGB, BGB, etc.) with a center-symmetric distribution. When *n*=3, its energy is mainly concentrated at $m = 1$. As *a* increases, its mode detection probability ${P_{m = 1}}$ gradually decreases from 0.58 (*a*=5) to 0.47 (*a*=35), and when transmitting to a distance of 2 km, the detection probability drops to about 0.18. This phenomenon can also be observed at $m ={-} 1$. As *a* increases, the OAM mode detection probability ${P_{m ={-} 1}}$ at the source plane decreases, but as the transmission distance increases, the final reception probability is around 0.15. Therefore, by controlling the ellipticity parameter *a*, the channel when the EGB propagates in atmospheric turbulence can be appropriately made more stable. If a larger ellipticity parameter is selected, its mode detection probability ${P_{m ={\pm} 1}}$ degrades less than a smaller ellipticity parameter. In addition, the research results show that when using the multiplexing characteristics of the EGB for data encoding and transmission, for different channels, the data loading can be selected according to different needs. Different channels are selected to deal with the distortion of turbulence by selecting the ellipticity parameter *a*.

Figure 4 shows the detection probability of adjacent modes when *a*=5, 15, 25, 35, that is, the probability of OAM crosstalk. Other calculation parameters are the same as Fig. 3. Since the OAM of the 3-order EGB is the superposition of the four states of −3, −1, 1, and 3, we have chosen five adjacent modes $m ={-} 4, - 2,0,2,4$ to study.

As shown in Fig. 4, regardless of the value of the parameter *a*, the increase and rate are the highest and fastest for mode $m = 0$ and $m = 2$. As mentioned above, when *n*=3, the OAM energy of the EGB is concentrated at $m = 1$, so the adjacent mode $m = 0$ and $m = 2$ have the strongest crosstalk. As the transmission distance *z* reaches 1 km, ${P_{m = 0}}$ and ${P_{m = 2}}$ reaches its peak and then stabilizes. As the transmission distance continues to increase, the detection probability ${P_{m = 0}}$ and ${P_{m = 2}}$ gradually decreases. It is evident that when the transmission distance increases, the EGB is continuously affected by atmospheric turbulence, the crosstalk also continues, and its energy gradually spreads into more modes. It is worth noting that the value of *a* seems to have nothing to do with the increase and rate in the detection probability ${P_{m = 0}}$. When the transmission distance reaches 1 km, the peak value is about 0.2, and then it tends to be stable. With the increase of *a*, the increase and rate of the detection probability ${P_{m = 0}}$ will decrease, and the detection probability ${P_{m = 4}}$ and ${P_{m ={-} 4}}$ will also decrease, indicating that the crosstalk of OAM can be reduced by increasing the value of *a*, so that the purity of OAM transmission in atmospheric turbulence will be higher.

We are now studying the influence of other turbulence parameters, such as the power spectrum index *α*, inner scale *l*_{0} and outer scale *L*_{0} on the detection probability of different OAM modes.

Figure 5 depicts the variation trend of the mode detection probability ${P_{m ={\pm} 1}}$ of the EGB with power spectrum index *α* for different inner scale *l*_{0}. The calculation parameters are: *λ*=1550 nm, *a*=5, *w*=0.05 m, $\widetilde C_n^2$=10^{−14}m^{3-}* ^{α}*,

*R*=0.05 m. The OAM mode here chooses $m ={\pm} 1$, and ${P_{m ={\pm} 1}}$ decreases first with the increase of

*α*, reaches a minimum value around

*α*=3.1, and then increases. This inflection point indicates that for EGB, it is disturbed by the strongest atmospheric turbulence around

*α*=3.1. As shown in Fig. 5(c), according to Eq. (22), the parameter

*T*(

*α*,

*z*) is proportional to the intensity of atmospheric turbulence. It first increases and then decreases with the increase of

*α*. The value of

*α*at the inflection point is consistent with Fig. 5(a) and Fig. 5(b), indicating that the atmospheric turbulence is the strongest, the disturbance to the beam is the strongest, the detection probability of OAM is the lowest, and the mode crosstalk is the strongest around

*α*=3.1. Since the inner scale is defined as the scale of small turbulent eddies, the disturbance to the beam becomes stronger as it increases. In Fig. 5(d), we show the variations of detection probability

*P*

_{m}_{=1}versus non-Kolmogorov parameter

*α*. As we discussed in Fig. 5(c), with the increase of

*α*, the disturbance of EGB decreases and the detection probability increases. In addition, we can observe that as the inner scale

*l*

_{0}(0.01 m, 0.1 m, 0.2 m) increases, the value of ${P_{m ={\pm} 1}}$ decreases. It is evident that in Fig. 5(c), as the inner scale

*l*

_{0}increases, the value of

*T*(

*α*,

*z*) decreases, and the atmospheric turbulence at this time is stronger, so ${P_m}$ also decreases. From Figs. 5(a)–5(c), it can be found that for the different values of

*l*

_{0}, the value of

*α*at the inflection point will have some drift.

Figures 6(a) and 6(b) show the dependence of the mode detection probability ${P_{m ={\pm} 1}}$ on the inner and outer scales when *a*=5. The inner scale *l*_{0} has a greater influence on the mode detection probability than the outer scale *L*_{0}. As we know, the outer scale will cause the beam position to drift(wander), and the inner scale will cause the fluctuation (scintillation). When the outer scale *L*_{0} is greater than 5 m, the probability of mode detection changes little. With the increase of the inner scale *l*_{0}, the detection probability gradually increases. The dependence of OAM mode detection probability ${P_{m ={\pm} 1}}$ on the generalized structural parameter $\widetilde C_n^2$ and the wavelength of light *λ* is also shown in Figs. 6(c) and 6(d). With the increase of turbulence intensity $\widetilde C_n^2$, ${P_{m ={\pm} 1}}$ gradually decreases, and the increase of wavelength will also cause the increase of ${P_{m ={\pm} 1}}$. As we expected, longer wavelengths are more beneficial when light waves propagate in atmospheric turbulence channels.

## 4. Comparison of EGB and LGB

Here we compare the OAM mode detection probability of the EGB and the LGB in [24]. Figure 7 shows the results of our research.

Figures 7(a) and 7(b) show the mode detection and crosstalk probability of the EGB and LGB, respectively. The calculation parameters are: *λ*=1550 nm, *a*=5, *w*=0.05 m, $\widetilde C_n^2$=10^{−14}m^{3-}* ^{α}*,

*l*

_{0}=0.01 m,

*L*

_{0}=1 m,

*R*=0.05 m.

Under the same calculation conditions, EGB has less OAM mode loss and more suitable for 2 km distance communication system compared with the LGB. It can be observed from Fig. 7(a) that when the topological charge of the LGB is also *l*=3, its OAM mode detection probability decreases with the increase of transmission distance; with the increase of transmission distance to 2km, its detection probability decreases from 1 to about 0.3, and its launched OAM mode loses 70% in atmospheric turbulence. In contrast to EGB, its three modes ${P_{m ={-} 1}}$, ${P_{m = 1}}$, ${P_{m = 3}}$ decrease from 0.58, 0.26, and 0.15 to 0.19, 0.14, and 0.1, respectively. The three modes lose a total of 56% in the atmosphere. Compared with the single-mode LGB, EGB has less OAM mode loss and less attenuation when propagating in atmospheric turbulence (we did not discuss the detection probability when $m ={-} 3$ here, because it accounts for a small proportion of the other three modes, only 0.01). In addition, when the transmission distance is between 1km and 2km, the OAM detection probability of a single-mode LGB declines faster than the three-mode decline rate of EGB. After 1km, the OAM detection probability of the three modes of EGB has basically stabilized, which indicates that under the same conditions, EGB is more suitable for long-distance communication scenarios than LGB.

Figure 7(b) shows the comparison of mode crosstalk probability between EGB and LGB. We have studied the probability of crosstalk between adjacent LGB of 1 and 2 (${P_{\Delta l = 1}}$ and ${P_{\Delta l = 2}}$) and the detection probability of adjacent modes of EGB ${P_{m ={-} 2}}$, ${P_{m = 0}}$, ${P_{m = 2}}$, ${P_{m = 4}}$ and compared them. It can be found that compared with LGB, the probability of crosstalk between adjacent modes ${P_{m = 0}}$ and ${P_{m = 2}}$ of EGB is smaller. However, for ${P_{m = 2}}$ and ${P_{m = 4}}$, the crosstalk probability is greater than that of the LGB before the transmission distance of 1km, and becomes relatively smaller after 1km. It is evident that before the transmission distance is 1km, the detection probability at ${P_{m = 2}}$ and ${P_{m = 4}}$ also receives the crosstalk between ${P_{m ={-} 1}}$ and ${P_{m = 3}}$. In the longer propagation length, due to the influence of atmospheric turbulence, the intensity scintillation and drift will be more severe, and the spiral phase will degenerate. In addition, due to the influence of diffraction effect and aperture, it is difficult to capture and receive all the signals in the experiment. Therefore, for the establishment of optical communication system, compared with choosing a longer propagation length, we think that the distance below 2 km is short.

Here we also consider why the EGB loses less OAM in atmospheric turbulence at the same distance. This is because when the order of EGB is *n*=3, its OAM spiral spectrum distribution is mainly concentrated on the modes of $m ={-} 1$ and $m = 1$. Some reference conclusions indicate that a lower topological charge is more suitable when transmitting in atmospheric channel. Therefore, the 3-order EGB has better transmission performance than the 3-order LGB. For the case of low-order EGB (such as order 1 and 2), its transmission characteristics must be better than that of LGB of the same order. However, low-order EGB has limited channels compared to high-order EGB, and high-order EGB greatly improves the capacity of optical communication systems through multi-channel multiplexing.

## 5. Higher-order EGB transmission performance

Figure 8 displays the detection probability ${P_m}$ of the EGB for several orders 3, 4, 5, 6, 7 at transmission distance *z*=0m (a, c, e, g, i) and *z*=2000m (b, d, f, h, j). The calculation parameters are: *λ*=1550nm, *a*=5, *w*=0.05m, $\widetilde C_n^2$=10^{−14}m^{3-}* ^{α}*,

*l*

_{0}=0.01m,

*L*

_{0}=1m,

*R*=0.05m. It is evident that higher-order EGB have more channels at source plane.

It can be observed from Fig. 8 that if we fixed the ellipticity parameter *a*, the transmission performance of higher-order EGB is not much different than that of 3-order EGB. When the transmission distance is 2000m, the spiral spectrum of the 7-order EGB spreads into more modes. It is worth noting that as the order increases, the mode energy always concentrated at $m = 1$ through atmosphere turbulence, and its detection probability ${P_{m = 1}}$ is not much different (both greater than 0.16). It seems that the order of EGB has little effect on its transmission performance. Physically, the main reason for this phenomenon is that the modes interval of EGB is fixed, and its energy are mainly degraded to the low modes. Since the higher-order EGB has multi-channel and can expand capacity, we believe high-order EGB has advantages in the field of OAM-based multiplexing communications.

Certainly, regarding the application of EGB in OAM-based multiplexing communication systems, the order is not as good as higher. For higher-order EGB, due to the increase of its channels, its transmission performance will be greatly reduced and crosstalk of the spiral spectrum will increase. Therefore, it is necessary to consider such as the number of channels and degradation or crosstalk caused by atmospheric turbulence, and then select the order according to specific needs. The limitation of EGB is about its small modes interval, some reference had proposed that a larger modes interval is usually used in optical communication systems based on OAM multiplexing [45]. However, according to [46], the average crosstalk of multiplexed OAM beams with a mode interval of 2 is not much different from that of OAM beams with an interval of 3 or 4.

## 6. Optimization of ellipticity parameter

We find out the corresponding relation of ellipticity parameter *a* to OAM spiral spectrum distribution of high-order EGB, and display it in Figs. 9 and 10. The calculation parameters are *λ*=1550 nm, *w*=0.05 m, $\widetilde C_n^2$=10^{−14}m^{3-α}, *α*=11/3, *l*_{0}=0.01 m, *L*_{0}=1 m, *R*=0.05 m. Due to the spiral spectrum distribution characteristics of the high-order EGB, for odd-orders, it always has a mode *m*=1, and for even-orders, it always has a mode *m*=2. Therefore, for the optimal value of parameter *a*, we only need to ensure that the detection probability is maximized at *m*=1 (odd orders) and *m*=2 (even orders), because low-order accounts for more energy and its anti-turbulence characteristics are stronger.

Figure 9 displays the dependence of detection probability ${P_{m = 1}}$ for odd-orders with ellipticity parameter *a* at different transmission distances. The ellipticity parameter *a* is divided into two intervals: 0≤*a*≤1 [Figs. 9(a), 9(c), 9(e)] and *a*>1 [Figs. 9(b), 9(d), 9(f)]. Whatever 0≤*a*≤1 or *a*>1, the detection probability ${P_{m = 1}}$ first increases and then decreases with the increase of *a* under different transmission distances. There exists an extreme value that maximizes ${P_{m = 1}}$ as we marked in the figure. For 0≤*a*≤1, the ellipticity parameter *a* has a drift under different transmission distance, and as the order increases, the optimal value of *a* is approaching to the left. For *a*>1, the optimal value of *a* is approaching to the right. This optimal value is approximately the same for testing several different turbulence intensities and detector apertures.

Figure 10 displays the dependence of detection probability ${P_{m = 2}}$ for even-orders with ellipticity parameter *a* at different transmission distances. As mentioned above, for 0≤*a*≤1, the ellipticity parameter *a* has a drift under different transmission distance, and for *a*>1, the optimal value of *a* is approaching to the right. This is because when *a*>1, as the order increases, the increase in *a* will distribute the mode energy to lower mode. Similarly, when 0≤*a*≤1, the decrease of *a* will more distribute the mode energy of EGB to the lower mode.

Figure 11 displays the relationship between *a* and *n* in some limiting cases. Whether *n* is odd or even, we choose larger *n* and find the corresponding relationship between optimal value *a* and *n* is still satisfied. This means that we can optimally select parameter *a* according to different transmission distance scenarios. We fitted and organized a formula. For communication system with a transmission distance of less than 2 km, the relationships between the optimal value and the order are: $a = 0.06 - 0.05n + 0.002{n^2} - 0.00002{n^3}(0 \le a \le 1)$ and $a = 1.63 + 0.6n - 0.007{n^2} + 0.00005{n^3}({a > 1} )$ when the order of EGB is odd. When the order is even, the relationships are: $a = 0.\textrm{74 - 0}\textrm{.07}n\textrm{ + 0}\textrm{.002}{n^2} - 0.00002{n^3}(0 \le a \le 1)$ and $a = 0.\textrm{7 + 0}\textrm{.32}n - \textrm{0}\textrm{.004}{n^2} + 0.00003{n^3}({a > 1} )$.

## 7. Conclusion

We have developed a theoretical model for calculating the spiral spectrum of EGB through non-Kolmogorov turbulence. On the basis of this model, the effect of ellipticity parameters, orders, turbulence power spectrum index, inner scale and outer scale and other turbulence parameters on the OAM detection and crosstalk probability have been investigated. The results show that the ellipticity parameter has a significant impact on the OAM spectral distribution of the EGB. In addition, compared with the outer scale, the inner scale has a greater influence on the OAM mode detection probability, and the increase of the inner scale will cause the OAM mode detection probability to increase significantly. Moreover, a longer wavelength of beam can also increase the probability of detection. As the EGB is the finite sum of LGB with different topological charges, we also compared the mode detection and crosstalk probability of the EGB and the LGB. The conclusion points out that under the same conditions, the attenuation and degradation of the OAM mode of EGB in atmospheric turbulence is smaller. For the communication background with a transmission distance greater than 1km, we believe that the EGB is superior to the Laguerre-Gaussian beam. For the order between 3 and 7, the transmission performance of EGB degrades slightly. We have concluded that the relationships of optimal ellipticity parameter *a* and order *n* for odd-order EGB are $a = 0.06 - 0.05n + 0.002{n^2} - 0.00002{n^3}(0 \le a \le 1)$ and $a = 1.63 + 0.6n - 0.007{n^2} + 0.00005{n^3}({a > 1} )$, wheras $a = 0.\textrm{74 - 0}\textrm{.07}n\textrm{ + 0}\textrm{.002}{n^2} - 0.00002{n^3}(0 \le a \le 1)$ and $a = 0.\textrm{7 + 0}\textrm{.32}n - \textrm{0}\textrm{.004}{n^2} + 0.00003{n^3}({a > 1} )$ for even-order. Our results may be helpful in the fields of short-distance optical communication and OAM-based multiplex communication.

## Funding

National Natural Science Foundation of China (61875156, U20B2059); 111 Project (B17035).

## Disclosures

The authors declare no conflicts of interest.

## References

**1. **L. Allen, M. W. Beijersbergen, R. J. Spreeuw, and J. P. Woerdman, “Orbital angular momentum of light and the transformation of Laguerre-Gaussian laser modes,” Phys. Rev. A **45**(11), 8185–8189 (1992). [CrossRef]

**2. **M. Padgett and R. Bowman, “Tweezers with a twist,” Nat. Photonics **5**(6), 343–348 (2011). [CrossRef]

**3. **A. M. Yao and M. J. Padgett, “Orbital angular momentum: origins, behavior and applications,” Adv. Opt. Photonics **3**(2), 161–204 (2011). [CrossRef]

**4. **J. Wang, J. Y. Yang, I. M. Fazal, N. Ahmed, Y. Yan, H. Huang, Y. Ren, Y. Yue, S. Dolinar, M. Tur, and A. E. Willner, “Terabit free-space data transmission employing orbital angular momentum multiplexing,” Nat. Photonics **6**(7), 488–496 (2012). [CrossRef]

**5. **A. E. Willner, H. Huang, Y. Yan, Y. Ren, and S. Ashrafi, “Optical communications using orbital angular momentum beams,” Adv. Opt. Photonics **7**(1), 66–106 (2015). [CrossRef]

**6. **L. Torner, J. P. Torres, and S. Carrasco, “Digital spiral imaging,” Opt. Express **13**(3), 873–881 (2005). [CrossRef]

**7. **G. Gibson, J. Courtial, M. J. Padgett, M. Vasnetsov, and S. Franke-Arnold, “Free-space information transfer using light beams carrying orbital angular momentum,” Opt. Express **12**(22), 5448–5456 (2004). [CrossRef]

**8. **M. Malik, M. O’Sullivan, B. Rodenburg, M. Mirhosseini, J. Leach, M. P. J. Lavery, M. J. Padgett, and R. W. Boyd, “Influence of atmospheric turbulence on optical communications using orbital angular momentum for encoding,” Opt. Express **20**(12), 13195–13200 (2012). [CrossRef]

**9. **Y. Ren, H. Huang, G. Xie, N. Ahmed, Y. Yan, B. I. Erkmen, N. Chandrasekaran, M. P. Lavery, N. K. Steinhoff, M. Tur, S. Dolinar, M. Neifeld, M. J. Padgett, R. W. Boyd, J. H. Shapiro, and A. E. Willner, “Atmospheric turbulence effects on the performance of a free space optical link employing orbital angular momentum multiplexing,” Opt. Lett. **38**(20), 4062–4065 (2013). [CrossRef]

**10. **Y. Ren, Z. Wang, P. Liao, L. Li, G. Xie, H. Huang, Z. Zhao, Y. Yan, N. Ahmed, A. Willner, M. P. Lavery, N. Ashrafi, S. Ashrafi, R. Bock, M. Tur, I. B. Djordjevic, M. A. Neifeld, and A. E. Willner, “Experimental characterization of a 400 Gbit/s orbital angular momentum multiplexed free-space optical link over 120 m,” Opt. Lett. **41**(3), 622–625 (2016). [CrossRef]

**11. **N. Mphuthi, L. Gailele, I. Litvin, A. Dudley, R. Botha, and A. Forbes, “Free-space optical communication link with shape-invariant orbital angular momentum Bessel beams,” Appl. Opt. **58**(16), 4258–4264 (2019). [CrossRef]

**12. **L. Gong, Q. Zhao, H. Zhang, X. Y. Hu, K. Huang, J. M. Yang, and Y. M. Li, “Optical orbital-angular-momentum-multiplexed data transmission under high scattering,” Light Sci. Appl. **8**(1), 268–278 (2019).

**13. **S. Yu, “Potentials and challenges of using orbital angular momentum communications in optical interconnects,” Opt. Express **23**(3), 3075–3087 (2015). [CrossRef]

**14. **N. Bozinovic, Y. Yue, Y. Ren, M. Tur, P. Kristensen, H. Huang, A. E. Willner, and S. Ramachandran, “Terabit-scale orbital angular momentum mode division multiplexing in fibers,” Science **340**(6140), 1545–1548 (2013). [CrossRef]

**15. **M. J. Willner, H. Huang, N. Ahmed, G. Xie, Y. Ren, Y. Yan, M. P. Lavery, M. J. Padgett, M. Tur, and A. E. Willner, “Reconfigurable orbital angular momentum and polarization manipulation of 100 Gbit/s QPSK data channels,” Opt. Lett. **38**(24), 5240–5243 (2013). [CrossRef]

**16. **S. Li and J. Wang, “Performance evaluation of analog signal transmission in an orbital angular momentum multiplexing system,” Opt. Lett. **40**(5), 760–763 (2015). [CrossRef]

**17. **J. Du and J. Wang, “High-dimensional structured light coding/decoding for free-space optical communications free of obstructions,” Opt. Lett. **40**(21), 4827–4830 (2015). [CrossRef]

**18. **J. A. Anguita, M. A. Neifeld, and B. V. Vasic, “Turbulence-induced channel crosstalk in an orbital angular momentum-multiplexed free-space optical link,” Appl. Opt. **47**(13), 2414–2429 (2008). [CrossRef]

**19. **B. Rodenburg, M. Mirhosseini, M. Malik, O. S. Magana-Loaiza, M. Yanakas, L. Maher, N. K. Steinhoff, G. A. Tyler, and R. W. Boyd, “Simulating thick atmospheric turbulence in the lab with application to orbital angular momentum communication,” New J. Phys. **16**(3), 033020 (2014). [CrossRef]

**20. **C. Chen, H. Yang, S. Tong, and Y. Lou, “Changes in orbital-angular-momentum modes of a propagated vortex Gaussian beam through weak-to-strong atmospheric turbulence,” Opt. Express **24**(7), 6959–6975 (2016). [CrossRef]

**21. **S. Fu and C. Gao, “Influences of atmospheric turbulence effects on the orbital angular momentum spectra of vortex beams,” Photonics Res. **4**(5), B1–B4 (2016). [CrossRef]

**22. **Y. Zhang, Y. Wang, J. Xu, J. Wang, and J. Jia, “Orbital angular momentum crosstalk of single photons propagation in a slant non-Kolmogorov turbulence channel,” Opt. Commun. **284**(5), 1132–1138 (2011). [CrossRef]

**23. **Y. Yuan, D. Liu, Z. Zhou, H. Xu, J. Qu, and Y. Cai, “Optimization of the probability of orbital angular momentum for Laguerre-Gaussian beam in Kolmogorov and non-Kolmogorov turbulence,” Opt. Express **26**(17), 21861–21871 (2018). [CrossRef]

**24. **J. Zeng, X. Liu, C. Zhao, F. Wang, G. Gbur, and Y. Cai, “Spiral spectrum of a Laguerre-Gaussian beam propagating in anisotropic non-Kolmogorov turbulent atmosphere along horizontal path,” Opt. Express **27**(18), 25342–25356 (2019). [CrossRef]

**25. **J. Ou, Y. Jiang, J. Zhang, H. Tang, Y. He, S. Wang, and J. Liao, “Spreading of spiral spectrum of Bessel–Gaussian beam in non-Kolmogorov turbulence,” Opt. Commun. **318**, 95–99 (2014). [CrossRef]

**26. **M. Cheng, L. Guo, J. Li, and Q. Huang, “Propagation properties of an optical vortex carried by a Bessel-Gaussian beam in anisotropic turbulence,” J. Opt. Soc. Am. A **33**(8), 1442–1450 (2016). [CrossRef]

**27. **Y. Xu and Y. Zhang, “Bandwidth-limited orbital angular momentum mode of Bessel Gaussian beams in the moderate to strong non-Kolmogorov turbulence,” Opt. Commun. **438**, 90–95 (2019). [CrossRef]

**28. **Y. Zhu, L. Zhang, Z. Hu, and Y. Zhang, “Effects of non-Kolmogorov turbulence on the spiral spectrum of Hypergeometric-Gaussian laser beams,” Opt. Express **23**(7), 9137–9146 (2015). [CrossRef]

**29. **Y. Zhang, M. Cheng, Y. Zhu, J. Gao, W. Dan, Z. Hu, and F. Zhao, “Influence of atmospheric turbulence on the transmission of orbital angular momentum for Whittaker-Gaussian laser beams,” Opt. Express **22**(18), 22101–22110 (2014). [CrossRef]

**30. **P. Li, S. Liu, T. Peng, G. Xie, X. Gan, and J. Zhao, “Spiral autofocusing Airy beams carrying power-exponent-phase vortices,” Opt. Express **22**(7), 7598–7606 (2014). [CrossRef]

**31. **V. V. Kotlyar, A. A. Kovalev, and A. P. Porfirev, “Elliptic Gaussian optical vortices,” Phys. Rev. A **95**(5), 053805 (2017). [CrossRef]

**32. **M. R. Dennis, “Rows of optical vortices from elliptically perturbing a high-order beam,” Opt. Lett. **31**(9), 1325–1327 (2006). [CrossRef]

**33. **V. V. Kotlyar, S. N. Khonina, A. A. Almazov, V. A. Soifer, K. Jefimovs, and J. Turunen, “Elliptic Laguerre-Gaussian beams,” J. Opt. Soc. Am. A **23**(1), 43–56 (2006). [CrossRef]

**34. **M. A. Bandres and J. C. Gutierrez-vega, “Ince–Gaussian beams,” Opt. Lett. **29**(2), 144–146 (2004). [CrossRef]

**35. **E. G. Abramochkin and V. G. Volostnikov, “Generalized Gaussian beams,” J. Opt. A: Pure Appl. Opt. **6**(5), S157–S161 (2004). [CrossRef]

**36. **A. A. Kovalev and V. V. Kotlyar, “Lommel modes,” Opt. Commun. **338**, 117–122 (2015). [CrossRef]

**37. **G. Molina-Terriza, J. Recolons, J. P. Torres, L. Torner, and E. M. Wright, “Observation of the Dynamical Inversion of the Topological Charge of an Optical Vortex,” Phys. Rev. Lett. **88**(1), 013601 (2001). [CrossRef]

**38. **I. S. Gradshteĭn, I. M. Ryzhik, Y. V. Geronimus, and M. Y. Tseytlin, * Table of integrals, series, and products* (Academic Press, 2007).

**39. **L. C. Andrews and R. L. Phillips, * Laser Beam Propagation Through Random Media* (The International Society for Optical Engineering Press, 2005).

**40. **C. Paterson, “Atmospheric turbulence and orbital angular momentum of single photons for optical communication,” Phys. Rev. Lett. **94**(15), 153901 (2005). [CrossRef]

**41. **H. T. Yura, “Mutual Coherence Function of a Finite Cross Section Optical Beam Propagating in a Turbulent Medium,” Appl. Opt. **11**(6), 1399–1406 (1972). [CrossRef]

**42. **Y. Zhu, M. Chen, Y. Zhang, and Y. Li, “Propagation of the OAM mode carried by partially coherent modified Bessel–Gaussian beams in an anisotropic non-Kolmogorov marine atmosphere,” J. Opt. Soc. Am. A **33**(12), 2277–2283 (2016). [CrossRef]

**43. **H. I. Sztul and R. R. Alfano, “The Poynting vector and angular momentum of Airy beams,” Opt. Express **16**(13), 9411–9416 (2008). [CrossRef]

**44. **Y.-D. Liu, C. Gao, M. Gao, and F. Li, “Coherent-mode representation and orbital angular momentum spectrum of partially coherent beam,” Opt. Commun. **281**(8), 1968–1975 (2008). [CrossRef]

**45. **S. Fu, S. Zhang, T. Wang, and C. Gao, “Measurement of orbital angular momentum spectra of multiplexing optical vortices,” Opt. Express **24**(6), 6240–6248 (2016). [CrossRef]

**46. **A. J. Willner, Y. Ren, G. Xie, Z. Zhao, Y. Cao, L. Li, N. Ahmed, Z. Wang, Y. Yan, and P. Liao, “Experimental demonstration of 20Gbit/s data encoding and 2 ns channel hopping using orbital angular momentum modes,” Opt. Lett. **40**(24), 5810–5813 (2015). [CrossRef]