## Abstract

Heterodyne efficiency is referred as a measure of the quality for the coherent laser communication. The heterodyne efficiency not only reflects the matching of phase and amplitude between the received signal and the local oscillator, but also reveals the polarization matching between the two beams. Different from the common heterodyne efficiency, a revised heterodyne efficiency is proposed by considering the polarization aberrations of optical system. Based on the Polar and Pauli-Zernike decomposition algorithms, the effects of polarization aberrations on the output polarization states are analyzed and shown graphically. The variations of the heterodyne efficiency are investigated by including the separate component of polarization aberrations in mixing of two perfectly matched Gaussian beams. Depending on the modified heterodyne efficiency, an off-axis optical system with a periscopic scanner is modeled and used to discuss the variations of the heterodyne efficiency. A further investigation for three different coatings is accomplished to verify the effects the varied polarization aberrations have on the heterodyne efficiency. The analysis indicates that the modified heterodyne efficiency not only can provide a comprehensive description of the coherent detection system, but also can be used to evaluate and minimize the polarization aberrations of optical system.

© 2017 Optical Society of America

## 1. Introduction

Incoherent detection, or direct detection, has poor noise rejection properties especially for the detection of weak signals or when the detector has low sensitivity. On the other hand, heterodyne detection is a more powerful detection technique that has been used in different applications owing to its noise-reduction capabilities especially in the free space laser communication systems [1,2]. Heterodyne detection is well recognized as a method of obtaining quantum-limited reception in spectral range where direct detection cannot be made quantum-limited. Heterodyne detection can be performed by mixing the received optical signal with a locally generated signal on a photo-detector surface and measuring the electrical output of the detector. The heterodyne efficiency is a measure of the ability of the heterodyne detection system to evaluate the performance of the coherent optical systems. For efficient coherent mixing at the detector plane, the locally generated signal parameters should match the received signal parameters and the heterodyne efficiency reflects this matching. However, the optical signal is received by the optical antenna, which means that the aberrations of optical system will impair the heterodyne detection and lower the heterodyne efficiency if the aberrations are not carefully controlled. The aberrations of optical system change the amplitude, phase and polarization states of optical signal and degrade the matching of received signal and local oscillator (LO) signal.

The heterodyne efficiency of coherent signals has been studied for a long time. For example, Fink [3] derived expressions for the signal-to-noise ratio (SNR) and the heterodyne efficiency in a coherent detection system of deterministic fields. Takenaka and Tanaka discussed the SNR for Gaussian fields [4], and Tanaka and Ohta studied the effect of the tilt and the offset of the received signal on the heterodyne efficiency of Gaussian beams [5]. Furthermore, Tanaka and Saga derived the optimal conditions for maximum heterodyne efficiency in coherent systems in the presence of background radiation [6]. These studies are mainly focused on the electric field distributions of signals and oscillators and their effects on the heterodyne efficiency. Then, Cohen [7] studied the effects of signal wavefront misalignment for various combinations of received signals and local oscillators. Chambers [8] presented a model for the heterodyne efficiency in the presence of wavefront distortion due to aberration in optical systems, and later Delautre et al. [9] studied these experimentally, using spatial light modulators. These references are mainly aimed at studying the wavefront aberrations of optical systems and their effects on the heterodyne efficiency. However, all the references mentioned above have a common assumption that the polarization states of the received signals and the local oscillators are uniform and identical across the detector.

The aberrations of optical system mainly contain wavefront aberration, apodization, and polarization aberration [10]. The distributions of the amplitude, phase, and polarization states of received signals are influenced by the aberrations of optical system. In coherent detection technique, the polarization changes induced by polarization-mode dispersion (PMD) and polarization-dependent loss (PDL) in fiber systems are discussed in references [11,12]. The effects of coherence and polarization changes on heterodyne detection in free space are discussed in reference [13], yet the detailed analysis of polarization aberrations of optical systems and their effects on heterodyne detection are not particularly performed. For this reason, the principal purpose of the present work is to give a comprehensive description of the heterodyne efficiency when the polarization aberrations are included in the coherent detection system. By introducing the polarization aberrations of the optical system, we incorporate the aberrations into the calculations of the modified heterodyne efficiency. Furthermore, we use the modified heterodyne efficiency to evaluate and minimize the polarization aberrations, and to improve the performance of the coherent laser communication system.

In this paper, we propose a modified heterodyne efficiency by including the polarization aberrations, and perform the related analysis. The paper is structured as follows. Section 2 describes the specific model for coherent detection and proposes the modified heterodyne efficiency with polarization aberrations. Section 3 introduces the representations of polarization aberrations of optical systems. In Section 4, examples of specific aberration in mixing of deterministic signal and LO fields will be shown to illustrate the effects of polarization aberrations on the heterodyne efficiency. In Section 5, an off-axis afocal optical system will be used to illustrate the extension of the modified heterodyne efficiency to the coherent laser communication systems. Section 6 gives the concluding remarks.

## 2. Modified formulation for heterodyne efficiency with polarization aberrations

Heterodyne efficiency is the ratio of average effective coherent power to total incoherent power in the system. Following the procedure in reference 2, the signal ${E}_{s}={\widehat{e}}_{s}S{U}_{s}\left(r\right)\mathrm{exp}\left(-i{\omega}_{s}t\right)$ is the electric field at the detector plane, where ${\widehat{e}}_{s}$ is a unit vector in the direction of polarization of ${E}_{s}$, $S$ is the maximum amplitude, ${U}_{s}\left(r\right)$ is the normalized complex field function (including phase), and ${\omega}_{s}$ is the signal photon angular frequency. Similarly, let ${E}_{l}^{}$, ${\widehat{e}}_{l}$, $L$, ${U}_{l}\left(r\right)$, and ${\omega}_{l}$ be the corresponding quantities for the LO. The quantum efficiency is uniform over the area of the detector. Therefore, based on the assumption that the signal and LO have the same polarization states across the detector plane, the SNR is written as [3]:

In order to estimate the polarization change of the received signal and the corresponding degradation of the heterodyne efficiency, we present a specific model to show those effects in heterodyne detection. The model is shown in Fig. 1. The polarization states of the received signals are transformed by the polarization aberrations of the optical antenna (OA). The polarization properties of the optical antenna can be expressed as the Jones matrix ${J}_{sys}$. A polarizer P which is not contained in real coherent detection system, is used in this model to ensure the polarization states of the signals and LOs are parallel to each other. Then, the linearly polarized signal and LO enter the optical hybrid (OH) which combines the two beams. The detailed explanations of the optical hybrid are described in reference [14].

Due to the polarization aberration of optical antenna, the polarizations of the received signals are no longer uniform across the exit pupil. In order to realize the combination of the signal and LO, the polarization state of the two beams must be parallel to each other, which is achieved by the polarizer P. In this model, the LO is 45deg polarized, and the polarizer’s transmission axis is 45deg in the x-y plane. The Jones matrix of an ideal polarizer P at 45deg is expressed as

the Jones matrix of the optical system is expressed as ${J}_{sys}$_{${E}_{s}={\widehat{e}}_{s}S{U}_{s}\left(r\right)\mathrm{exp}\left(-i{\omega}_{s}t\right)$}and ${{E}^{\prime}}_{s}^{}={{\widehat{e}}^{\prime}}_{s}S{U}_{s}\left(r\right)\mathrm{exp}\left(-i{\omega}_{s}t\right)$, ${z}_{0}$is the characteristic impedance of the medium surrounding the detector and the integrals in Eq. (8) are the area of the detector. If the optical system has small polarization aberrations, the incident polarization state ${\widehat{e}}_{in}$ remains unchanged and the polarization efficiency ${\gamma}_{pol}$ is 100% for 45deg linear polarized and 50% for circular polarized. Then the available power ${P}_{s}$ is equal to the received power $P$ or half of the received signal power $P$. Hence, the polarization efficiency ${\gamma}_{pol}$ and ${P}_{s}$ are related to the polarization aberrations of optical system. By substituting ${P}_{s}$, ${\gamma}_{pol}$ into Eq. (1), the SNR including the polarization variations of signals is rewritten as

## 3. Polarization aberrations and the decompositions of Jones matrix

The Jones matrix ${J}_{sys}$ characterizes the polarization aberrations of optical system. Chipman proposed an analytical decomposition method in his dissertation [15]. Ruoff and Totzeck introduced a new concept of orientation of Zernike polynomials, which constitutes a base function representation of retardation and diattenuation of optical system [16]. McIntyre described a comparison of various representations of polarization aberrations [17]. Yamamoto, Kye, and Levinson described the Pauli-Zernike representation of polarization aberration [18]. Geh et al. presented a method to decompose the Jones pupils into quantities that represented a clear physical interpretation [19]. In this paper, we will combine the methods in references 18,19 and give an intuitive description of the polarization aberrations.

The Jones matrices can be defined in terms of the physical properties [19]. Polar decomposition theory, which employs a singular value decomposition of the complex Jones matrix, shows that any Jones matrix can be expressed as the order-independent product of a homogenous partial polarizer (${J}_{pol}$), a homogeneous pure retarder (${J}_{ret}$), a homogeneous rotator (${J}_{rot}$), and a scalar component (${J}_{S}$). Homogenous in this paper refers to an element with orthogonal eigenpolarizations. The physical meanings of these elements are listed in Table 1. It is noted that, strictly speaking, the rotator is not a basic polarization element, because it can be realized by a suitable combination of linear retarders.

Therefore, the Jones matrix ${J}_{sys}$ can be written as [19]:

_{$d$}represents the relative amplitude difference between the orthogonal eigenpolarizations. The relationship between diattenuation ${D}_{dia}$ and $d$ is given by ${D}_{dia}=\frac{2d}{1+{d}^{2}}$. The retardance ${\delta}_{ret}$ is equal to $2\varphi $. ${\psi}_{p}$ and ${\psi}_{r}$ represent the bright axis direction and fast axis direction of diattenuation and retardance, respectively; ${\delta}_{p}$ and ${\delta}_{r}$ represent the ellipticities of the diattenuation and retardance. The rotation angle $\zeta $ is related to the rotations of the coordinate system. If the optical system has no rotation device, the angle $\zeta $ is zero and the rotation matrix ${J}_{rot}$ reduces to the identity matrix. In Section 3, the angle $\zeta $ is set to zero.

The matrices ${J}_{pol}$ and ${J}_{ret}$ correspond to the polarizer and retarder oriented at ${\psi}_{p}$ and ${\psi}_{r}$. By decomposing ${J}_{pol}$ and ${J}_{ret}$ into Pauli-spin basis, the Jones matrices can be written as:

the coefficients in Eqs. (15) and (16) areThe Jones matrices calculated across the pupil of the optical system construct the Jones pupil. The Jones matrix at each location of the Jones pupil describes the polarization of a single ray path through the entire optical elements. This Jones matrix is unique for a particular field and pupil position. For the coherent laser communication system, the field of the system is rather limited that we only analyze the central field.

After the decompositions of the Jones matrix, the diattenuation and retardance can be considered independently. Additionally, the Pauli coefficients can be further decomposed into the scalar Zernike polynomials, resulting in a compact pupil specification of only a series of Zernike coefficients. In this paper, the first 9 scalar Zernike coefficients are used to describe the polarization aberrations. Therefore, the coefficients can be written as

## 4. Heterodyne efficiency with polarization aberrations

For the purpose of illustrating the effects of polarization aberrations have on heterodyne efficiency, we consider mixing two ideal Gaussian beams which are initially matched in both amplitude and phase. Assuming that the signal and local oscillator fields have the same polarization and are normally incident on the detector at their beam waists, the heterodyne efficiency maintains the maximum value 100% when the local oscillator has the same spot size as the signal [3,4]. With a large detector collecting all the energy from the signal and LO, and the heterodyne efficiency ${\gamma}_{i.f.}$ is 100%, the modified heterodyne efficiency in Eq. (10) is simplified as

The calculation of polarization efficiency ${\gamma}_{pol}$ is associated with the polarization states of the received signal, which is shown in Eq. (8). In this section, the incident signals ${\widehat{e}}_{in}$ are assumed to be 45deg polarized and right-handed circular polarized, and the corresponding polarization efficiency is analyzed separately.

#### 4.1 Incident signals are 45deg polarized

If the incident signals are 45deg polarized, the Jones vector is expressed as ${\widehat{e}}_{in}=1/\sqrt{2}{\left[\begin{array}{cc}1& 1\end{array}\right]}^{T}$. By substituting ${\widehat{e}}_{in}$into Eqs. (5) and (6) and Eq. (8), the polarization efficiency ${\gamma}_{pol}$ is written as

The Jones matrices of the optical system are obtained by the methods described in Section 3. The angle $\zeta $ is zero and the rotation matrix ${J}_{rot}$ is the identity matrix. To understand the contributions of each term in the polarization aberration expansions, the polarization effects of the first 9 scalar Zernike terms are studied. Figures 2–5 show the effects of diattenuation and retardance on the output polarization states. The plots of the line segments and ellipses indicate the magnitudes and orientations of the output polarizations. The line segments of the polarization state shown in each panel correspond to different pupil positions. Different polarization states in each panel means that the polarization aberrations constituted by the Zernike coefficients vary with the pupil positions.

Each figure contains nine separate maps, which corresponds to the first 9 scalar Zernike polynomials. The input values for this study have been chosen to be large enough to make the effects of the aberration plainly visible. The separate scalar Zernike expansion coefficients of ${d}_{1}$ and ${d}_{2}$ are set to 0.5. The output polarization states with x/y and 45/135deg linear diattenuation are shown in Figs. 2 and 3. For phase and retardance, the separate Zernike expansion coefficients of ${r}_{1}$ and ${r}_{2}$ are also set to 0.5. The output polarization states with x/y and 45/135deg linear retardance are shown in Figs. 4 and 5. In Figs. 2–5, all maps have coefficients ${d}_{0}$ and ${r}_{0}$ set to 1.0. These two parameters are the uniform identity matrices and represent the ideal nonpolarizing optical system. The units for phase and retardance are radians.

By substituting the Jones matrices ${J}_{sys}$ and Jones vectors ${\widehat{e}}_{in}$ into Eq. (5), the orientation angles and ellipticities of the output polarization states are calculated by the output Jones vectors ${\widehat{e}}_{s}$. These coefficients in Figs. 2–5 are orders of magnitude larger than typical diattenuation and retardance in lens and mirrors, but allow the nature of the polarization aberration to be readily interpreted. The diattenuation and retardance not only have the magnitudes, but also have their own directions. It is noted that the orientation of diattenuation corresponds to the orientation of the polarization state with the maximum transmission, and the orientation of retardance corresponds to the polarization state with the smaller phase shift [10,15,20]. In Fig. 2, the Pauli coefficient ${d}_{1}$ represents the diattenuation along the coordinate axes, and ${d}_{1}>0$ represents the x-axis diattenuation and ${d}_{1}<0$ represents the y-axis diattenuation. The positive diattenuation corresponds to the clockwise rotation and the negative diattenuation corresponds to the counter-clock rotation. In Fig. 3, the coefficient ${d}_{2}$ corresponds to the linear diattenuation along 45/135deg directions, and ${d}_{2}>0$ represents the 45deg diattenuation and ${d}_{2}<0$ represents the 135deg diattenuation. Since the orientation of the incident polarization is parallel or orthogonal to the 45/135deg transmission axis, the output polarization states maintain 45deg polarized, and the amplitudes are related to the magnitudes of 135deg diattenuation. In Fig. 4, ${r}_{1}>0$ indicates that the phase of y-axis advances before x-axis, which produces the right-handed polarization. Similarly, ${r}_{1}<0$ corresponds to the left-hand polarization. In Fig. 5, the incident 45deg polarization is perpendicular to the directions of the 135deg retardance. The retardance only changes the amplitude of the output polarization states. The output polarization states maintain 45deg polarized.

In Figs. 2–5, the polarization states of the received signal ${\widehat{e}}_{s}$ are calculated and plotted graphically. It is noted that the polarization states shown in Figs. 2–5 are the signals before the polarizer P. The polarization states of signal ${\widehat{e}}_{s}{}^{\prime}$ are obtained by substituting ${\widehat{e}}_{s}$ into Eq. (6). The polarization efficiency ${\gamma}_{pol}$ is defined as the ratio of the power of ${{E}^{\prime}}_{s}$ to ${E}_{s}$. According to the calculation of ${\gamma}_{pol}$ depicted in Eq. (8), the polarization efficiency is calculated by varying the magnitudes of each scalar Zernike coefficient in Eqs. (19) and (20). Figure 6 illustrates the reduction of heterodyne efficiency with the increment of the diattenuation and retardance.

In Figs. 6(a) and 6(b), the polarization efficiency ${\gamma}_{pol}$ decreases as the increment of the diattenuation and retardance along the coordinate axes. In Figs. 6(c) and 6(d), the polarization efficiency maintains 100%. The reason for this phenomenon is that, when the incident signals are 45deg polarized, the diattenuation and retardance along the coordinate axes will significantly alter the polarization states and lead to great energy loss. The diattenuation and retardance along 45/135deg directions keep the output polarization states 45deg polarized. These distributions of signal polarizations will have no energy loss after passing through the 45deg polarizer.

By decomposing ${d}_{1}$ and ${r}_{1}$ into the scalar Zernike polynomials and calculating the corresponding polarization efficiency, Fig. 6 shows that Z_{1} has the greatest effect on polarization efficiency. Z_{2}, Z_{3}, Z_{4} and Z_{9} have the similar effects on the polarization efficiency and the effects are much less than Z_{1}. The other components, from Z_{5} to Z_{8}, have the smallest effects out of the 9 terms. According to the analysis of the polarization efficiency in Fig. 6, if the incident signals are 45deg polarized, it is evident that the linear diattenuation and retardance along the coordinate axes have more effects than the other aberrations on the heterodyne efficiency. The further investigation shows that the Z_{1} components of the x/y linear diattenuation and retardance have the greatest effects on the coherent detection system.

#### 4.2 Incident signals are right-handed circular polarized

If the incident signals are right-handed circular polarized, the Jones vector is expressed as${\widehat{e}}_{in}=1/\sqrt{2}{\left[\begin{array}{cc}1& i\end{array}\right]}^{T}$. By substituting ${\widehat{e}}_{in}$into Eqs. (5) and (6) and Eq. (8), the polarization efficiency ${\gamma}_{pol}$ is written as

The calculations of the polarization efficiency for right-handed circular polarized signals resemble the calculations for 45deg polarized signals. Figures 7–10 show the effects of diattenuation and retardance on the output polarization states when the incident signals are right-handed circular polarized. These line segments and ellipses indicate the magnitudes and orientations of the output polarizations. The separate scalar Zernike expansion coefficients are also set to 0.5.

By substituting the Jones matrices ${J}_{sys}$ and Jones vectors ${\widehat{e}}_{in}$ into Eq. (5), the orientation angles and ellipticities of the elliptical polarization states are obtained. In Fig. 7, when the incident signals are right-handed circular polarized, the transmission of one axis converts the circular polarization into elliptical polarization, and the corresponding orientation of the elliptical polarization is parallel to the transmission axis. The ellipticity of the elliptical polarization is related to the magnitude of the diattenuation. The larger diattenuation corresponds to the smaller ellipticity. In Fig. 8, the orientation angles of the elliptical polarization are 45deg or 135deg, and the larger diattenuation results in the smaller ellipticity. In Fig. 9, ${r}_{1}>0$ indicates that the phase of y-axis advances before x-axis, and produces the 135deg elliptical polarization. ${r}_{1}<0$ produces to the 45deg elliptical polarization. The ellipticity of each ellipse on the map is related to the magnitude of the retardance. The larger retardance corresponds to the smaller ellipticity. In Fig. 10, ${r}_{2}>0$ corresponds to the x-axis elliptical polarization, and ${r}_{2}<0$ corresponds to the y-axis elliptical polarization. The larger magnitude of retardance corresponds to the smaller ellipticity.

When the incident signals are right-handed circular polarized, the polarization efficiency is analyzed by varying the magnitudes of each scalar Zernike coefficient in Eqs. (19) and (20). Figure 11 illustrates the reduction of polarization efficiency with the increment of the diattenuation and retardance.

In Figs. 11(a) and 11(d), the polarization efficiency ${\gamma}_{pol}$ maintains 50% with the increment of the diattenuation and retardance. The orientation angles of the major axis of the elliptical polarization shown in Figs. 7 and 10 are 0deg and 90deg, and the projection polarizations of the elliptical polarization to the 45deg polarizer are always 50%. Hence, the diattenuation and retardance alter the ellipticities of the output polarization states, but the polarization efficiency is always 50%.

In Figs. 11(b) and 11(c), the polarization efficiency ${\gamma}_{pol}$ varies with the increment of the diattenuation and retardance. The increase and decrease of the polarization efficiency are related to the scalar Zernike coefficients and their corresponding magnitudes. In Figs. 8 and 9, the orientation angles of the major axis of the elliptical polarization are 45deg and 135deg. If the orientation angle is 45deg and parallel to the 45deg polarizer, the major axis of the elliptical polarization can pass through the 45deg polarizer. Under this circumstance, the smaller ellipticity leads to the smaller energy loss. However, if the orientation angle is 135deg, the minor axis of the elliptical polarization can pass through the polarizer, and the smaller ellipticity leads to the greater energy loss. Some of the elliptical polarizations are converted into linear polarizations shown in Figs. 8 and 9, if the orientation angles are 135deg, the power of the signal after passing through the 45deg polarizer is zero.

By decomposing ${r}_{1}$ and ${d}_{2}$ into the scalar Zernike polynomials and calculating the corresponding polarization efficiency, Figs. 11(b) and 11(c) show that Z_{1} has the greatest effect on heterodyne efficiency. Z_{2}, Z_{3}, Z_{4} and Z_{6} have the similar effects on the polarization efficiency and the effects are much less than Z_{1}. Compared with Z_{1}, Z_{9} has the opposite effects on the polarization efficiency. The other components, Z_{5}, Z_{7} and Z_{8}, have the smallest effects out of the 9 terms.

From the analysis made above, we can conclude that the polarization efficiency ${\gamma}_{pol}$ in Eqs. (8), (22) and (23) can provide an evaluation of the polarization matching in coherent detection system. The effects of the polarization aberrations on the heterodyne efficiency mainly depend on the polarization states of the incident signals. For the real coherent laser communication system, the heterodyne efficiency that considering the polarization aberrations of optical system is not performed yet. Hence, in the next section, the heterodyne efficiency for the real coherent laser communication system is analyzed. The heterodyne efficiency is further used to minimize the polarization aberrations of optical system.

## 5. Heterodyne efficiency analysis of a coherent laser communication System

In this section, a coherent laser communication system is modeled, and the polarization aberration and heterodyne efficiency are analyzed by using the method described in Sections 2-4. The optical system is used for the space laser communication. The system consists of four mirrors and two rotation axes, which are named as planar mirror M1, planar mirror M2, rotation axis 1, rotation axis 2, primary mirror (PM) and secondary mirror (SM). M1, M2 and two perpendicular rotation axes constitute the periscopic scanner. The rotation of the two axes will deflect the laser beam along the global sphere system and the detailed description is presented in reference [21]. In this paper, the criterion to describe rotations is defined as follows: the rotation axis 1, left-handed about the y axis, only controls the rotation of mirror M1. The corresponding rotation angle is referred to as the elevation angle $\beta $. The rotation axis 2, left-handed about the z axis, controls the total rotation of the combination of M1 and M2. The corresponding rotation angle is referred to as the azimuth angle $\alpha $. The PM and SM are both off-axis parabolic mirrors with off-axis values 220 mm and 22 mm, respectively. The off-axis value means the offset of the mirror center decentered from the optical axis. The 300mm diameter PM is the aperture stop of the system. The magnification of the optical system is 0.1. Table 3 shows the detailed optical prescription of the coherent detection system. The operating wavelength is 1550 nm. The layout of the optical system is depicted in Fig. 12.

The analysis of the optical systems is mainly focused on the center field. The four mirrors are coated with the bare gold films. In reference [21], it is noted that the diattenuation and retardance of the periscopic scanner vary continuously and periodically with the rotation angles $\alpha $ and $\beta $. In Fig. 12, the polarization aberrations of the optical system are generated by the periscopic scanner and the two parabolic mirrors. Therefore, the polarization aberrations of the combined system also vary continuously and periodically with the rotation angles. The rotations in $\alpha $ change the orientations of diattenuation and retardance and keep their magnitudes constant. The rotations in $\beta $ not only change the magnitudes, but also change the orientations of the polarization aberrations.

Assuming that the signal and local oscillator fields are Gaussian beams which are initially matched in both amplitude and phase. The signal and local oscillator beams are complete polarized light in this paper. As assumed in Section 4, the heterodyne efficiency ${\gamma}_{i.f.}$ is equal to 100%. The modified heterodyne efficiency $\gamma $ is equal to the polarization efficiency ${\gamma}_{pol}$. The polarization properties of the optical system are analyzed by performing the polarization ray-tracings [22,23]. The polarization ray-tracing calculus is built and performed by the optical design software Code V. The algorithm codes are compiled by using the macro languages in Code V. The polarization transformation matrices ${J}_{sys}$ are obtained and decomposed by the Polar and Pauli-Zernike decomposition algorithms. By substituting ${\widehat{e}}_{in}$ and ${J}_{sys}$ into Eqs. (5)–(8), the polarization efficiency is calculated and analyzed with the rotations of the two axes. By varying the rotations $\alpha $ and $\beta $ in the range [0°, 360°] with steps of 5°,the polarization efficiencies of this coherent laser communication system are calculated. Figure 13 illustrates the variations of the polarization efficiency when the incident signals are 45deg polarized, and Fig. 14 illustrates the variations of the polarization efficiency when the incident signals are right-handed circular polarized.

Figures 13 and 14 show the polarization efficiency varies continuously and periodically with $\alpha $ and $\beta $. The polarization efficiency ${\gamma}_{pol}$ becomes functions of the scan angles ($\alpha $,$\beta $). In Fig. 13, the maximum polarization efficiencies are close to 100% and the maximum values occur at $\alpha +\beta ={0}^{\circ},{180}^{\circ},{360}^{\circ}$. The minimum polarization efficiencies are close to zero, and the minimums occur at $\alpha +\beta ={90}^{\circ},{270}^{\circ}$. In Fig. 14, the maximum polarization efficiencies are close to 0.66, and the maximums occur at $\beta ={0}^{\circ},{180}^{\circ},{360}^{\circ}$. The minimum polarization efficiencies are close to 0.49, and the minimums occur at $\beta ={90}^{\circ},{270}^{\circ}$. Moreover, the maximums and minimums vary continuously and periodically with the rotations of the azimuth angle $\alpha $ when $\beta ={0}^{\circ},{90}^{\circ},{180}^{\circ},{270}^{\circ},{360}^{\circ}$. In Table 1, the diattenuation and retardance of the optical system change the orientation angles and ellipticities of the output polarization states. The rotator ${J}_{rot}$ rotates the output polarization states. The combinations of these effects induce the variations of the polarization efficiencies. In Fig. 13, the maximum polarization efficiencies are close to 100% and the minimums are zero. However, in Fig. 14, the maximum polarization efficiencies are close to 0.66 and the minimums are close to 0.49. Comparing these two peak-to-valley values, when the incident signals are right-handed circular polarized, this system will obtain moderate and stable polarization efficiency. If the incident signals are 45deg polarized, the polarization variations may be changed by adjusting the polarization of the transmitted signals or the LO beam at the receiver. However, for the space laser communication link, the orientation angles of the signals are complicated and unpredictable, which make it difficult to adjust the polarization states of the transmitted signals or the LOs at the receiver. For the coherent laser communication link, the severe variations of the polarization efficiency will degrade the quality of communication; even will result in the interruption of the communication. Hence, the moderate polarization efficiency ensures good communication quality without interruption. Therefore, for the coherent laser communication, the circular polarized signals are much better than the linear polarized signals. In references 2,24, the signals are circular polarized in the space laser communication terminals. Another benefit for circular polarization is that the received weak laser beam can be isolated form the transmitted powerful laser beam with orthogonal polarization by using a quarter wave-plate in front of the internal optics [1,2,24].

The fluctuations in the polarization efficiency can cause the fluctuations in the communication link, which in turn affects the communication quality. When the incident signals are right-handed circular polarized, if the optical system is an ideal nonpolarizing system, the polarization efficiency remains unchanged with the rotations of the periscopic scanner. The fluctuations of the polarization efficiency in Fig. 14 indicate that the optical system has polarization aberrations. Hence, the gold-coated optical system is not a good choice for the coherent laser communication. To evaluate the polarization aberrations of the four mirrors, we choose three groups of rotation angles ($\alpha $,$\beta $) and perform the polarization ray-tracings. The Jones matrices ${J}_{sys}$ are decomposed by the Polar and Pauli-Zernike decomposition algorithms. The Pauli and scalar Zernike coefficients are calculated and shown in Figs. 15–17. The ellipticities of the total diattenuation and retardance are calculated, and the Pauli coefficients ${d}_{3}$ and ${r}_{3}$ are plotted graphically. These three groups of rotation angles ($\alpha $,$\beta $) are (0°, 0°), (0°, 90°), (45°, 45°). The calculations of the scalar Zernike coefficients are performed by the least square algorithm.

All the diattenuation and retardance shown in Figs. 15–17 are viewed from the exit pupil, so all the maps are compared directly in the same pupil. The maximum diattenuation and retardance are obtained when the elevation angle $\beta ={0}^{\circ}$ and the minimum values are obtained when $\beta ={90}^{\circ}$ [20,21]. The scalar Zernike coefficients for diattenuation in Figs. 15–17 are in the order of 10^{−3}, and have small effects on the polarization states. The Pauli coefficients ${d}_{3}$ and ${r}_{3}$ in Figs. 15–17 are also in the order of 10^{−4}, and can be ignored safely. Hence, the retardance is the major factor that affects the output polarization state. The optical system in Fig. 12 consists of two rotating axes. Therefore, when performing the Polar decompositions of the Jones matrix${J}_{sys}$, the rotation angle $\zeta $ of ${J}_{rot}$ in Eq. (11) is no longer zero.

The two mirrors of the periscopic scanner are perfectly flat with no deformation occurring while rotating. Hence, the polarization aberrations induced by the scanner are the Z_{1} components of the scalar Zernike coefficients shown in Figs. 15–17. The off-axis parabolic mirrors mainly induce the rest components of the scalar Zernike coefficients. The results in Section 4 indicate that the Pauli coefficient ${r}_{1}$ affects the heterodyne efficiency and ${r}_{2}$ has no effect on the heterodyne efficiency. In Fig. 11(b), the Z_{1} component of ${r}_{1}$ has the greatest effect on heterodyne efficiency. When ($\alpha $,$\beta $) = (0°, 0°), the Z_{1} component of the Pauli coefficients ${r}_{1}$ is −0.143. When ($\alpha $,$\beta $) = (0°, 90°), the corresponding value is 0.008. Comparing the magnitudes of the Z_{1} components, the Z_{1} component of the retardance is mainly induced by the periscopic scanner. The magnitudes of the rest components of the scalar Zernike coefficients in Figs. 15–17 are much smaller than the Z_{1} component. Hence, the polarization efficiency is mainly affected by the retardance of the periscopic scanner. In Fig. 17, when ($\alpha $,$\beta $) = (45°, 45°), the Z_{1} components of ${r}_{1}$ is 0.078. The rotations of $\alpha $ and $\beta $ change the Z_{1} component of the Pauli coefficient ${r}_{1}$. The retardance converts the circular polarization into the elliptical polarization and the rotator ${J}_{rot}$ rotates the orientation angle of the elliptical polarization to different angle. The combinations of these two effects result in the variations of the polarization efficiency, which are shown in Fig. 14.

Based on the modified heterodyne efficiency and the decompositions of the polarization aberration, the analysis indicates that the fluctuations of the polarization efficiency are mainly affected by the polarization aberrations of the periscopic scanner. In reference [21], the results show that aluminum film is more appropriate than gold film for the polarization maintaining periscopic scanner. Two different coatings are used to verify the effects the varied polarization aberrations have on the heterodyne efficiency. Figure 18 shows the variations of the polarization efficiency. These two different coatings are aluminum film and polarization-maintaining (PM) film. The aluminum film is the bare metallic film. The polarization-maintaining film is designed by the software Code V. The polarization-maintaining film is optimized at the specific conditions that the wavelength is 1550 nm and the incident angle is 45deg. The diattenuation and retardance of the film are well controlled. The polarization-maintaining film is used to demonstrate the stability of the polarization efficiency when the polarization aberrations are carefully controlled. The incident signals are right-handed circular polarized.

The calculations of the polarization efficiency for aluminum and polarization-maintaining coatings resemble the calculations for gold coatings. In Fig. 18(a), the maximum polarization efficiencies are close to 0.60, and the minimum values are close to 0.5. The maximum polarization efficiencies occur at $\beta ={0}^{\circ},{180}^{\circ},{360}^{\circ}$, and the minimums occur at $\beta ={90}^{\circ},{270}^{\circ}$. The variations of the polarization efficiency for aluminum coating are similar to the variations shown in Fig. 14. In Fig. 14, the maximum and minimum polarization efficiencies for gold coating are 0.66 and 0.49. However, in Fig. 18(b), the maximum polarization efficiencies are close to 0.51, and the minimum values are close to 0.49. Comparing these three different peak-to-valley values, the polarization-maintaining coating induces the smallest variations of the polarization efficiency. According to the variations of heterodyne efficiency, the polarization-maintaining coating can provide a stable heterodyne detection. When the periscopic scanner is coated with the PM coating, it is noted that the slight fluctuations of the polarization efficiency are mainly affected by the residual polarization aberrations of the periscopic scanner and the off-axis mirrors.

The analysis of the polarization efficiency for the optical system is used to illustrate the significance of the modified heterodyne efficiency. The modified heterodyne efficiency provides a method to evaluate the effects of the polarization aberrations on the coherent detection, and in turn can be used to evaluate the magnitudes of polarization aberrations. The Polar and Pauli-Zernike decompositions provide an analytical method to analyze the polarization aberrations of optical system. The combination of these two methods can be used to minimize the polarization aberrations and further to improve the performance of the coherent laser communication system.

## 6. Conclusion

Heterodyne efficiency is an important indicator of the performance of the coherent laser communication. A modified heterodyne efficiency of mixing signal and local oscillator beams has been derived by considering the polarization aberrations of optical system. Based on the Polar and Pauli-Zernike decompositions of Jones matrix, we derive the analytical expressions for the polarization aberrations of optical system, and incorporate them into the calculations of heterodyne efficiency. By introducing the different distributions of polarization aberrations, the modified heterodyne efficiencies are calculated and compared for different input signals. When the incident signals are 45deg polarized and right-handed circular polarized, the heterodyne efficiencies are analyzed separately by considering the separate first 9 scalar Zernike polynomials. The results show that when the incident signals are 45deg polarized, the Z_{1} components of the linear diattenuation and retardance along the x/y axis have the greatest effects on the heterodyne efficiency. However, if the incident signals are right-handed circular polarized, the heterodyne efficiency is mainly affected by the Z_{1} component of the retardance along x/y axis or the Z_{1} component of the diattenuation along 45/135deg direction. Based on the detailed analysis, it is concluded that the effects of the polarization aberrations on the heterodyne efficiency are related to the polarization states of the input signals, and the modified heterodyne efficiency can provide a comprehensive description of the heterodyne efficiency.

Based on the modified heterodyne efficiency formula, a specific coherent laser communication system is modeled and analyzed. When the incident signals are 45deg and right-handed circular polarized, the heterodyne efficiencies of the system are analyzed by considering all possible scan angles. The comparisons of the heterodyne efficiencies show that the right-handed circular signals are more suitable for the coherent laser communication than the linear polarized signals. By using the Polar and Pauli-Zernike decompositions of the polarization aberrations, the results show that the heterodyne efficiencies are mainly affected by the retardance of the periscopic scanner. According to this conclusion, three different coatings are used to minimize the polarization aberrations of the optical system, and the corresponding heterodyne efficiencies are investigated separately. The analysis shows that the modified heterodyne efficiency not only can provide a complete evaluation of the heterodyne efficiency, but also can be used to evaluate and minimize the polarization aberrations of optical system, and to further improve the performance of the coherent laser communication.

## Funding

National High Technology Research & Development Program of China (Grant NO.2011AA12A103).

## References and links

**1. **K. Böhmer, M. Gregory, F. Heine, H. Kämpfner, R. Lange, M. Lutzer, and R. Meyer, “Laser communication terminals for the European data relay system,” Proc. SPIE **8246**, 82460D (2012). [CrossRef]

**2. **M. Gregory, F. Heine, H. Kämpfner, R. Lange, M. Lutzer, and R. Meyer, “Commercial optical inter-satellite communication at high data rates,” Opt. Eng. **51**(3), 031202 (2012). [CrossRef]

**3. **D. Fink, “Coherent detection signal-to-noise,” Appl. Opt. **14**(3), 689–690 (1975). [CrossRef] [PubMed]

**4. **T. Takenaka, K. Tanaka, and O. Fukumitsu, “Signal-to-noise ratio in optical heterodyne detection for Gaussian fields,” Appl. Opt. **17**(21), 3466–3471 (1978). [CrossRef] [PubMed]

**5. **K. Tanaka and N. Ohta, “Effects of tilt and offset of signal field on heterodyne efficiency,” Appl. Opt. **26**(4), 627–632 (1987). [CrossRef] [PubMed]

**6. **K. Tanaka and N. Saga, “Maximum heterodyne efficiency of optical heterodyne detection in the presence of background radiation,” Appl. Opt. **23**(21), 3901–3904 (1984). [CrossRef] [PubMed]

**7. **S. C. Cohen, “Heterodyne detection: phase front alignment, beam spot size, and detector uniformity,” Appl. Opt. **14**(8), 1953–1959 (1975). [CrossRef] [PubMed]

**8. **D. Chambers, “Modeling of heterodyne efficiency for coherent laser radar in the presence of aberrations,” Opt. Express **1**(3), 60–67 (1997). [CrossRef] [PubMed]

**9. **D. Delautre, S. Breugnot, and V. Laude, “Measurement of the sensitivity of heterodyne detection to aberrations using programmable liquid-crystal modulator,” Opt. Commun. **160**(1-3), 61–65 (1999). [CrossRef]

**10. **G. Yun, K. Crabtree, and R. A. Chipman, “Skew aberration: a form of polarization aberration,” Opt. Lett. **36**(20), 4062–4064 (2011). [CrossRef] [PubMed]

**11. **N. Nabavi and T. J. Hall, “Symmetric signal and local oscillator polarization diverse coherent optical receiver,” Opt. Express **24**(3), 2391–2405 (2016). [CrossRef] [PubMed]

**12. **E. Ip, A. P. T. Lau, D. J. F. Barros, and J. M. Kahn, “Coherent detection in optical fiber systems,” Opt. Express **16**(2), 753–791 (2008). [CrossRef] [PubMed]

**13. **M. Salem and J. P. Rolland, “Effects of coherence and polarization changes on the heterodyne detection of stochastic beams propagating in free space,” Opt. Commun. **281**(20), 5083–5091 (2008). [CrossRef]

**14. **R. Garreis and C. Zeiss, “90° optical hybrid for coherent receivers,” Proc. SPIE **1522**, 210–219 (1991). [CrossRef]

**15. **R. A. Chipman, “Polarization aberrations,” Ph.D. thesis (University of Arizona, 1987).

**16. **J. Ruoff and M. Totzeck, “Orientation Zernike polynomials: a useful way to describe the polarization effects of optical imaging systems,” J. Micro/Nanolith. MEMS MOEMS **8**(3), 031404 (2009). [CrossRef]

**17. **G. R. McIntyre, J. Kye, H. Levinson, and A. R. Neureuther, “Polarization aberrations in hyper-numerical-aperture projection printing: a comparison of various representations,” J. Microlith., Microfab., Microsyst. **5**(3), 033001 (2006).

**18. **N. Yamamoto, J. Kye, and H. J. Levison, “Polarization aberration analysis using Pauli-Zernike representation,” Proc. SPIE **6520**, 65200Y (2007). [CrossRef]

**19. **B. Geh, J. Ruoff, J. Zimmermann, P. Gräupner, M. Totzeck, M. Mengel, U. Hempelmann, and E. Schmitt-Weaver, “The impact of projection lens polarization properties on lithographic process at hyper-NA,” Proc. SPIE **6520**, 65200F (2007). [CrossRef]

**20. **W. S. Tiffany Lam, R. Chipman, and R. A. Chipman, “Balancing polarization aberrations in crossed fold mirrors,” Appl. Opt. **54**(11), 3236–3245 (2015). [CrossRef] [PubMed]

**21. **Y. Yang and C. Yan, “Polarization property analysis of a periscopic scanner with three-dimensional polarization ray-tracing calculus,” Appl. Opt. **55**(6), 1343–1350 (2016). [CrossRef] [PubMed]

**22. **G. Yun, K. Crabtree, and R. A. Chipman, “Three-dimensional polarization ray-tracing calculus I: definition and diattenuation,” Appl. Opt. **50**(18), 2855–2865 (2011). [CrossRef] [PubMed]

**23. **G. Yun, S. C. McClain, and R. A. Chipman, “Three-dimensional polarization ray-tracing calculus II: retardance,” Appl. Opt. **50**(18), 2866–2874 (2011). [CrossRef] [PubMed]

**24. **M. Toyoshima, H. Takenaka, Y. Shoji, Y. Takayama, Y. Koyama, and H. Kunimori, “Polarization measurements through space-to-ground atmospheric propagation paths by using a highly polarized laser source in space,” Opt. Express **17**(25), 22333–22340 (2009). [CrossRef] [PubMed]