## Abstract

We analytically obtain the exact fundamental limit of 4-level amplitude shifted keying formats (4ASK) with direct detection optically preamplified receivers. The optimum multilevel spacing and the corresponding decision thresholds, which depend both on the signal to noise ratio and optical bandwidth, are obtained numerically considering the Chi-squared distribution of each level. The quantum limit under the optimum level spacing is 127.5 photons/bit, which is about 0.2 dB smaller than the results by the Gaussian approximation. Over a broad range of the signal to noise ratio and the optical bandwidth, we have found that not only the bit error rate but also the optimum level spacing are well predicted by Gaussian method, although the three decision thresholds are all underestimated.

© 2007 Optical Society of America

## 1. Introduction

With the rapid growth of Internet traffic congestion, efficiently using signal bandwidth is essential to increase the transmission capacity over already installed optical fibers and amplifiers. Multilevel amplitude-shifted-keying (ASK) format such as 4ASK has recently attracted much attention because it offers an increased spectral efficiency at a reduced symbol rate without too complex transmitting and receiving ends [1–5]. Benefited from the reduced symbol rate, the tolerances to chromatic dispersion (CD) and polarization mode dispersion (PMD) are enhanced compared to traditional binary format [1, 3, 5]. In addition, the logic circuit at both the transmitter and receiver can be operated with half the bit-rate speed, thus lowering the electronic cost [5]. One interesting problem of 4ASK format is the level spacing with an optically pre-amplified receiver, which is directly related to the bit error rate of the system. Extensive works based on the optimum level spacing have been studied [1, 6, 7]. In these papers, the noises are all simplified as Gaussian distribution to pursue the optimum level spacing and the corresponding bit error rate. Unfortunately, the accuracy and the appropriateness of Gaussian approximation to the optimum level spacing and the bit error rate have not been confirmed yet. Although the Gaussian approximation is good for calculating bit error rate in binary format, there has no report regarding the Gaussian method in predicting the optimum level in 4ASK format. An exact analysis to 4ASK will be necessary since establishing accurate estimates of the performance can yield useful insights in practical system design. Recently, the exact performance of 4ASK transmission systems was explored by using Karhunen–Loeve series expansion (KLSE) and saddle point approximation method, which generally speaking is semi-analytical and not explicit [5]. With this method, the optimum multilevel spacing of the 4ASK is achieved by scanning the middle two energy levels for each pair of optical and electrical bandwidth to get the minimum bit error rate. They also provide the optimum receiving sensitivity, which is generally called quantum limit [8], by KLSE method. However, searching the optimum multilevel spacing and threshold levels by scanning each parameter to achieve the minimum error rate would be very time consuming. Therefore, an explicit and rigorous expression for the bit error rate, which has never been investigated to the authors’ knowledge, is required since it provides fast computing and gives a direct relation between the system inputs and the overall performance.

In this paper, we give an exact formula of bit error rate for an optical 4ASK format with optimum level spacing using an optically pre-amplified receiver. This formula is derived under the assumption that an ideal optical filter and an integrate and dump circuit are used at the receiver. The quantum limit for a 4ASK signal is derived as 127.5 photons/bit, considering the chi-squared distribution of each level. The exact bit error rates and optimum level spacing are compared to those evaluated by Gaussian approximation. We conclude that the Gaussian method not only give a good evaluation to the bit error rate but also the optimum level spacing for a 4ASK format, although the three optimum decision thresholds are all underestimated. We further confirm that the inter-symbol interference (ISI) considered Gaussian approximation used for binary format [9] would still works for a practical 4ASK format when suffered by ISI.

## 2. The optimum level and quantum limit for 4ASK

We firstly consider the error probability formula of a receiver consisting of an ideal optical filter, a square law detector and an electrical integrate and dump circuit, and then the quantum limit will be the specific case of this formula. We assume that the ASE is the dominant noise and ignore the other noises. The baseband envelope of the transmitter output can be written as:

where T is the symbol duration,${a}_{k}\in \left\{\sqrt{{E}_{y,}}y=\mathrm{0,1,2,3}\right\}$ with $\sqrt{{E}_{0}}=0<\sqrt{{E}_{1}}<\sqrt{{E}_{2}}<\sqrt{{E}_{3}}$ is the modulated amplitude corresponding to the *k _{th}* symbol,

*p*(

*t*) is the pulse shape satisfying ${\int}_{\frac{-T}{2}}^{\frac{T}{2}}{p}^{2}\left(t\right)\mathrm{dt}=1$, and thus ${E}_{y}={\int}_{\frac{-T}{2}}^{\frac{T}{2}}{s}^{2}\left(t\right)\mathrm{dt}$ represents the symbol energy. For each symbol, the signal

*s*(

*t*) and the filtered noise

*n*(

*t*) can be modeled by orthonormal functions

*ϕ*(

_{i}*t*) over the pulse interval T [10–12]:$s\left(t\right)={\sum}_{i=1}^{2M}{s}_{i}{\varphi}_{i}\left(t\right)$ with energy of ${\sum}_{i=1}^{2M}{{s}_{i}}^{2}\in \{{E}_{0},{E}_{1},{E}_{2},{E}_{3}\}$ and the filtered noise $n\left(t\right)={\sum}_{i=1}^{2M}{n}_{i}{\varphi}_{i}\left(t\right)$, where

*n*’s are independent Gaussian random variables with mean and variance of 0 and

_{i}*N*

_{0}/2, respectively. M = BT is proportional to the optical bandwidth B. After the electrical integrate and dump circuit, the probability density function (pdf) of the decision variable

*χ*with symbol energy of

*E*

_{y=1,2,3}> 0 is [10–12]:

and that with symbol energy of *E*
_{0} = 0 is :

where the decision random variable *χ* has been normalized to the noise level *N*
_{0} and *S _{y}* =

*E*/

_{y}*N*

_{0}is the normalized symbol energy. Assuming that Gray coding is used, the symbol error rate

*P*(

*e*|

*S*) for each symbol

_{i}*S*can be written as:

_{i}$$\phantom{\rule{.2em}{0ex}}P\left(e\mid {S}_{i}\right)=1-{Q}_{M}(\sqrt{{2S}_{i}},\sqrt{{2\gamma}_{i}})+{Q}_{M}(\sqrt{{2S}_{i}},\sqrt{{2\gamma}_{i+1}})\phantom{\rule{.2em}{0ex}}\mathrm{for}\phantom{\rule{.2em}{0ex}}i=\mathrm{1,2}$$

$$P\left(e\mid {S}_{3}\right)=1-{Q}_{M}(\sqrt{{2S}_{3}},\sqrt{{2\gamma}_{3}}))\phantom{\rule{10.5em}{0ex}}$$

where *γ _{i}* is the threshold normalized to the noise level

*N*

_{0}, ${Q}_{M}\left(a,b\right)={\int}_{b}^{\infty}\frac{{x}^{M}}{{a}^{M-1}}\mathrm{exp}\left(-\frac{\left({x}^{2}+{a}^{2}\right)}{2}\right){I}_{M-1}\left(\mathrm{ax}\right)\mathrm{dx}.$. is the Marcum’s Q function, and

*I*is the modified Bessel function of the first kind. The bit error rate (BER) is almost equal to half the averaged symbol error rate under a good signal to noise ratio [13, p. 503]:

_{M}Usually the levels of *S*
_{0} = 0 and *S*
_{3} are known and we only need to find the values of *S*
_{1,2} and *γ*
_{1} ~ *γ*
_{3} to minimize the error rate. The minimum error rate implies that the partial derivatives of *P _{e}* with respect to

*S*

_{1,2}and

*γ*

_{1}~

*γ*

_{3}are all equal to zeros, which means:

The derivatives with respect to the thresholds are known as the maximum-likelihood principle and those with respect to *S*
_{i=1,2} determine the optimum locations of the middle two levels within the given thresholds (*γ _{i}* and

*γ*

_{i+1}). Using the two partial-derivative properties of Marcum’s Q functions:

where Eq. (8) can be easily proved with the two following equalities:

Using Eqs. (4)–(8), we can obtain the following five equalities:

$$\frac{{\partial P}_{e}}{{\partial S}_{1}}=0=>{\left(\frac{{b}_{1}}{{b}_{2}}\right)}^{M}{e}^{-\frac{1}{2}\left({{b}_{1}}^{2}-{{b}_{2}}^{2}\right)}\frac{{I}_{M}\left({a}_{1}{b}_{1}\right)}{{I}_{M}\left({a}_{1}{b}_{2}\right)}=1\phantom{\rule{1.2em}{0ex}}$$

$$\frac{{\partial P}_{e}}{{\partial \gamma}_{2}}=0=>{\left(\frac{{a}_{2}}{{a}_{1}}\right)}^{M-1}{e}^{-\frac{1}{2}\left({{a}_{1}}^{2}-{{a}_{2}}^{2}\right)}\frac{{I}_{M-1}\left({a}_{1}{b}_{2}\right)}{{I}_{M-1}\left({a}_{2}{b}_{2}\right)}=1$$

$$\phantom{\rule{1.2em}{0ex}}\frac{{\partial P}_{e}}{{\partial S}_{2}}=0=>{\left(\frac{{b}_{2}}{{b}_{3}}\right)}^{M}{e}^{-\frac{1}{2}\left({{b}_{2}}^{2}-{{b}_{3}}^{2}\right)}\frac{{I}_{M}\left({a}_{2}{b}_{2}\right)}{{I}_{M}\left({a}_{2}{b}_{3}\right)}=1\phantom{\rule{3.2em}{0ex}}$$

$$\frac{{\partial P}_{e}}{{\partial \gamma}_{3}}=0=>{\left(\frac{{a}_{3}}{{a}_{2}}\right)}^{M-1}{e}^{-\frac{1}{2}\left({{a}_{2}}^{2}-{{a}_{3}}^{2}\right)}\frac{{I}_{M-1}\left({a}_{2}{b}_{3}\right)}{{I}_{M-1}\left({a}_{3}{b}_{3}\right)}=1$$

where
${a}_{i}=\sqrt{2{S}_{i}}$
and
${b}_{i}=\sqrt{2{\gamma}_{i}}$
. With Eq. (9), we can uniquely obtain the middle two levels *S*
_{1,2} and the thresholds *γ*
_{1} ~ *γ*
_{3} . Note that to avoid the overflow of the modified Bessel function, the exponential term should be shifted into the integrand of the integral formula of the modified Bessel function. An alternative method avoiding over- and underflow problems is to put these derivatives in the moment generating function (MGF) domain using the method similar to that described in Ref. [14].

Although Eq. (5) is for ideal optical filter and electrical integration circuit, it is with the same form as that of an optical matched filter *h*(*t*) = *p*
^{*}(*T* - *t*) when M = 1 [10, 15]. For the matched filter, the normalized multilevel spacing and the optimum decision thresholds under a high signal to noise ratio (
$SNR={E}_{b}/{N}_{0}=1/8*{\displaystyle {\sum}_{i=0}^{3}{S}_{i}}$
, where *E _{b}* is the averaged energy per bit) can be easily solved as (

*L*

_{0},

*r*

_{1},

*L*

_{1},

*r*

_{2},

*L*

_{2},

*r*

_{3},

*L*

_{3}) = (0, 1/36, 1/9, 1/4, 4/9, 25/36, 1), where

*L*=

_{i}*S*/

_{i}*S*

_{3}and

*r*=(

_{i}*γ*

_{1}-

*M*)/

*S*

_{3}. For here the crude approximation

*I*

_{0}(

*x*)≈

*e*and the assumption of

^{x}*f*

_{S0}(

*γ*

_{1}) =

*f*

_{S1}(

*γ*

_{1}) =

*f*

_{S1}(

*γ*

_{2}) =

*f*

_{S2}(

*γ*

_{2}) =

*f*

_{S2}(

*γ*

_{3}) =

*f*

_{S3}(

*γ*

_{3}) are used. Note that the squared roots of the asymptotic optimum levels and thresholds are equally spaced as ( $\sqrt{{L}_{0}},\sqrt{{r}_{1}},\sqrt{{L}_{1}},\sqrt{{r}_{2}},\sqrt{{L}_{2}},\sqrt{{r}_{3}},\sqrt{{L}_{3}},$ ) = (0/6, 1/6, 2/6, 3/6, 4/6, 5/6, 6/6).

The minimum bit error rate and the optimum multilevel spacing can also be approximated by using the Gaussian method. The means and variances of the random variable *χ* for each
symbol are *μ _{i}* =

*M*+

*S*and σ

_{i}_{i}

^{2}=

*M*+ 2

*S*, respectively [10]. The optimized multilevel spacing can be derived by equalizing the

_{i}*Q*factors defined as

_{i}*Q*= (

_{i}*μ*-

_{i}*μ*

_{i-1}) /(σ

_{i}+ σ

_{i-1}) of the three eyes. The optimum thresholds of the three eyes are located at

*γ*= (

_{i}*μ*

_{i-1}+

*μ*σ

_{i}_{i-1}) /(σ

_{i}+ (σ

_{i-1}). The BER with Gaussian method is written as :

where *Q*
_{1}=*Q*
_{2}=*Q*
_{3} = *Q*. Note that each symbol carriers two bits.

## 3. Results and discussions

For the results shown here we only consider the ASE noise with the same polarization as the signal. The ASE orthogonal to signal can be easily included by doubling the value of M [10, 11]. The BER versus the photons/bit are presented in Fig .1 with M = 1, 10, 30, 50, where the photons/bit relates to SNR as photons/bit = *SNR* = *E _{b}*/

*N*[16]. The BER approximated by Gaussian scheme only slightly overestimate those of exact solutions. For various optical bandwidths M, the sensitivity differences between the chi-squared and Gaussian distributions are less than 0.2 dB. For the optical matched filter, ie: M = 1, the photons/bit and SNR required to achieve a BER of 10

_{o}^{-9}is about 127.5 and 21 dB, respectively, and this figures out the best performance that 4ASK format can reach under the optimum multilevel spacing. Figures 2(a)–2(d) shows the relation between SNR and optimum levels

*L*and the thresholds

_{i}*r*for various optical bandwidths M. The optimum levels and the thresholds are descending as SNR increases. For poor SNR and broader optical bandwidth M, the noise is less dependent to the signal and the optimum levels are closer to be equally spaced. For high SNR and narrower optical bandwidth, the signal dependent noise dominants and the levels and the thresholds will approach the asymptotic values (

_{i}*L*

_{0},

*L*

_{1},

*L*

_{2},

*L*

_{3},

*r*

_{1},

*r*

_{2},

*r*

_{3}) = (0, 1/9, 4/9, 1, 1/36, 1/4, 25/36).

Figure 2(e) shows the optimum spacing and thresholds at a BER of 10^{-9} as a function of M. The optimum multilevel spacing predicted by Gaussian method is quite close to those by exact solution over a broad range of SNR and M while the decision thresholds are slightly underestimated.

Figure 3 shows the relation between the optical bandwidth M and the required photons/bit (SNR) at a BER of 10^{-9}. The results of binary format, which we have reviewed from Ref. [10], are also shown for comparison. The worse performance for 4ASK has been explained and described in Ref. [1]. In both the cases of binary and quaternary formats, the SNR are overestimated by the Gaussian approximation for a broad range of M from 1 to 100. The results from Gaussian method are more accurate as M becomes larger in both formats due to the central limit theorem when M approaches infinity. We also found that the quaternary format is not that sensitive to the bandwidth of optical filter, M. Since the variance of *χ* for each symbol is σ^{2}
_{i} = *M* + 2*S _{i}*, the quaternary format with higher SNR dominating the variance will behave more insensitive to M. This leads to the result that the gain of binary format over quaternary varies from 5.2 to 3.5 dB as M increases from M =1 to 100. The gain ~5 dB of binary format described in previous reports [1, 16] comes from the result of using an optical filter.

In Fig. 4 we show the BER for the return-to zero 4ASK (RZ-4ASK) format considering the ISI effect. The data rate is operated at 20Gbps. The exact BER, which is calculated by the Karhunen-Loeve based method [17], is compared to that of the Gaussian approximation [1, 9]. The receiver uses a 2^{nd} order optical Gaussian band-pass and 5^{th} order electrical Bessel low-pass filters. The 3dB-bandwidths (BW) of optical and electrical filters are 25 and 7GHz, respectively. The results show that the BER calculated by the exact and Gaussian methods are with a difference of less than 0.2dB in both the cases of back to back (b2b) and after 30km standard single mode fiber (SSMF) transmission. This confirms the accuracy of using the Gaussian method for the 4ASK signal when the ISI exists.

## 4. Conclusions

We have evaluated the exact optimum multilevel spacing and the corresponding bit error rate for the 4-level amplitude-shift keying format in presence of optical pre-amplified direct-detection receiver, and compared these results with Gaussian approximation. The quantum limit is 127.5 photons/bit considering the exact noise characteristic on each level. We have found that with the Gaussian method, the optimum multilevel spacing and the bit error rate can be well predicted and approximated while the thresholds are all underestimated. The accuracy for evaluating the optimum levels and bit error rate also implies that the ISI-considered Gaussian approximation would still works for a practical 4ASK format with arbitrary optical and electrical filters. In addition, we point out that the 4ASK format is more insensitive to the optical bandwidth and the previously proposed ~5.2-dB gain of binary over quaternary format comes from a narrowband optical filter. Note that the method in this paper for the optimum symbol spacing can also be used in other advanced modulation format such as 8ASK, 8ADPSK or 16ADQPSK [18], in which the time-consuming scanning method would be inefficient.

## Acknowledgments

This work was supported in part by the National Science Council of R. O. C. under Contract NSC 95-2752-E-009-009-PAE, NSC 95-2221-E-155-072, and NSC 95-2221- E-155-059.

## References and links

**1. **S. Walklin and J. Conradi, “Multilevel signaling for increasing the reach of 10Gbps lightwave systems,” J. Lightwave Technol. **17**, 2235 (1999). [CrossRef]

**2. **S. K. Ibrahim, S. Bhandare, and R. Noe, “Narrowband 20-Gbps quaternary intensity modulation generated by duobinary 10-Gbps modulaltion in 2 quadratures,” ECOC 2005 Paper Th 2.6.5.

**3. **S. K. Ibrahim, S. Bhandare, and R. Noe, “20-Gbps quaternary intensity modulation based on duobinary modulation with unequal amplitude in two polarizations,” IEEE Photon. Technol. Lett. **14**, 1482 (2006). [CrossRef]

**4. **S. K. Ibrahim, S. Bhandare, and R. Noe, “Narrowband 2×10 Gbit/s Quaternary Intensity Modulation Based on Duobinary Modulation in *Two Polarizations with Unequal Amplitudes*,” OFC 2006 Paper OThI2.

**5. **J. Zhao, L. Huo, C. K. Chan, L. K. Chen, and C. Lin, “Analytical investigation of optimization, performance bound, and chromatic dispersion tolerances of 4-amplitude-shifted keying format,” OFC 2006 Paper JThB15.

**6. **J. Rebola and A. V. T. Cartaxo, “On the quaternary level spacing signaling optimization for increasing the transmission distance in optical communication systems,” in Proc. of ConfTele2001 3rd Conference on Telecommunications, Figueira da Foz **23**, 514 (2001).

**7. **N. Avlonitis, E. M. Yeatman, M. Jones, and A. Hadjifotiou, “Multilevel amplitude shift keying in dispersion uncompensated optical systems,” IEE Proc. Optoelectron. **153**, 101 (2006). [CrossRef]

**8. **P. J. Winzer, “Optically preamplifier receiver with low quantum limit,” Electron. Lett. **37**, 582 (2001). [CrossRef]

**9. **L. Ribeiro, J. Da Rocha, and J. Pinto, “Performance evaluation of EDFA preamplified receivers taking into account intersymbol interference,” J. Lightwave Technol. **13**, 225 (1995). [CrossRef]

**10. **P. A. Humblet and M. Azizoglu, “On the bit error rate of lightwave systems with optical amplifier,” J. Lightwave Technol. **9**, 1576 (1991). [CrossRef]

**11. **D. Marcuse, “Derivation of analytical expressions for the bit-error probability in lightwave systems with optical amplifiers,” J. Lightwave Technol. **8**, 1816 (1990). [CrossRef]

**12. **J. P. Gordon and L. F. Mollenauer, “Effects of fiber nonlinearities and amplifier spacing on ultra-long distance transmission,” J. Lightwave Technol. **9**, 170 (1991). [CrossRef]

**13. **L. W. Couch, *Digital and Analog Communication Systems* (Prentice Hall, Upper Saddle River, N.J., 2001).

**14. **C. W. Helstrom, “Computing the generalized Marcum Q-function,” IEEE Transactions on Information Theory **38**, 1422 (1992). [CrossRef]

**15. **G. Bosco, A. Carena, V. Curri, R. Gaudino, and P. Poggiolini, “Quantum limit of direct-dection receivers using duobinary transmission,” IEEE Photon. Technol. Lett. **15**, 102 (2003). [CrossRef]

**16. **J. M. Kahn and K. P. Ho, “Spectral efficiency limits and modulation/detection techniques for DWDM systems,” IEEE J. Sel. Top. Quantum Electron. **10**, 259 (2004). [CrossRef]

**17. **G. Bosco and P. Poggiolini, “The Impact of receiver imperfections on the performance of optical direct-detection DPSK,” J. Lightwave Technol. **23**, 842 (2005). [CrossRef]

**18. **K. Sekine, N. Kikuchi, and S. Sasaki, “Modulation parameter tolerance for 8- and 16-APSK (amplitude-and phase-shift keying) signals,” OFC 2006 Paper JThB13.