## Abstract

Exact signal statistics for fiber-optic links containing a single optical pre-amplifier are calculated and applied to sequence estimation for electronic dispersion compensation. The performance is evaluated and compared with results based on the approximate chi-square statistics. We show that detection in existing systems based on exact statistics can be improved relative to using a chi-square distribution for realistic filter shapes. In contrast, for high-spectral efficiency systems the difference between the two approaches diminishes, and performance tends to be less dependent on the exact shape of the filter used.

©2005 Optical Society of America

## 1. Introduction

Inexpensive, flexible electronic techniques for channel impairments mitigation have recently become a focus of research in fiber-optic systems. It is well known that a Maximum Likelihood Sequence Estimator (MLSE) is the optimal equalizer that minimizes the probability of sequence error for channels with intersymbol interference (ISI) [1]. MLSE has been broadly utilized in other areas of communication in the past two decades. The use of MLSE in optical communications was first suggested for coherent systems in the early 1990’s [2–3]. This research was not pursued because of the development of Erbium doped fiber amplifiers (EDFA) that enabled non-coherent communication schemes based on on-off keying (OOK) capable of spanning trans-oceanic distances [4] with all-optical compensation. Interest in electronic equalization was revived in the early 2000’s [5–6]. The achievable performance bounds and error analysis of MLSE for electronic dispersion compensation (EDC) were investigated in [7–9] for short-haul links containing no optical amplifiers, whereas the performance of MLSE for systems dominated by polarization mode dispersion (PMD) was analyzed in [10–11]. Recently, the first prototype of a Viterbi equalizer operating at transmission speeds up to 10.7 Gb/s was demonstrated [12–13].

In this paper, we extend the previous analysis of MLSE to optical links containing a single optical pre-amplifier. The approximate statistics in these links is often treated as non-central chi-squared as reported by Humblet and Azizoglu [14]. However, while chi-square statistics are appealing because of the closed form expression for the statistical distribution and the error probability, it is relatively easy to show that they do not correspond to the true statistics of the detected signal for existing systems. In the first part of the paper, we present our results on the exact statistics of the electrical signal, derived by means of Karhunen-Loeve (KL) expansions, and provide insight into the results. In the second part of the paper, we apply these statistics to sequence estimation (SE) when the received signal is corrupted by the presence of intersymbol interference (ISI) due to chromatic dispersion and/or narrow filtering. Finally, we introduce Narrowly Filtered OOK (NF-OOK) systems that have a potential of significantly increasing the dispersion limited reach and/or spectral efficiency of the existing fiber-optic systems when used in conjunction with EDC.

## 2. Maximum Likelihood (ML) Detection

A natural criterion of performance optimization in a communication system is that of minimizing the probability of error [15] *P*_{e}
. Symbol-error minimization under the known signal statistics is equivalent to the Likelihood Ratio test [15,1]. Thus, for optimal detection, knowledge about signal statistics is of prime importance. The simplest (and the most widely studied) case is the detection of known signals in Gaussian noise. Application of the likelihood ratio test for this case simplifies to the minimization of the Euclidean distance between the detected signal and the noiseless responses in signal space [16–17,7]. When the noise is not Gaussian, the likelihood ratio test still provides the optimal solution but does not correspond to minimization of the Euclidean distance.

A KL-based approach to statistics derivation for optically preamplified systems has been reported previously [18–21]. However, KL expansion in [18] was applied in the frequency domain, whereas in [19–21] the authors apply the saddle-point approximation in order to calculate the bit-error-rate in a link without ISI, short of generating a PDF capable of describing channel statistics applicable for use in sequence estimation. More recently, KL expansion was used in [22–23] to calculate the statistics of the parametric gain in a link with a chain of optical amplifiers, as well as the effect of multipath interference (MPI) [24] in fiber optic links. In this work, we provide a derivation of signal statistics from a statistical optics point of view, irrespective of the modulation format and/or shape of the optical and electrical filters used. Additionally, we extend previously published results by comparing the error performance when detection is performed based on the exact statistics to those derived from approximate non-central chi square distribution. We focus on the fundamental reasons for the difference in the two approaches, or the lack of it. At the time of preparation of this manuscript, a group from Alcatel research has presented results on sequence estimation at OFC 2005 [25]. Although initiated in the KL expansion, the PDF used in SE by the authors was signal dependent Gaussian for the system under consideration.

Consider detection in a single channel fiber-optic communication system with an optical pre-amplifier shown schematically in Fig. 1. The deterministic optical field with additive amplified spontaneous emission (ASE) noise is band-filtered by an optical band-pass filter with impulse response *h(t)*. The filtered optical signal and noise are converted to an electrical signal by a square-law detector (photodiode). The detector is followed by an integrate-and-dump filter that performs an integration of the electrical signal over a signaling interval *T*, resulting in a single sample *r(lT)* produced every *T* seconds.

In this work we treat propagation distances up to 200 km without all-optical dispersion compensation, such that the effects of polarization mode dispersion (PMD) can be neglected with respect to chromatic dispersion. For standard optical amplifiers, the statistics of the ASE are formally identical to a complex additive white Gaussian noise process with independent in-phase and quadrature components in both polarizations [26–27]. The total noise power per polarization is equal to *P*_{n}
=2${\nu}_{0}^{2}$=*n*_{sp}
(*G*-1)*B*_{o}*hf*, where *ν*_{0}
is the noise variance per quadrature component, *n*_{sp}
is spontaneous emission factor, *G* is the amplifier gain, *B*_{0}
is the optical filter bandwidth, *f* is the optical frequency, and *h* is Planck’s constant.

The presence of the band-pass optical filter *h(t)* produces colored optical noise characterized by an autocorrelation function *R̃*_{n}
(*τ*) per quadrature. In order to derive the statistics, it is necessary that the composite elements in the signal decomposition be independent. A standard approach of decorrelation of the noise is the Karhunen-Loéve (KL) expansion [15]. A random process *n(t)* in a time interval *T* can be represented as:

where *ϕ*_{k}*(t)* are orthonormal basis functions, and *n*_{k}
are random coefficients obtained by projecting the *n(t)* onto the basis *ϕ*_{k}*(t)*. In the KL expansion, the basis functions are the eigenfunctions of the solutions to an integral equation with the kernel *R*_{n}*(t-t’)* (see Appendix for details). In this case, the random variables *n*_{k}
, are uncorrelated [15]. Furthermore, since the initial noise process is Gaussian, so are the variables *n*_{k}
. Because the variables *n*_{k}
are uncorrelated and Gaussian, they are also independent.

If the bandwidth of the optical filter is sufficiently large, the correlation between noise samples becomes negligible, and the filtered noise process can be treated as approximately white, while the noise autocorrelation function can be approximated by the Dirac delta-function. In this case, any orthogonal set of functions satisfies the integral equation (A2) [28]. Particularly advantageous, when applicable, is the use of complex exponentials as a basis set, where the eigenvalues of the integral equation (A3) become samples of the noise power spectral density (or equivalently — the filter power spectral density). The results obtained by Azizoglu and Humblet were actually based on the expansion in terms of complex exponentials as the basis functions. In their original paper, the authors provide results for large time-bandwidth products, where the utilization of Fourier harmonics assumption is acceptable (*BT* ~*30*). However, the time-bandwidth product in existing systems is between 3 and 5. Thus, a more rigorous approach is necessary.

The filtered signal waveforms *s(t)*h(t)*, before the square-law detector can also be projected onto the noise eigenfunction basis *ϕk(t)*, resulting in a set of coefficients *S*_{k}
, such that both signal and noise are expressed in terms of the basis functions *ϕk(t)*:

The squaring operation (assumed to be instantaneous in this derivation) transforms the statistics of the signal plus a complex Gaussian noise process of each term *ρ*_{k}
in Eq. (2) to a non-central chi-squared distribution with two degrees of freedom (one per each quadrature component) in each polarization. The joint probability density for two orthogonal polarizations of the *k*-th term in Eq. (2) is equivalent to the probability density of a sum of two independent random variables. The joint PDF is thus a convolution between the PDF’s for each of the polarizations:

where * stands for the convolution operator, ${\sigma}_{k}^{2}$ is the *k*-th eigenvalue from the eigenvalue decomposition of Eq.(A3), *I*_{0}
is the modified Bessel function of order zero, and ${S}_{k}^{\mathit{\Vert}}$
and ${S}_{k}^{\perp}$ are the coefficients corresponding to the projection of the deterministic signal in each polarization in a bit slot onto the *k*-th eigenfunction. The sum of squares of coefficients, ${S}_{k}^{\Vert ,\perp}$ magnitudes equals the total energy of the deterministic waveform in the bit-slot. (${\sum}_{\mathrm{k}=0}^{\infty}$|${S}_{k}^{\Vert}$|^{2}+|${S}_{k}^{\perp}$|^{2}=*E*). Eq. (3) can equivalently be represented in terms of the characteristic function ${\Phi}_{{R}_{k}}\left(\omega \right)$ as:

The two polarization components from Eq. (3) have been left separated in a product to make the expression more readable. Each of the non-central variables *S*_{k}
has a different underlying noise variance ${\sigma}_{k}^{\mathit{2}}$
that is a consequence of the decorrelation process through the KL expansion. Note that the characteristic functions of chi-square variables represent a rare case where the actual signs in the denominator(s), as well as in the complex exponential depend on the adopted signs in the Fourier kernel due to the fact that the PDF is defined for positive values only.

In the final part of the detection scheme, samples from the random process are integrated by means of an integrate-and-dump circuit. Taking advantage of the orthogonality of the basis functions, the integration process is equivalent to a summation of squares of the coefficients:

where subscripts *R* and *I* stand for real and imaginary parts of the corresponding coefficients, and *j*=√-1. This step is equivalent to a summation of the independent (non-central) chi-square variables with four degrees of freedom (2 due to the complex nature of optical fields and 2 for the two polarizations of the field.). The resulting PDF is a convolution of an infinite number of terms of Eq. (3):

$$\underset{k=0}{\overset{\infty}{\mathit{Conv}}}\left\{\left[\frac{1}{2{{\sigma}_{k}}^{2}}\mathrm{exp}(-\frac{r+{{S}^{\parallel}}_{k}}{2{{\sigma}^{2}}_{k}})\xb7{I}_{0}\left(\frac{\sqrt{{{S}^{\parallel}}_{k}r}}{{{\sigma}_{k}}^{2}}\right)\right]*\left[\frac{1}{2{{\sigma}_{k}}^{2}}\mathrm{exp}\left(-\frac{r+{{S}^{\perp}}_{k}}{2{{\sigma}_{k}}^{2}}\right)\xb7{I}_{0}\left(\frac{\sqrt{{{S}^{\perp}}_{k}r}}{{{\sigma}_{k}}^{2}}\right)\right]\right\},$$

where 𝔽^{-1} stands for the Fourier transform inverse. Note that it is correct to represent the net characteristic function as a product if and only if the variables *ρ*_{k}
are independent. This is achieved only if the basis functions are chosen to be the eigenfunctions of the autocorrelation function of the colored Gaussian noise process. If the underlying process were not initially Gaussian, the use of KL expansion is not sufficient to provide independence, resulting in an incorrect PDF.

The electrical noise process originating from the detector and the supporting circuitry and/or an electrical filter can be readily included in Eq. (6). However, our prime interest is comparing the statistics when signal dependent noise is large, as in [14], so we omit those in this manuscript. Note that filtering in the electrical domain will affect the performance even if the electrical noise is neglected in that excessive filtering will introduce additional ISI, and thus deteriorate the performance.

The PDF (6) depends on both the eigenvalues of *R̃*_{n}
(*τ*) and signal coefficients ${S}_{k}^{\Vert}$ and ${S}_{k}^{\perp}$ (also uniquely defined by *R̃*_{n}
(*τ*)). The eigenvalues determine the weight of a particular non-central chi-square distribution term in the overall expression (6). Fig. 2 shows the distribution of eigenvalues for two particular filter shapes: Ideal (rectangular) filter and a Lorentzian shaped filter. These two filter shapes were chosen because closed form solutions for the corresponding eigen-quantities are known and tabulated [29–31].

The usefulness of KL expansion approach lies in the fact that the infinite sum in Eq. (6) is dominated by a small number of terms [28]. An accurate representation of the signal waveform requires that sufficient number of terms be kept in the expansion. This number depends on the compactness of the eigenfunctions. For example, the eigenfunctions of the ideal rectangular filter are prolate spheroid functions (PSS), known to be a complete set of functions with the smallest time-bandwidth product of all known functions [33]. It is owing to this property that PSS require the least number of terms in series expansions for a given accuracy [34]. On the other hand, the eigenfunctions for a Lorentzian-shaped filter are harmonic functions (sines for odd, and cosines for even orders). In order to have an accurate estimation of the signal energy, approximately six times more terms are necessary for a Lorentzian-shaped, than for a rectangular band-pass filter. Consequently, the two types of filter shapes represent two extreme filtering operations. Results for an arbitrary filter shape are generally between the two cases (unless an optical filter with a more gradual roll-off than that of a Lorentzian is used). In practice, it is sufficient to keep the terms down to 10^{-5} of the largest eigen-value to compute the PDF. This last statement is particularly important in a case of a Lorentzian filter, as the time-bandwidth product for such a filter cannot be defined in terms of root-mean-square (RMS) quantities. Note that the information on the compactness of eigen functions is contained in the rate of decrease of eigenvalues for a particular filter power spectral density [33,35], since, in effect, the eigenvalues provide the information on the energy contained in each KL mode. For example, if ordered in the descending order, the first few eigenvalues, for the *rectangular filter*, are approximately unity (see Fig. 2), whereas after the sharp drop-off, the remaining eigenvalues are nearly zero. This sharp descent of eigenvalues was the foundation of the definition of Slepian’s famous *2BT* theorem (note that we define the full filter bandwidth as B, rather than 2B as used in Slepian’s original work) [33]. If only the eigenvalues equal to one are included in Eq (6), the resulting PDF (for a rectangular filter) is the non-central chi-squared with *4BT* degrees of freedom (according to our notation). Therefore, any difference between the exact statistics and chi-square arises from the inclusion of eigenvalues in the “roll-off” region, which, in effect, describe the extent of noise correlations among adjacent bit-slots. The eigenvalues in the roll-off region become important for small *BT* products as substantial difference between the two PDF’s occurs (see Fig. 3). This is a consequence of the fact that for small *BT* products, the number of eigenvalues close to unity is commensurate with the number of eigenvalues in the roll-off region and the impact of the effect of the latter on the overall PDF becomes significant.

As an illustration of the derived exact statistics and the approximate non-central chi-square PDF, Fig. 3 shows histograms obtained simulating two different values of time-bandwidth products for NRZ modulation for ‘all-zeros’ (or a ‘zero-rail’) response where the dissimilarity between the to PDF’s is the largest (see next Section). The chi-square function, with the number of degrees of freedom defined by the *BT* product of the system, under-estimates the histogram around the maximum value, whereas the statistical distribution derived in this section follows histogram much more closely, particularly for the low time-bandwidth product, *BT*=1 shown in Fig. 3(b). As explained above, the discrepancy in the PDF’s originates in neglecting noise correlations that result if the basis functions are not chosen to be the eigenfunctions of *R*_{n}*(t)*. The major assumption in the presented derivation is that the noise statistics before the square-law detector is that of a colored Gaussian process. The most obvious case when this assumption is satisfied is in a link, either with, or without all-optical dispersion compensation that contains a single optical preamplifier, such that nonlinear mixing of signal and noise occurs only due to the square-law detection and not in propagation. Alternatively, the signal statistics will obey those derived in this work, as long as signal-noise and noise-noise nonlinear mixings in propagation can be neglected.

## 3. MLSE in links containing a single optical pre-amplifier

The major impact of intersymbol interference (ISI) in communication systems is to increase the number of responses at the output with respect to the input [9]. Each of the responses at the output will have a different corresponding PDF’s (i.e. likelihood functions). Fig. 4 shows an example of the PDF’s, calculated using the method in the previous section, in both linear and the logarithmic scale, for a total span of ISI of 3 bits for a signal to noise ratio (SNR) of 10 dB. The span of ISI is defined as the total number of bits affected by the energy leakage due to ISI to adjacent bit-slots. The calculated PDF’s are shown for the following responses: *(i)* ‘all-zeros’ in magenta, *(ii)* ‘all-marks’ in dark red, *(iii)* ‘isolated space’ in blue, and *(iv)* ‘isolated mark’ in light orange. These responses are characteristic in that in each one of them a mark (or a space) is surrounded by marks (or spaces) for the whole extent of ISI. In all three parts of the figure, superimposed are the non-central chi-squared distributions (dashed lines) for the same time - bandwidth product *BT*. The relatively long extent of the tails of the distributions is due to their ~*exp(-x)* dependence. For large SNR, the derived PDF’s become more Gaussian-like with characteristic symmetry around the mean and ~*exp(-x*^{2}*)* behavior. Note additionally that PDF’s corresponding to the responses with substantial deterministic field (such as marks, at back-to-back), the difference between the exact statistics and chi-square is significantly smaller than for responses with less optical power (spaces at back-to-back).

In order that MLSE be applicable, the noise samples in different bit-slots need to be independent [1]. Note that the independence of noise samples, in a typical fiber-optic system shown in Fig. 1, cannot be achieved without a bank of filters that would achieve decorrelation physically. Thus, the practical receiver under consideration does not achieve optimal ML performance as there will always be some noise correlations between the adjacent bit-slots not taken into account in detection. However, the marginal densities derived in the previous Section give a superior description of signal statistics than chi-square PDF’s (see Fig. 3). Fig. 5 shows the performance of suboptimal, but practical sequence estimation (SE) in a fiber-optic communication system operating at 10 Gb/s using the exact statistics derived in the previous section at four different distances of propagation for two different filter shapes. All simulations were carried out for NRZ waveforms assuming a raised cosine shape with a roll-off parameter of 0.6 with the extinction of 13 dB. The performance curves are given with respect to the ratio of the average optical energy per bit *E*_{b}
to the total noise power spectral density *N*_{0}
. Note that, we assume that the four noise sources (the in-phase and in-quadrature components in two orthogonal polarizations) are independent, such that *N*_{0}
in this case equals four times the power spectral density per noise component N_{0} ’/2. This quantity is, in our opinion, the proper figure of merit that establishes a fair comparison between the signal and the noise.

As shown in Fig. 5, for an ideal rectangular filter, there is virtually no difference in SE performance for a time-bandwidth product *BT*=5 (typical of the existing systems at 10 Gb/s with a 50 GHz band-pass filter), even though the resulting likelihood functions for the chi-squared distributions and the exact distributions are clearly different (See Fig. 4(a)). To clarify this surprising result, let us first remind the reader that a decision threshold is defined by the intersection of two likelihood functions. Note that the mean values, as well as the locations of maxima of both the chi-squared distributions and the true PDF’s virtually coincide (Fig. 4(a)). If sequence estimation is performed assuming chi-squared distributions for all responses, the abscissae of intersections of the different chi-squares are extremely close to the intersection abscissae of the true PDF’s (see Fig. 4(b) that shows zoomed in image of intersections of the exact statistical distributions in solid lines, and approximate chi-square curves from Fig. 4(a)). Different decisions on the incoming bits in using the two statistical distributions can occur only when a sample falls into the region between the intersections for the exact distributions and chi-square. This event has a relatively low probability of occurrence for such closely spaced likelihood intersections of the two families of PDF’s. The resulting BER assuming non-central chi-square statistic is, thus, asymptotically the same as that of the true statistic which can be observed in Fig. 5(a). The performance difference is further reduced because the distinction between the true statistics and chi-square becomes smaller for large values of *SNR*. Note that according to Eq. (A4) the increase of *SNR* reduces the effect of the eigenvalues in the roll-off region. This case is very much similar to the Gaussian approximation of a Poisson distribution where the overestimate of Gaussian relative to Poisson on the lower end is balanced with the underestimate on the higher end of the distribution. In a Gaussian approximation to a Poisson distribution, these effects cancel providing a fine estimate for total BER, even though the PDF approximation is not nearly as good. To summarize, the performance estimates of fiber optic links (with or without ISI) containing optical preamplifiers, based on the non-central chi-square distribution yield asymptotically correct result assuming rectangular band-pass filters, even though the calculations are based on an incorrect statistics.

The results in the Fig. 5(b), on the other hand, show chi-square statistics to be inadequate for filter shapes significantly different from an ideal rectangular filter with equal filter bandwidth. For this general case, a KL expansion must be used. The performance difference for the exact statistics and chi square for non-rectangular filters is reduced at longer propagation lengths (see Fig. 5(b)). As discussed in the previous section, when the responses with substantial amount of the deterministic field (in which the difference between the true statistics and chi-square is diminished) considerably outnumber channel responses with smaller deterministic energy content, which occurs with the increase of the ISI span at longer propagation distances, the average performance based on chi-square distributions becomes closer to the decision making based on the exact statistics.

As the *BT* product is reduced, the eigenvalues uniformly decrease regardless of the filter used [28] (see Fig. 2 for *BT=1*) and are close for different filter shapes. The statistics become substantially different from chi-square for systems with a *BT* product on the order of 1 (see Fig. 3(b)), and the distribution obtained through KL expansion is required. The considerable difference in PDF’s from chi-square distribution originates in the decrease of eigenvalues, mentioned above. Fig. 6 shows the performance of the system narrowly filtered with a 10 GHz optical filter, in a 10 Gb/s link, at back-to-back and 150 km for a rectangular, and a Lorentzian shaped optical filter.

The system with *BT=1* in Fig. 6 has the total span of ISI equal to 5 bits at back-to-back (2 bits on each side of an isolated mark, plus the mark itself), and, as such, cannot be used in conjunction with standard bit-by-bit decision. SE is capable of retrieving the information for such a system at back-to-back (see Fig. 6(a)). Additionally, SE can extract the information at the distance of 150 km (see Fig. 6(a)) with a power penalty of 1.5 dB, which is in clear contrast with a 4.5 dB penalty in a common system with BT=5 (see Fig. 5). Fig. 6(b) shows the performance of the system with *BT*=1 at back-to-back and 150 km filtered by a Lorentzian type filter with performance close to rectangular filter. Although the probability distributions are clearly different for *BT*=1, the power penalty for using an incorrect, chi-square, likelihood function is around only 0.3 dB. This remarkable behavior is again attributed to the proximity of the likelihood intersections of the chi-squared distributions to those of the exact channel statistics.

Narrowly filtered OOK systems (NF-OOK) systems can only be implemented jointly with electronic dispersion compensation. Note that tight filtering is necessary both at a transmitter, before launching (or multiplexing) to a transport fiber, and at a receiver in order to *(i)* limit the amplifier noise, as well as *(ii)* the crosstalk from the neighboring channels. Possible benefits from these systems are two-fold. They provide a very simple realization of high spectral efficiency, operating using standard on-off keying (OOK) modulation. In addition, the reduced spectral content of these systems makes them substantially more resilient to chromatic dispersion, yielding longer dispersion limited reach, even with respect to systems employing phase modulation such as duobinary [36], or DQPSK [37–38]. These systems allow a limited penalty in performance at back-to-back, whereas they could potentially give large dividends in extending the dispersion limited reach. This is because the span of ISI of such systems, which determines circuit complexity for EDC, grows at a very slow rate due to their forced small spectral content. The main scope of this paper is signal statistics, however, and thorough investigation of performance of NF-OOK systems will be published elsewhere.

## 4. Conclusion

We have analyzed the statistics associated with detection in fiber optic communication systems with a single pre-amplifier. The analysis by means of a Karhunen-Loeve expansion clearly indicates that the signal statistics are heavily dependent on the shape of the optical filter used and can be substantially different from the commonly assumed non-central chi-square distribution for the time-bandwidth products in existing systems. However, while inadequate for describing the signal statistics, we have found chi-square statistics to be surprisingly good for BER assessment when square-law detector is preceded by an ideal rectangular optical band-pass filter. In contrast, strict analysis using KL expansion is necessary for BER assessment for more realistic filter shapes. To the best of our knowledge, we have for the first time introduced Narrowly filtered OOK systems that have a potential of substantially increasing spectral efficiency and/or dispersion limited reach of the existing WDM systems, when used in conjunction with electronic dispersion compensation. The analysis of such systems, as well as the signal processing based on the MLSE favor strict calculation of signal statistics.

## Appendix

In the appendix we include several important relationships of Karhunen-Loeve expansion used for computation of eigen-quantities required for the signal PDF calculation.

The relationship between ASE with noise power spectral density *N*_{0}*’/2* and variance ${\nu}_{\mathit{0}}^{\mathit{2}}$
per quadrature, per polarization, and the noise autocorrelation function (of a wide-sense stationary AWGN process) after filtering with a filter, *H*(*f*), (in the optical domain) is established by the following Fourier transform relationship:

where *R̃*_{n}
(*τ*) is the autocorrelation function of the noise process in one quadrature (the noise processes in the two quadratures are independent), and 𝔽^{-1}{·} represents the inverse Fourier transform.

As required by the derivation, in order to de-correlate variables *n*_{k}
, the expansion needs to be done in terms of eigenfunctions of the filtered noise autocorrelation function *R*_{n}*(t-t’)* [15] that is, according to Eq. (A1), closely related to *R̃*_{n}
(*τ*):

where * stands for the convolution operator and *h** is the conjugate of the optical filter impulse response. Eq. (A2), can be expressed in the normalized form:

where

It is obvious that eigen-quantities need to be calculated only once for a given system configuration. Specifically, while the eigenfunctions are uniquely defined, the eigenvalues need only be scaled by the value of the noise variance for the particular conditions to be able to generate the PDF for a given signal to noise ratio. Note that the eigenvalue equation in Eq. (A2) is expressed in terms of a single quadrature component of the field (either the real or the imaginary part). Assuming that real and imaginary parts of the noise process are independent, Eq. (A2) can be rewritten for each polarization by multiplying it by a factor of 2. In this case, however, the eigenvalues obtained from (A2), and used as noise variances for KL terms in Eq. (2)–(4) need to be divided by 2.

For an arbitrary filter shape, the eigen-quantities need to be calculated numerically. The simplest, though sometimes not sufficiently accurate approach to find the eigen-quantities consists of forming a square autocorrelation matrix of the filtered noise process *Γ*(same as autocorrelation function of the optical filter for AWGN input) of order equal to (or greater than) the number of samples per bit period in the simulation. The middle row of this symmetric Toeplitz matrix should be the central portion of the autocorrelation function - symmetric around zero. The eigen-value decomposition of this matrix provides approximate eigenvalues and sampled versions of the eigenfunction for the given filtered white noise process. However, standard routines in double precision can be imprecise for this particular problem when several eigenvalues are closely spaced as in the case of a flat-top filter. In this case special numerical approaches have to be adopted [32, 35]. Note that if the continuous autocorrelation function *Rn(τ)* is being sampled and represented by an *N*×*N* matrix *Γ*, although the elements of *Γ* can be made arbitrarily close to the autocorrelation function *R*_{n}*(τ)*, this does not necessarily imply that the eigenvalues of the matrix will converge to the eigenvalues of the autocorrelation function for any common matrix norm [39]. However, the imprecision in the PDF calculation will marginally influence the BER, in a way similar to the BER estimate based on the chi-square distribution treated in the main body of the paper.

## Acknowledgments

This work was supported by NSF, UC Discovery Grants Program, DARPA and AFOSR.

## References and Links

**1. **S. Benedetto and E. Biglieri, *Principles of Digital Transmission* (Kluwer Academic/Plenum Publishers, 1999).

**2. **J.H. Winters and R.D. Gitlin, “Electrical signal processing techniques in long-haul fiber-optic systems,” IEEE Trans. Commun. **38**, 1439–1453 (1990). [CrossRef]

**3. **J.H. Winters and S. Kasturia, “Constrained maximum likelihood detection for high-speed fiber optic systems,” in Proc. GLOBECOM ‘91, 1574–1579 (1991).

**4. **N. S. Bergano “Undersea Communication Systems” in *Fiber optic telecommunications IVB*,
Ivan Kaminow and T. Li, Eds. (Elsevier Science2002).

**5. **H.F. Haunstein et al., Design of near optimum electrical equalizers for optical transmission in the presence of PMD”, in Proc. OFC, 2001, Paper WAA4-1.

**6. **H. Bulow and G. Thielecke, “Electronic PMD mitigation-from linear equalization to maximum-likelihood detection” in Proc. OFC 2001, 2001, Paper WAA3-1.

**7. **N. Alić, G. C. Papen, L. B. Milstein, P. H. Siegel, and Y. Fainman, “Performance Bounds of MLSE in Intensity Modulated Fiber Optic Links,” *Fiber optic communication theory and techniques*,
Enrico Forestieri Ed.) 2004 Tyrrhenian International Workshop on Digital Communications, paper 4.5, (2004).

**8. **N. Alić, G. C. Papen, and Y. Fainman, “Theoretical Performance Analysis of Maximum Likelihood Sequence Estimation in Intensity Modulated Short-Haul Fiber Optic Links,” Proc. IEEE LEOS Annual Meeting, Puerto Rico, paper ThB3, (2004).

**9. **N. Alić, G. C. Papen, L. B. Milstein, P. H. Siegel, R. E. Saperstein, F. Parvaresh, N. Santhi, and Y. Fainman, “Performance Analysis of Maximum Likelihood Sequence Estimation in Short-Haul Intensity Modulated Fiber Optic Links,” submitted to Journal of Lightwave Technology.

**10. **A. J. Weiss, “On the Performance of Electrical Equalization in Optical Fiber Transmission Systems”, IEEE Photon. Technol. Lett. **15**, 1225–1227 (2003). [CrossRef]

**11. **H. F. Haunstein, W. Sauer-Greff, A. Dittrich, K. Sticht, and R. Urbansky, “Principles for Electronic Equalization of Polarization-Mode Dispersion” J. Lightwave Technol. **22**1169–82 (2004). [CrossRef]

**12. **A. Faerbert, S. Langenbach, N. Stojanovic, C. Dorschky, T. Kupfer, C. Schulien, J.-P. Elbers, H. Wernz, H. Griesser, and C. Glingener, “Performance of a 10.7 Gb/s Receiver with Digital Equaliser using Maximum Likelihood Sequence Estimation,” Proc. of ECOC’O4, Th.4.1.5, (2004).

**13. **J.-P. Elbers, H. Wernz, H. Griesser, C. Glingener, A. Faerbert, S. Langenbach, N. Stojanovic, C. Dorschky, T. Kupfer, and C. Schulien, “Measurement of the Dispersion Tolerance of Optical Duobinary with an MLSE-Receiver at 10.7 Gb/s,” Proc. of OFC’O5,OthJ4, (2005).

**14. **P.A. Humblet and M. Azizoglu, “On the Bit Error Rate in Lightwave Systems with Optical Amplifiers,” J. Lightwave Technol. **9**, 1576–82 (1991). [CrossRef]

**15. **R.N McDonough and A.D. Whalen, *Detection of Signals in Noise*, Second Edition (San Diego, Academic Press1995).

**16. **J. M. Wozencraft and I. M. Jacobs, *Principles of Communication Engineering* (Waveland Press, reprint1990).

**17. **E. Arthurs and H. Dym, “On the Optimum Detection of Digital Signals in the Presence of White Gaussian Noise — A Geometric Interpretation and a Study of Three Basic Data Transmission Systems”, IRE Trans. on Communication Systems **10**, 336–372 (1962). [CrossRef]

**18. **J. Lee et al., “Bit-error-rate analysis of optically preamplified receivers using an eigenfunction expansion method in optical frequency domain”, J. Lightwave Technol. **12**, 1224–1229 (1994). [CrossRef]

**19. **I. T. Monroy and G. Einarsson, “On Analytical Expressions for the Distribution of the filtered Output of Square Envelope Receivers with Signal and Colored Gaussian Noise Input,” IEEE Transactions on Communications **49**, 19–23 (2001). [CrossRef]

**20. **G. Jacobsen, K. Berlitzon, and Z. Xiapin, “WDM Transmission System Performance: Influence of non-Gaussian Detected ASE Noise and Periodic DEMUX Characteristic,” J. Lightwave Technol. **16**, 1804–1812 (1998). [CrossRef]

**21. **I. T. Monroy and G. Einarsson, “Bit Error Evaluation of Optically Preamplified Direct Detection Receivers with Fabry-Perot Optical Filters,” J. Lightwave Technol. **15**, 1546–1553 (1997). [CrossRef]

**22. **G. Bosco, A. Carena, V. Curri, R. Gaudino, P. Poggiolini, and S. Benedetto, “A Novel Analytical Method for the BER Evaluation in Optical Systems Affected by Parametric Gain,” IEEE Photon. Technol. Lett. **12**, 152–4, (2000). [CrossRef]

**23. **G. Bosco, A. Carena, V. Curri, R. Gaudino, P. Poggiolini, and S. Benedetto, “A Novel Analytical Approach to the Evaluation of the Impact of Fiber Parametric Gain on the Bit Error Rate” IEEE Trans. Commun. **49**, 2154–63 (2001). [CrossRef]

**24. **I. B. Djordjevic and B. Vasic, “Receiver Modeling for Optically Amplified Communication Systems,” International J. Electron. Commun. **57**, 381–390 (2003). [CrossRef]

**25. **F. Buchali and H. Bulow, “Correlation sensitive Viterbi equalization of 10 Gb/s signals in bandwidth limited receivers,” in Proc. OFC 2005, Paper F020.

**26. **R. Loudon and T.J. Shepherd, “Properties of the Optical Quantum Amplifier,” Optica Acta , **31**, 1243–1269 (1984). [CrossRef]

**27. **C. Dorrer, C.R. Doerr, I. Kang, R. Ryf, J. Leuthold, and P.J. Winzer, “Measurement of eye diagrams and constellation diagrams of optical sources using linear optics and waveguide technology,” J. Lightwave Technol. **23**, 178–186 (2005). [CrossRef]

**28. **B. Saleh, *Photoelectron Statistics, With Applications to Spectroscopy and Optical Communication*, (Springer-Verlag, 1978).

**29. **C. Flammer, *Spheroidal Wave Functions* (Stanford Univ. Press, 1957).

**30. **D. Slepian and E. Sonnenblick, “Eigenvalues Associated with Prolate Spheroidal Wave Functions of Zero Order,” Bell Sys. Tech. J. **45**, 1745–59 (1965).

**31. **D. Slepian, “Fluctuations of Random Noise Power,” Bell Sys. Tech. J. **37**, 163 (1958)

**32. **D. Slepian, “A Numerical Method Of Determining EigenValues And EigenFunction Of Analytic Kernels” SIAM Journal of Numerical Analysis , **5**, 586–600 (1968). [CrossRef]

**33. **D. Slepian, “On Bandwidth,” Proc. Of IEEE , **64**, 292–300 (1976). [CrossRef]

**34. **I. C. Moore and M. Cada, “Prolate spheroidal wave functions, an introduction to the Slepian series and its properties” Appl. Comput. Harmon. Anal. **16**, 208–230 (2004). [CrossRef]

**35. **D.B. Percival and A.T. Walden*Spectral Analysis for Physical Applications: Multitaper and Conventional Univariate Techniques* (Cambridge, Cambridge University Press, 1993). [CrossRef]

**36. **K. Yonenaga and S. Kuwano, “Dispersion-Tolerant Optical Transmission System Using Duobinary Transmitter and Binary Receiver,” J. Lightwave Technol. **15**, 1530–1537 (1997). [CrossRef]

**37. **R. A. Griffin and A. C. Carter, “Optical differential quadrature phaseshift keying (oDQPSK) for high capacity optical transmission,” in Proc. OFC, 2002, Paper WX6C.

**38. **A.H. Gnauck and P.J. Winzer, “Optical phase-shift-keyed transmission,” J. Lightwave Technol. **23**, 115–130 (2005). [CrossRef]

**39. **P. Stoica and R. Moses, *Introduction to Spectral Analysis* (Prentice Hall, 1997).