## Abstract

We present a rigorous analysis defining the fundamental performance limits of duobinary line-coding for optical communications. First, we compare the back-to-back performance of duobinary and intensity modulation systems in an AWGN channel satisfying the Nyquist criterion, with both linear and quadratic receiver. We show that, also for duobinary and quadratic receiver, matched filtering is the best achievable condition. Then, we derive a detailed performance analysis of duobinary in an ASE-noise-limited direct-detection optical system considering noise on the entire space of polarizations. We show that for duobinary line-coding the expression of the bit error rate depends both on the shape of the transmitted pulse and on the receiver optical filter. Comparing duobinary coded and uncoded intensity modulation systems, we show the intrinsic advantages of using the duobinary line-coding in a system based on quadratic detection. Finally, some results for realistic setups are obtained through simulation and compared to the fundamental limits in order to show how close to those limits state-of-the-art systems can operate.

© 2008 Optical Society of America

## 1. Introduction

Duobinary line-coding (DB) is a so-called partial-response transmission format [1]. This class of formats makes use of bit correlation to reduce the spectral width and then potentially increase spectral efficiency. Correlation is introduced through some amount of controlled pulse Inter-Symbol Interference (ISI). The transmission based on DB was first proposed in the 60’s by A. Lender [2], for radio frequency (RF) communications [2]. Its high spectral efficiency was the aspect that made it attractive in that context. Later, it was overcome by multilevel modulation schemes that could reach even higher spectral efficiencies.

DB has re-emerged in the field of optical communications some years ago. An early example of optical duobinary, proposed by Gu et al. [3] in 1993, was based on three-level amplitude modulation, but sensitivity problems hampered this solution. In 1995, Yonenaga et al. [4] proposed to combine DB with both amplitude (AM) and phase (PM) modulation. The AM/PM optical DB can be easily implemented using intrinsic properties of Mach-Zehnder (MZ) modulators. Moreover, it can be received using a standard Direct-Detection (DD) receiver (RX) as the photodetector acts as decoder, and presents strong tolerance to chromatic dispersion thanks to the high spectral efficiency. However, the proposed transmitter (TX) implementation [4] requires special delay-and-add electric filters and a dual-arm MZ modulator.

In 1996, Penninckx et al. [5] proposed a different duobinary implementation, called Phase-Shaped Binary Transmission (PSBT), which makes use of a conventional low-pass Bessel filter and a single-arm modulator. In 2001, PSBT has been successfully employed to transmit 80 channels at 40 Gbit/s, over 300 km, with a channel spacing of only 50 GHz [6].

Several comprehensive review papers on the advantages and disadvantages of the use of optical DB have been published in the technical literature. A review of such results is well summarized in [7].

The presented results are mainly focused on the high spectral efficiency of duobinary and its high resilience to fiber chromatic dispersion [8,9]. Further results are about the robustness of duobinary to non-linear effects [10] and about the possibility of sensibly improving the sensitivity of duobinary receivers by a careful optimization of both electric and optical filter bandwidths [11–14].

The purpose of this paper is to extend the results shown in [15], in order to present a rigorous analysis of the back-to-back performance of DB for systems limited by the Amplified Spontaneous Emission (ASE) noise introduced by the optical amplification. Besides the analysis of DB performance, we also study the standard “binary coded” intensity modulation as a reference for a comparison, because this modulation format is still today the mostly used in optical communications systems.

In Section 2, some results about binary and duobinary coding in AWGN channels with linear receiver are recalled. Then, in Section 3, we analyze in detail the fundamental limits of DB in AWGN channels with quadratic detection through a simplified approach. Section 4 is devoted to introduce the detailed analysis of performance evaluation for optical DB in ASE limited channels considering noise on the entire space of polarizations, showing the full agreement with results described in Section 3.

In Section 5, we analyze by simulation some realistic scenarios based on transmitters and receivers using the state-of-the-art technology, discussing how close to the fundamental limits DB systems typically perform. Differently from [11–14], where a Gaussian approximation was used, we evaluate the system performance by resorting to an accurate semi-analytical simulation technique [16] based on the Karhunen-Loève decomposition of signal and noise [17–19]. Finally, in Section 6 we discuss the obtained result and we draw some conclusions.

## 2. Fundamental limits of binary coding and duobinary line-coding in additive white Gaussian noise (AWGN) channels with a linear receiver

In this section, the analysis will be restricted to the case of a channel satisfying the Nyquist criterion, which means that there is no inter-symbol interference (ISI) at the sampling time [1]. The noise at the receiver input is assumed to be white and Gaussian, with double sided spectral density *N*
_{0}/2. This assumption is verified in optical systems whenever the ASE noise is dominant, which is the case in WDM systems. The performance of binary and duobinary modulation formats will first be evaluated in the case of an AWGN channel as a reference: in optical communication systems it corresponds to the case of a coherent receiver. Quadratic detection, which is more usual in optical systems today, will be investigated later in Section 3.

#### 2.1 Binary coding on AWGN channel

The baseband equivalent channel, shown in Fig. 1, consists in an emission filter *HE(f)*, a noise source and a receiving filter *HR (f)* followed by a decision circuit.

Let us then assume a global response satisfying the Nyquist criterion, *nY(t)*, with a Fourier transform *NY(f)*. The data {*a _{k}*}at the channel input are uncorrelated, taking on the values ±1, and we assume the signal at the system input to be written as:

where *A* is a constant and *δ* the Dirac impulse. Let us note that, even if the signal given by Eq. (1) does not exist physically, the emitted signal can always be regarded without any restriction as the output of a filter *HE (f)* with signal (1) at its input.

It is well known that the error probability is minimum when filtering is equally shared in module between the emitter and receiver side, i.e the emission and reception filters satisfy:

Phase terms, which must be such that *H _{E}(f)·H_{R}(f)=N_{Y}(f)*, are not considered because they do not impact the power budget. When and only when

*N*is real positive, the couple (

_{Y}(f)*H*) is a couple of matched filters, the noise variance at the sampling time is

_{E}, H_{R}*N*

_{0}/2 and the error probability is then given by the classical relationship [1]:

$$=\frac{1}{2}\mathrm{erfc}\sqrt{\frac{{P}_{m}T}{{N}_{0}}}=\frac{1}{2}\mathrm{erfc}\sqrt{\frac{E}{{N}_{0}}}$$

where *P _{m}* is the mean power at the output of the emission filter,

*E*is the energy of the pulse sent into the line (equal to

*A*·

*h*,

_{e}(t)*h*being the inverse Fourier transform of

_{e}(t)*H*). erfc is the complementary error function.

_{E}(f)If the Nyquist transfer function is not real positive, there is a penalty and the error probability becomes:

*λ* equals one when *N _{Y}(f)* is real positive, and it exceeds one in any other case. The power penalty in dB is given by 10log

*λ*

^{2}.

To illustrate this point, let us consider one specific case for the global response transfer function:

It can be easily verified that it satisfies Nyquist’s criterion, nevertheless *λ* equals 4/*π*, which corresponds to a power penalty of 10log(16/*π*
^{2})=2.1 dB.

#### 2.2 Duobinary coding on AWGN channel

The basic idea of partial response coding consists in introducing some amount of controlled ISI, which can be removed at the receiving end: this process allows reducing the spectral width of the signal and some increase of spectral efficiency can be expected. The counterpart is some power penalty.

A partial response signal can be obtained either by modulating a Nyquist waveform by correlated binary data, or by modulating a linear combination of delayed Nyquist waveforms by binary uncorrelated data. The fact that the pulse has a finite number of non zero samples, instead of only one in the case of Nyquist criterion, explains the name of partial response. The introduction of partial response coding can be viewed as a filtering of the signal, i.e as the multiplication of the global transfer function *NY(f)* by a 1/*T* periodic transfer function *C(f)*. In the case of duobinary modulation scheme, which we are more particularly interested in here, *C(f)* is given by:

It corresponds in the time domain to the addition of a signal and a replica delayed by *T*. Exactly as in the previous problem, the optimum performance is obtained when filtering is equally shared in module between the transmitter and the receiver, which means that:

In the case of duobinary modulation, we have three possible signal values at the sampling time, 0 and ±2*A*. Let us note that any function proportional to *C(f)* can be considered, it does not change performance, because signal and noise are equally impacted. As an example, considering *C(f)*/2, signal at the sampling can take on the values 0 and ±*A*.

Data precoding, which is required to correctly extract the binary information from the three levels signal, is not explicitly mentioned here: let us just recall that the levels 0 and ±2*A* are associated to the two values of the binary data. Precoding is described in section 4.

Neglecting that they do not have the same *a priori* probability and then, the optimum threshold position is not exactly in the middle, assuming a high enough signal to noise ratio which allows neglecting the fact that the correct decision region for ±2*A* rigorously consists in disjoint stripes, the error probability can be written as:

where the noise variance is written as:

The average emitted power is *µA ^{2}/T*, and then the error probability is given by

When the Nyquist channel transfer function is real positive, *µ* can be easily obtained.

In this case, the power penalty in comparison with antipodal signaling (neglecting factor 3/4) due to duobinary coding is:

This result is valid only when the duobinary coding is applied to a Nyquist channel with a real positive transfer function. If it is not, the degradation is always higher.

## 3. Fundamental limits of duobinary line-coding in AWGN channels with a quadratic receiver

Let us consider now the case of quadratic detection, the most usual in optical communication systems today, with the same assumption of a real positive Nyquist transfer function and data taking on now the values 0 or 1. For ordinary binary coding, we have at the sampling time a signal value which is either 0 or *A*
^{2}, according to the transmitted data, while the noise components have a variance *N*
_{0}/2.

Let us assume now duobinary coding, with equally shared filtering. The signal at the sampling time will be either 0 or 4*A*
^{2}. The variance of each noise component is *µ N*
_{0}/2, where *µ* is given by (11).

In the case of binary transmission, the mean power is *A*
^{2}/2*T*, while it is *µ A ^{2}/T* in the case of duobinary transmission.

Table 1 compares the signal value and noise variance at the sampling time in both situations. It is clear that achieving the same error probability requires the following relationship between the average powers for both modulation schemes:

The gain is then:

A more detailed expression of the error probability has been derived in [15] taking into account the noise components orthogonally polarized respectively to the signal, because both polarizations must rigorously be taken in account in optical receivers. Taking that into account would not change the comparison between binary and duobinary.

This result can be easily explained as follows. For antipodal signaling and linear receiver, duobinary line-coding has a penalty of 2.1 dB, (Eq.(12)), which is due to the difference in terms of noise power, the distance between possible signal levels at the sampling time being 2*A* in both cases. This noise penalty remains valid with quadratic detection.

The possible signal values at the sampling time are respectively 0 and *A*
^{2} for OOK, 0 and 4*A*
^{2} for duobinary, but the emitted average power is twice higher for duobinary. Achieving the same signal levels at the sampling time requires the OOK emitted power to be increased by 3 dB. The resulting penalty Δ_{Q} between OOK and duobinary with quadratic detection is then:

This gain of 0.91 dB compared to OOK is the best which can be achieved. In particular, it is obtained in the case of the minimum bandwidth Nyquist filtering (ideal rectangular filter of bandwidth 1/*T*) which is real positive. Let us now examine two examples where it is not.

#### 3.1 First example.

Duobinary filtering must be equally split to obtain the best performances. Let us assume it is entirely located at the emitting end, applied to rectangular waveforms. Defining rect(*t*) as the classical NRZ waveform equal to 1 over a time interval *T* and zero elsewhere, the average emitted power is in the two situations:

The receiving filter is matched to the NRZ pulse and is then identical for OOK and duobinary. The signal at the sampling time (respectively *A*
^{2} and 4 *A*
^{2}) can be expressed as 2*P _{m}* in both cases. In these conditions, duobinary and OOK perform equally.

#### 3.2 Second example

Let us now *a priori* choose a channel impulse response, for example *x(t)*, which is a triangle shifted by *T*/2, verifying effectively *x*(0)=*x(T)*=1, *x(kT)*=0 for any other integer *k* [15]. Its Fourier transform is:

*X(f)* cannot be written as *M(f)·C(f)*, i.e. obtained by applying the duobinary filtering to any signal *M(f)*. In the time domain it means that *x(t)* cannot be written as *m(t)*+*m(t-T)*.

Then it is not possible to identify a transmission channel consisting in three bricks, emitting filter, duobinary filtering (eventually split between emitter and receiver), and receiving filter. In fact, the channel has a global impulse response which verifies the Nyquist criterion at (2*k*+1)*T*/2, *k* integer, but sampling is done at *kT*, *k* integer. In the following, this scheme will be called TSNC (*Time Shifted Nyquist Channel*).

In this case, the mean power is exactly the same as with antipodal signals, i.e. twice that of OOK signals. Possible signal values are 0 and ±*A*, before detection and 0 and *A*
^{2} after, i.e. the same with OOK as well as with TSNC modulation scheme. The noise variance is identical for OOK and TSNC. In conclusion, the fact that the emitted power must be doubled for the same performance explains the 3 dB degradation found in [15].

#### 3.3 Third example

NRZ signaling with duobinary partial response corresponds to a global impulse response *z(t)* which is the sum of the triangle response of previous example plus its replica delayed by *T*. *z*(*T*/2) and *z*(3*T*/2) are equal to +1, and then the sampled signal can take on three values 0 and ±2*A*. We are here in the case of a real positive Nyquist channel combined with DB coding. If DB coding is equally split between the emitting and receiving side, the (negative) penalty against OOK is -0.91 dB.

#### 3.4 Comments

This approach of section 3.3 can be applied to all the Nyquist responses which extend over an interval of duration 2*T*, obtained by convolving a time limited impulse *u(t)* of duration *T* with itself. The autocorrelation function *x(t)* verifies the Nyquist criterion because it is time limited to a 2*T* interval, it is symmetrical, and then it can be viewed as a duobinary impulse when shifting the sampling time from the maximum by *T*/2. And effectively the ratio o *x(T*/2)/*x*(0) (proportional to the signal to noise ratio at the sampling time) will determine performance, as mentioned in [15]. This dependence will be considered in details in section 4 of this paper.

The advantage of duobinary and more generally of any partial response signaling is through the introduction of filtering within the emitted signal, to create controlled ISI which can be removed at the receiver, at the expense of some penalty. This allows restricting the signal bandwidth and then reduces the impact of the impairments due to the selectivity of the transmission medium. This argument must be cautiously used, because in fact the Nyquist bandwidth is needed: for a smaller one, unwanted ISI is introduced besides the aforementioned controlled ISI, and it cannot be cancelled in the decoding process. Nevertheless, it is true that duobinary exhibits some advantages, for example as far as robustness against chromatic dispersion is concerned.

## 4. Detailed analysis of optical duobinary line-coding performances in two polarizations ASE noise channels and quadratic receiver

In this section we extend and report in detail some results presented in [15]. Moreover, following the new findings presented in Section 3, we are able to state that matched filtering is the best condition also for duobinary and a quadratic receiver. This allows to draw some more general conclusions from results previously presented in [15], in particular to show that the best result among the impulse shapes considered in reference [15] is effectively the best which can be achieved.

The standard structure of a duobinary system that can be found in early papers [3], [4] and in textbooks [1] is pictorially described in Fig 2. First, a bit sequence *p _{n}* is obtained through the suitable precoding of the information bit sequence

*a*:

_{n}where “⊕” means EXCLUSIVE OR and “*a*̄_{n}” means NOT(*a _{n}*). Both

*a*∊{0,1} and

_{n}*p*∊{0,1}, i.e., are unipolar binary sequences. Then, the processing

_{n}is applied and the binary unipolar sequence *p _{n}* results in a binary bipolar sequence

*b*∊{-1,1}. Finally,

_{n}*b*is used to form the transmitted signal through filtering in the emission filter with impulse response

_{n}*u(t)*and up-conversion to the carrier frequency

*f*

_{0}:

where *P*̄_{S} is the average power of the transmitted signal *s _{TX}(t)*;

*T*is the inverse of the bit-rate

*R*;

_{B}*ω*

_{0}=2

*πf*

_{0}, where

*f*

_{0}is the optical carrier frequency; and

*ν*̂

_{‖}is the complex unit vector defining the polarization of the modulated signal, i.e. the signal carrying information. If

*u(t)*is strictly limited to a time interval of duration

*T*, i.e. the partial response coding is applied entirely at the receiver, signal of Eq. (20) is effectively a DPSK (Differential Phase Shift Keying) signal [1]. Indeed, in [20] it was shown that DPSK can be seen as a duobinary transmission for which the precoding occurs at the transmitter side and the duobinary coding/decoding at the receiver side.

The effect of the channel is assumed to be limited to ASE noise insertion, the signal at the receiver input becomes then:

where *n*
_{‖}(*t*) is the ASE noise on the polarization of the modulated signal and *n*
_{⊥}(*t*) is the ASE noise on the polarization orthogonal to the modulated signal. Note that:

where *n*
_{‖i}(*t*), *n*
_{‖q}(*t*), *n*
_{‖i}(*t*), and *n*
_{‖q}(*t*) are statistically independent real, white and Gaussian random processes whose power spectral density is *N*
_{0}/2. In practical systems, *N*
_{0}/2 is set by the overall amount of noise introduced by the in-line optical amplifiers.

Making use of the pre-coding defined in Eq. (17), needed to recover the transmitted data stream on a bit-by-bit basis, the duobinary modulated filtered received pulse *x(t)*, i.e., the pulse used for the decision, is required to have *two* identical non-zero samples *T* seconds apart [1]:

In practice, duobinary pulses are *defined* as pulses that spread over two bit intervals, i.e., pulses that correlate adjacent bits through the introduction of a controlled amount of ISI (cf. Sections 2 and 3). Equation (23) specifies a constraint on solely the received filtered pulse *x(t)*,i.e., on the signal resulting from the convolution of the TX pulse with the RX filter impulse response. It means that the required correlation can be introduced by means of both *u(t)*, using a pulse having a temporal support wider than the bit duration, and *h _{RX}(t)*, choosing a suitable optical filter that adds a controlled amount of correlation, always provided that the requirement given in Eq. (23) is verified. Hence, the trade off we have to balance is between more spectral efficiency, wider

*u(t)*, and less noise impairment, wider

*h*. The optimum penalty expressed by Eq. (12) in the case of a linear channel is exactly the result of this trade off.

_{RX}(t)If the received pulse satisfies the duobinary pulse constraint express in Eq. (23), the complex envelope of the filtered optical signal at the optimum sampling instant *t _{opt}* can be written as:

where *c _{n}* can assume three levels, depending on the values of

*b*and

_{n}*b*:

_{n-1}As an example, the noisy eye diagram of the duobinary signal before photo-detection is shown in Fig. 3. It can be easily shown, by tracing all the steps from the transmitted information sequence *a _{n}* to the received sequence

*c*, that both

_{n}*c*=+2 and

_{n}*c*=-2 correspond to

_{n}*a*=1, whereas

_{n}*c*=0 corresponds to

_{n}*a*=0. Therefore, using a simple MOD2 decoder, which performs a modulo-2 operation on the sequence

_{n}*c*, full recovery of the original information bit sequence is therefore possible employing bit-by-bit decision, as we expected to obtain imposing the ISI condition of Eq. (23).

_{n}*n*
_{‖F}(*t*) is the filtered ASE noise on the polarization parallel to the modulated signal and *n*
_{⊥F}(*t*) is the filtered ASE noise on the polarization orthogonal to the modulated signal. Note that:

where *n*
_{‖Fi}, *n _{‖Fq}, n_{⊥Fi}* and

*n*

_{⊥Fq}are statistically independent Gaussian random variables with zero mean and variance equal to:

where *B _{eq,RX}* is the equivalent noise bandwidth of the receiver filter.

The optical signal is photo-detected (direct-detection) yielding the decision signal, whose expression at the optimum sampling time is:

Note that direct detection acts as the MOD 2 decoder required in order to recover the transmitted bit sequence *a _{n}* from

*c*because

_{n}*a*is indeed equal to ¼

_{n}*C*.

^{2}_{n}Considering insertion of ASE noise, the obtained decision variable is a random variable statistically distributed as a 4-degree of freedom Chi-square distribution [1]. The variance is the one reported in Eq. (27) and the centrality parameter is:

Therefore, accordingly with the theory reported in [1], the error probability can be analytically written as:

where *Q*
_{2} is a Marcum Q-function of order 2 [1] and *ϕ* is the normalized decision threshold that must be optimized for each value of the OSNR and each pulse shape. It is important to remark that Eq. (30) is the general analytical expression of the error probability for the optical duobinary modulation using a standard DD receiver. This expression shows the importance of the choice of the pulse shape in systems based on duobinary modulation. In general, the error probability depends both on the transmitted pulse *u(t)* and on the optical filter impulse response *h _{RX}(t)*.

From Section 3, we know the best performance can be achieved using an optical filter matched to the received pulse. Recalling that in this case *B _{eq,RX}*=

*x(T*/2) [15], we can express

*P*for duobinary modulation using bit-by-bit direct detection and matched optical filter as follows:

_{E}where the Optical Signal-to-Noise Ratio (*OSNR*) is defined as:

Since one of the most attractive aspects of duobinary is its high spectral efficiency, we selected among all the possible pulse shapes satisfying the Nyquist criterion the one with the smallest possible bandwidth occupation. Such a pulse shape is defined in time-domain by the following expression [3], [1]:

corresponding to the spectrum *X(f)*=*F{x(t)*}:

Eq. (33) shows that this channel corresponds to the application of duobinary coding to a minimum bandwidth Nyquist pulse (the so called brickwall spectrum, cf. section 3). The choice of this received pulse *x(t)*, together with the hypothesis of receiver matched filter, yields the following transmitted pulse shape *u(t)*:

where “*F*
^{-1}” means inverse Fourier transform. The time evolution of the received pulse shape *x(t)* with minimum bandwidth occupancy can be observed in Fig. 4.

Using this pulse we get *x*(0)=*π*/4, that, substituting into Eq. (31), means:

Supposing to operate at *P _{E}*=10

^{-9}, from Eq. (36) we derive that the requested OSNR is 16.2, or 12.09 dB. In order to operate at the same

*P*an IM system requires OSNR=13 dB. Therefore, we have verified that the use of duobinary modulation based on the received pulse

_{E}*x(t)*with minimum spectral occupancy (which satisfies the Nyquist criterion and has a real positive spectrum), together with a direct detection receiver employing a filter matched to the transmitted pulse, bears an advantage of 0.91 dB in terms of OSNR with respect to IM modulation, confirming the results of Section 3. Such results have been confirmed through simulation as well, both using the semi-analytical BER estimation based on the Karhunen-Loève decomposition of signal and noise and a brute force error counting.

## 5. Optimization of practical implementation of optical duobinary

In practical implementations of transmitters for optical duobinary, the cascade of bit processing and pulse shaping is not performed as two definite separate steps. In fact, after the pre-coding, the two level electrical signal is tight filtered and the resulting signal is used to drive an amplitude Mach-Zehnder modulator.

Narrow electric filtering is needed to introduce bit correlation and can be obtained using either a delay-and-add filter [4], [7] or a best-fit Bessel-filter approximation of it [5]. The last technique is called Phase-Shaped Binary Transmission (PSBT), and it is the technique that has gained the largest consensus in the community because of the simple setup and good performance.

Both the proposed modulation techniques use the properties of Mach-Zehnder amplitude modulators driven in the [-*V _{π}*;+

*V*

_{π}] range in order to obtain mixed amplitude-phase three level modulation [5]. For both the described techniques, pulse shaping can be further refined introducing a tight optical filter right after the optical modulator.

#### 5.1 System optimization

In order to perform a quantitative analysis, we simulated a typical PSBT setup using the commercial simulator OptSim^{™}.

The structure of the system we have considered is shown in Fig. 5. The bit-rate has been set to 10 Gbit/s. Specifically, we assumed to employ a pre-coder yielding the sequence *p _{n}*, followed by a 5-pole Bessel electrical filter with bandwidth

*B*. This signal drives, between symmetric transmission maxima, a Mach-Zehnder modulator biased at full extinction. The modulator can be eventually followed by an optical transmitter filter with bandwidth

_{TX,elt}*B*.

_{TX,opt}A similar filter with bandwidth *B _{RX,opt}* is placed at the receiver before the photodiode. For both the optical filters we chose second-order Supergaussian filters because many commercial AWGs (Array Waveguide Gratings) and de/interleavers have this shape. The receiver includes also a 5-pole Bessel post-detection filter whose bandwidth is

*B*. The photodetector is assumed to be ideal.

_{RX,elt}We suppose the signal propagation is affected only by ASE noise and performed several simulation runs, varying all transmitter and receiver filter bandwidths with step 0.1 GHz, in order to find the optimum combination of bandwidths values giving the best performance in terms of bit error rate.

Three different system configurations have been considered:

• System 1: No optical filter is used at the transmitter side

• System 2: An optical filter equal to the one at the receiver is placed at the transmitter side (this case mimic the use of the same type of AWG at transmitter and receiver side, a typical solution)

• System 3: A specifically optimized optical transmission filter is used

The optimization was performed at two different values of OSNR (8 dB and 12.5 dB), corresponding to BER values around 10^{-3} and 10^{-9} respectively. Simulations have been carried out using an accurate semi-analytical simulation technique based on the Karhunen-Loève decomposition of signal and noise. The results are shown in Table 2.

All three system configurations, at both OSNR values, turned out to be quite insensitive to the value of the post-detection filter bandwidth: in fact, there was a sort of “saturation” of the performance for a bandwidth values greater than 20 GHz. Being the optical filtering very tight, performances does not depend too much on post-detection filter, if it is wide enough with respect to signal bandwidth.

In System 1, the absence of the optical filter at the transmitter requires a tighter electrical filtering, together with a narrower receiver optical filter. Differences in System 2 and 3, where we only remove the constraint of same optical filtering both at transmitter and receiver, are not significant: only small adjustments are observed.

Performances in terms of bit error rate versus OSNR using the optimum bandwidth values obtained previously are reported in Fig. 6. Since the values of the optimum bandwidths obtained using OSNR of 8 dB and 12.5 dB are very close and yield to almost coincident performance, we chose to use only the set of bandwidth obtained at 12.5 dB to generate the curves.

The performance of the three system configurations are almost coincident, showing that the use of an optical filter at the transmitter is not required in a duobinary single-channel system for optimizing the shape of the transmitted pulse. Moreover, the penalty with respect to the fundamental limits derived in Sections 3 and 4 is very small: from 0.2 dB for System 3 to 0.4 dB for System 1.

Since the value of the optimum RX electrical filter bandwidth (35 GHz) is not quite realistic, we also show in Fig.6 the performance of System 1 when a standard 10 Gbit/s PIN receiver, with a bandwidth equal to 7.5 GHz, is used. The bandwidth of both the TX electrical filter and the RX optical filter are optimized, yielding the following values: *B _{TX,elt}*=2.9 GHz and

*B*=7.0 GHz. In this more realistic scenario, the penalty with respect to the fundamental limit is around 1 dB at BER=10

_{RX,opt}^{-9}.

Note that the use of standard filter bandwidths at the RX for both electrical and optical filters (e.g. *B _{RX,elt}*=7.5 GHz and

*B*=35 GHz, typical of WDM systems with channel spacing equal to 50 GHz) would yield a much worse performance, i.e. more than 5 dB OSNR penalty at 10

_{RX,opt}^{-4}with respect to the theoretical limit.

## 6. Conclusions

In this paper we have proved that the best solution is matched filtering also for duobinary with a quadratic receiver. In a previous paper [15] the error probability was analytically derived, the sensitivity computed for different pulse shapes, and a gain of 0.91 dB compared to conventional binary OOK could be obtained, but a theoretical demonstration had not been given yet. In this paper, we theoretically prove that the best performance is obtained in a matched filter configuration, and that duobinary can achieve a maximum sensitivity gain of 0.91 dB with respect to intensity modulation. The best sensitivity obtained in [15] is then effectively the highest which can be achieved.

Considering realistic implementation of transmitter and receiver for duobinary in optical communication systems, we have evaluated by simulations that achievable performances are not far away from fundamental limits: less than 0.5 dB loss can be obtained with state-of-the-art components.

## Acknowledgments

This work was supported by the Projects BONE (“Building the Future Optical Network in Europe”) and EURO-FOS, two Networks of Excellence funded by the European Commission through the 7th ICT-Framework Programme. The authors would like to thank RSoft Design Group, Inc for supplying the simulation tool OptSim®.

## References and links

**1. **J. G. Proakis, *Digital communications*, 2nd edition (New York, McGraw-Hill), 1989.

**2. **A. Lender, “The duobinary technique for high-speed data transmission,” IEEE Trans. Commun. and Elecron. , **82**, 214–218 (1963).

**3. **X. Gu and L.C. Blank, “10 Gbit/s unrepeated three-level optical transmission over 100 km of standard fibre,” Electron. Lett. , **29**, 2209–2211 (1993). [CrossRef]

**4. **K. Yonenaga, S. Kuwano, and S. Norimatsu, “Optical duobinary transmission system with no receiver sensitivity degradation,” Electron. Lett. , **31**, 302–304 (1995). [CrossRef]

**5. **D. Penninckx, M. Chbat, L. Pierre, and J.-P. Thierry, “The Phase-Shaped Binary Transmission (PSBT): a new technique to transmit far beyond the chromatic dispersion limit,” in *Proceedings of ECOC 1996* (Oslo, Norway, Sep. 1996), 2, pp. 173–176.

**6. **H. Bissesur, G. Charlet, C. Simonneau, S. Borne, L. Pierre, C. De Barros, P. Train, W. Idler, and R. Dischler, “3.2 Tb/s 80×40 Gb/s C-band transmission over 3×100 km with 0.8 bit/s/Hz efficiency,” in *Proceedings of ECOC 2001* (Amsterdam, The Netherlands, Sep. 2001), 6, pp. 22–29.

**7. **T. Ono, Y. Yano, K. Fukuchi, T. Ito, H. Yamazaki, M. Yamaguchi, and K. Emura, “Characteristics of optical duobinary signals in terabit/s capacity, high-spectral efficiency WDM systems,” J. Lightwave Technol. , **16**, 788–797 (1998). [CrossRef]

**8. **K. Yonenaga and S. Kuwano, “Dispersion tolerant optical transmission system using duobinary transmitter and binary receiver,” J. Lightwave Technol. , **15**, 1530–1537 (1997). [CrossRef]

**9. **D. Penninckx, M. Chbat, L. Pierre, and J.-P. Thierry, “The Phase-Shaped Binary Transmission (PSBT): a new technique to transmit far beyond the chromatic dispersion limit,” IEEE Photon. Technol. Lett. , **9**, 259– 261 (1997). [CrossRef]

**10. **W. Kaiser, M. Wichers, T. Wuth, W. Rosenkranz, C. Scheerer, C. Glingener, A. Färbert, J.-P. Elbers, and G. Fischer, “SPM limit of Duobinary transmission,” *Proceedings of ECOC 2000* (Munich, Germany, Sep. 2000), 3, pp. 31–32,.

**11. **X. Zheng, F. Liu, and P. Jeppesen, “Receiver optimization for 40-Gb/s optical duobinary signal,” IEEE Photon. Technol. Lett. , **13**, 744–746 (2001). [CrossRef]

**12. **I. Lyubomirsky and B. Pitchumani, “Impact of Optical Filtering on Duobinary Transmission,” IEEE Photon. Technol. Lett. , **16**, 1969–1971 (2004). [CrossRef]

**13. **G. Bosco, A. Carena, V. Curri, R. Gaudino, and P. Poggiolini, “Modulation formats suitable for ultrahigh spectral efficient WDM systems,” IEEE J. Sel. Top. Quantum Electron. , **10**, 321–328 (2004). [CrossRef]

**14. **N. B. Pavloviæ and A. V. T. Cartaxo, “Optimized Bandwidth Limited Duobinary Coding Format for Ultra Dense WDM Systems,” *in Proc. of ICTON 2005*, paper We P.3.

**15. **G. Bosco, A. Carena, V. Curri, R. Gaudino, and P. Poggiolini, “Quantum limit of direct detection optically preamplified receivers using duobinary transmission,” IEEE Photon. Technol. Lett. , **15**, 102–104 (2003). [CrossRef]

**16. **G. Bosco and R. Gaudino, “On BER estimation in optical system simulation: Monte-Carlo vs. semi-analytical techniques”, ECOC 2000 Proceedings, Workshop on Modelling and design of optical networks and systems, (Munich, Germany, Sep.2000), paper 3.3.

**17. **E. Forestieri, “Evaluating the error probability in lightwave systems with chromatic dispersion, arbitrary pulse shape and pre- and post-detection filtering,” J. Lightwave Technol. , **18**, 1493–1503 (2000). [CrossRef]

**18. **J. S. Lee and C.S. Shim, “Bit error rate analysis of optically preamplified receiver using an eigenfunction expansion method in optical frequency domain”, J. Lightwave Technol. , **12**, 1224–1229 (1994). [CrossRef]

**19. **G. Bosco et al., “A novel analytical approach to the evaluation of the impact of fiber parametric gain on the bit error rate,” IEEE Trans. Commun. , **49**, 2154–2163 (2001). [CrossRef]

**20. **D. Penninckx, H. Bissessur, P. Brindel, E. Gohin, and F. Bakhtu, “Optical Differential Phase Shift Keying (DPSK) direct detection considered as a duobinary signal,” *Proceedings of ECOC 2001* (Amsterdam, Sep. 2001), paper We.P.40.