## Abstract

In this paper we evaluate numerically the advantage of combining convolutional coding and bandwidth-efficient modulation. We compare different multilevel modulation formats, line codes, and hard/soft decision decoding. Compared with DPSK modulation (with the same bandwidth and information transmission rate), an improvement of almost 5 dB is observed for bit error rates around 10^{-8}. We also study the robustness to intersymbol interference in the form of chromatic dispersion, and find that the improvement of the coded transmission lines improves over the uncoded even in presence of chromatic dispersion.

© 2006 Optical Society of America

## 1. Introduction

The growing demand on transmission bandwidth and cost reduction in optical communications (mainly driven by the increased Internet use and significant xDSL installation efforts) is stimulating the area of fiber optic transmission in novel research directions. The traditional way of coping with this required capacity increase has been to increase the number of wavelengths in wavelength-division multiplexed (WDM) systems. However, this may not be as cost effective as other methods, and therefore novel approaches to increase the capacity and reduce sensitivity to transmission impairments are being considered. Among these are novel, bandwidth-efficient modulation formats and different kinds of forward error-correction (FEC) coding.

Bandwidth-efficient modulation schemes, such as, e.g., duobinary modulation or differential phase shift keying (DPSK), offer a more relaxed distortion behavior under the impact of linear and nonlinear transmission effects than conventional on-off keying [1, 2]. Various high-rate block codes, such as Reed-Solomon (RS) codes [3] or low-density parity check (LDPC) codes [4] have been studied for improving the optical transmission. Their benefit is their modest bandwidth requirement (i.e., low overhead) which is usually below 10 %. This makes them less susceptible to bandwidth dependent signal distortions such as, e.g., chromatic dispersion.

Another class of error-correction codes, convolutional codes, has up to now been only rudi-mentarily explored for optical communications. The first investigations of optical systems using convolutional codes was done very early and in the context of pure position modulation [5]. The drawback of convolutional coding in optical transmission is a significant increase of the occupied bandwidth due to the high coding overhead, which in this study is 100 %, i.e., every information bit is encoded into two bits that will be transmitted as one symbol. On the other hand, convolutional codes have many attractive features such as simple encoding and decoding, since they work on continuous bit streams rather than blocks of data that need to be buffered and synchronized. This means that encoding and decoding circuits for convolutional coding can be implemented with a low gate count and low power consumption [6]. Additionally, both hard and soft decision decoding algorithms are straightforward to implement with these codes, for optimum sequence as well as bit detection [7, Ch. 12].

The high overhead has been the main obstacle for convolutional codes and rendered them little interest in fiber communications, since, e.g., the influence from chromatic dispersion would be a significant limitation. However, to deal with the increased bandwidth requirements from the convolutional encoding, and thus to reduce the influence of chromatic dispersion, we propose to use bandwidth-efficient modulation formats. To make the comparison fair, however, we intend to compare coded and uncoded systems with the same information bit rate and bandwidth. That they have the same bandwidth means that the influence from chromatic dispersion can be expected to be comparable. The recent paper by Bülow *et al*. [8] was the first idea in this direction, i.e., to combine multilevel modulation with convolutional coding in optical communications. Using DQPSK and soft decision decoding, an improvement of around 4 dB was found with respect to uncoded DPSK transmission [8]. A later extension with concatenating Reed-Solomon codes with convolutional codes was reported in [9]. The use of multilevel modulation together with low-rate LDPC and RS codes was discussed also in [10].

Here, we will investigate these findings in more detail, by comparing different multilevel modulation formats, hard/soft decision decoding, and also different convolutional codes. We will demonstrate that the concept of asymptotic coding gain can give simple and useful estimations on how much sensitivity improvement one can expect from the convolutional codes (and also block codes). We will also investigate the robustness to chromatic dispersion of these transmission systems.

Even though the convolutional codes has inferior coding gain compared to e.g. block codes such as Reed-Solomon (RS) codes and low-density parity check (LDPC) codes, we believe that this study is of interest for several other reasons: (i) There is more or less no fundamental knowledge of the performance of these codes in optical communications. (ii) Convolutional codes have some benefits (e.g. encoder complexity, as stated above) and this work may be a starting point for subsequent research on the concatenation of convolutional and block codes that have proved successful in other areas of communications research [9]. (iii) The powerful Viterbi decoder might (apart from enabling soft decision decoding) also be used for equalizing purposes in the receiver. (iv) Convolutional coding have successful applications in Trellis-coded modulation, as proposed in [11], where coding together with clever partitioning of a multilevel-modulation signal constellation can significantly improve the performance.

We will investigate two different bandwidth-efficient (i.e., multilevel) modulation formats, namely, differential phase shift keying/amplitude shift keying [12–15] and differential quaternary phase-shift keying [16]. It is customary to use return-to-zero (RZ) modulation of the amplitude; hence, the modulation formats are called RZ-DPSK/ASK and RZ-DQPSK, respectively. The reason for the use of RZ modulation in amplitude is an increased receiver sensitivity [17,18] and dispersion tolerance [18]. Both RZ-DPSK/ASK and RZ-DQPSK are four-level modulation formats, capable of transmitting two information bits per symbol, so by doing this, the number of transmitted bits can be doubled without sacrificing bandwidth, compared with two-level formats with the same pulse shape. Due to the doubling of the transmitted bits, a convolutional code with a code rate of 1/2 can now be applied. There are many such codes that can be used, whereof we will discuss three (see Section 2.5) with respect to complexity and performance. For decoding, Viterbi soft and hard decision decoders will be compared. Un-coded non-return-to-zero on-off keying (NRZ-OOK) and RZ-DPSK will be used as reference systems, by which we quantify the improvement of the coded multilevel modulated systems.

## 2. Simulation setup

The simulation setup, used for the investigations, is displayed in Fig. 1. The incoming 10 Gb/s data enter first the convolutional encoder, which generates two data streams, i.e., a total of 20 Gb/s, which are then differentially precoded and finally enter the optical transmitter. The exact structure of the transmitter (and of the precoder) depends on the modulation format and is discussed in sections 2.1 and 2.2 below. After signal generation in the transmitter, the signal is fed into a standard single-mode fiber (SMF) with dispersion 17 ps/(nm∙km). The fiber is considered to be a linear dispersive element (i.e., all nonlinearities are neglected), as we intend to focus on the robustness to dispersion. The information bit rate is set to 10 Gb/s. The carrier wavelength is 1550 nm. By launching different input powers into the erbium doped fiber amplifier (EDFA) we change the optical signal-to-noise ratio (OSNR), since the amplified spontaneous emission noise from the EDFA is constant. This noise is modeled as additive, white, Gaussian noise (AWGN) with a single-sided noise power spectral density given by ${N}_{0}\approx \frac{F}{2}\mathrm{hv}\left(G-1\right)$ [19, pp. 77, 100], where *F* is the noise figure of the optical amplifier, *h* is Planck’s constant, *v* is the frequency, and G is the amplifier gain. It is assumed that the signal and the noise occur only in one polarization. The EDFA gain is *G* = 30 dB and the noise figure is *F* = 5 dB. After amplification, the signal passes an optical bandpass filter (BPF) with Gaussian frequency response and is detected in the receiver. The 3 dB bandwidth of the optical BPF is 40 GHz.

The simulations used random data and were continued for each set of parameters until 30–100 errors were obtained. Although chunks of a few thousand data bits at a time were used in the simulation for practical reasons, these data sequences where joined together so that practically unlimited word lengths were used, in consistence with the endless nature of the convolutional codes. The BER levels at 10^{-8} and below required several weeks of computation time on a 17-node computer cluster.

In the following, we will describe in more detail the layout of the transmitter and receiver for the modulation formats we compare.

#### 2.1. RZ-DPSK/ASK transmitter

The DPSK/ASK modulation format (also called QDPASK) was suggested by Ohm *et al*. [12] and Liu *et al*. [13], and further investigated in [14] and [15]. It is attractive since it is a simple way to realize two independent transmission channels; one using amplitude modulation and one using phase modulation (specifically, DPSK).

The transmitter consists of three Mach-Zehnder modulators (MZM), two used for data modulation and one for the RZ pulse carving as used in [15]. The setup is displayed in Fig. 2, along with the constellation of the signal levels in the complex plane. The amplitude levels (denoted *a* and *b* in the figure) shall be chosen so that the bit-error rates of the DPSK and the ASK transmission lines are approximately equal. How this is done will be discussed in more detail in Section 2.3.

#### 2.2. RZ-DQPSK transmitter

The used RZ-DQPSK transmitter setup and its signal constellation in the complex plane are displayed in Fig. 3. The transmitter consists of three stages. In the first two stages, an RZ-DPSK signal is formed by two Mach-Zehnder modulators. In the third stage, four symbols are generated from the binary PSK signal by a phase modulator (PM). The two 10 Gb/s data streams, applied to the first MZM and the PM, are the precoded output signals of the DQPSK precoder.

#### 2.3. RZ-DPSK/ASK receiver

The receiver for DPSK/ASK splits the incoming signal into two paths, one detecting the amplitude modulation and one for the DPSK signal (see Fig. 4). The amplitude path consists of a detector and low pass filter (LPF), which is a third order Butterworth filter with a 3 dB cutoff frequency of 7 GHz. The filtered signal finally enters the sample and decision (S & D) gate. In the phase path, the DPSK data is demodulated with a delay interferometer, which delays the signal in one path one bit period. The recombined signal is detected in a balanced receiver, after which it enters the LPF and the S & D circuits.

The optimum signal constellation, which is specified by the levels *a* and *b* in Fig. 2, deserves a special discussion. In a system limited by optical noise, such as, e.g., an optically preamplified system in the limit of high signal-to-noise ratio, the system behaves as with an additive white Gaussian noise channel, where the bit error rate will be a function of the separation between the signal levels in the complex plane, and the optimum case will be when the separation is equal, i.e., *a*/*b* = 1/3. In terms of the optical extinction ratio *ER* this equals *ER* = 20log_{10}(*b*/*a*) = 9.5 dB as noted in [13] and [15]. As an aside, we might note that for a system dominated by electrical noise, e.g., a thermally limited system, we require the electric eye openings of the ASK signal (which is proportional to *b*
^{2} - *a*
^{2}) and the DPSK signal (which is proportional to 2*a*
^{2}) to be equal, which gives *a*/*b* = 1/√3, or *ER* = 4.8 dB.

A complication, however, with this theoretical level-spacing result is that it does not account for the additional intersymbol interference that will arise due to the receiver LPF. In our simulations, we have seen that this causes the optimum *a*/*b* ratio to be somewhat higher than the predicted value of 1/3, and we found, numerically, an optimum value of *a*/*b* = 0.398, or *ER* = 8.0 dB, which is actually exactly the same optimum values that was found in the experiments [13,14]. It is this optimum value of the *a*/*b* ratio that we use in the simulations.

In the simulations of DPSK transmission, the receiver is equivalent to the phase path in Fig. 4.

### 2.4. RZ-DQPSK receiver

In the RZ-DQPSK receiver (see Fig. 5), the signal is split into two paths. In both paths, delay interferometers with a differential delay of one symbol ±π/4 are used [21]. The interferometer outputs are fed into balanced receivers. The combined output of the photodiodes is then filtered by an electrical LPF, which is a third order Butterworth filter with a 3 dB cutoff frequency of 7 GHz. The filtered output is then sampled and detected. In the case of hard decision decoding, only one decision threshold is used. In the case of soft decision decoding (see Section 2.5), the simulated detector does not quantize the output at all. According to [8], the use of an analog-to-digital converter with a resolution of three or four bits results in a low additional penalty of 0.5 and 0.2 dB, respectively.

### 2.5. The convolutional encoder and decoder

A convolutional encoder consists of a shift register with *K* - 1 delay stages, where *K* is the *constraint length*
*K* of the convolutional code. The constraint length is defined as the number of sequential input bits that influence the output, and it is a good measure of the complexity of the code. In our case we require a code of rate 1/2, i.e., 2 outputs for each input bit, or 100 % overhead. There are many rate 1/2 convolutional codes that can be selected, and we chose to work with (in order of complexity) the (5,7), the (133,171) and the (1167,1545) codes. In previous works [8, 9], the simple (5,7) code was used, but we will here also investigate the performance of higher complexity codes. The performance of a code is related to the distance between the code words in signal space, and for convolutional codes one defines the *minimum free distance*, *d _{f}*, of the code as the minimum Hamming distance (the number of disagreeing bits) between any two unequal coded sequences. Some properties of these codes are listed in Table 1. The naming convention will be briefly explained below.

As an example, the structure of the (1167,1545)-encoder is shown in Figure 6, and we see that at any time instant, the output streams depend on *K* = 10 input bits. Specifically, the “data 1” stream can be obtained, at any time instant, by multiplying these 10 bits with the binary mask 1001110111 and adding the results using binary addition, i.e., an exclusive-or operation. The standard nomenclature for identifying convolutional encoders is to use the octal form of the binary masks, so in our case, “data 1” is identified by the octal number 1167. Similarly, the “data 2” output corresponds to 1545 in octal form, and the convolutional code is thus denoted (1167,1545).

The convolutional decoder we use is based on the Viterbi algorithm [7, Ch. 12], which is based on a trellis diagram with 2^{K-1} states. This implies that the decoding complexity of the (1167,1545)-code is too high to be attractive in practice, since the Viterbi algorithm needs to keep track of 2^{9} states. The today more reasonable (with respect to decoding complexity) code (133,171) is therefore also studied.

An important benefit of decoding convolutional codes using the Viterbi algorithm is that soft decision decoding is straightforward to implement. For block codes soft decision decoding involves a significant added algorithmic complexity. Soft decision decoding means that the power level for each bit that is used by the decoder to form a decoding decision is treated as continuous number, rather than as a bit that is either one or zero. These finer-grained signal levels will now make the distance metric between the possible signal sequences in the decoder Euclidean, rather than a simple bit-difference count (Hamming metric) as is used in hard decision decoding. This means that soft decision decoding will perform significantly better, and as a rule of thumb a gain of 3 dB can be expected by using soft decision decoders. In our simulations we will use analogue levels in the soft decoding algorithm, since the penalty incurred by using e.g. 3 bit (8 levels) of quantization have been shown previously to be rather small [8], around 0.5 dB.

It is customary to define a *coding gain*, which is the ratio (usually in dB) between the power needed in an uncoded and coded system, resp., to attain the same bit error rate (BER). Furthermore, the *asymptotic coding gain* (ACG) is defined as the limit of the coding gain as the BER approaches zero. It can be shown [7, pp. 17–18, 531–534] that the ACG (in dB) for transmission over an AWGN channel with optimal detection is simply *ACG* = 10log_{10}(*Rd _{f}*) for soft decision and

*ACG*= 10log

_{10}(

*Rd*/2) for hard decision, compared to an uncoded reference system with the same energy per information bit. For example, the (1167,1545) convolutional code has

_{f}*R*= 1/2 and

*d*= 12, implying that

_{f}*ACG*= 7.8 dB for soft decision and 4.8 dB for hard decision. Since the ACG denotes an asymptotic limit, we can interpret it as the best coding gain one can expect to find, but it says nothing about how fast (in terms of, e.g., the signal-to-noise ratio) one will reach this limit. To answer that issue, simulations were performed.

#### 3. Results

The improvements of convolutional coded, four-level modulated transmission compared to un-coded two-level DPSK were evaluated by numerical simulations with and without the dispersion of an SMF. The information data rate (10 Gb/s) and bandwidth where the same in all cases.

### 3.1. Back to back

First we analyze the performance of coded and uncoded RZ-DPSK/ASK with hard decision decoding, in comparison with conventional NRZ-OOK. This is shown in Figure 7. The BER is plotted as a function of the input signal power to the EDFA. Since the amount of noise generated by the EDFA is constant, the input signal power is also proportional to the OSNR. Here we see that the 10 Gb/s NRZ-OOK performs comparably with the uncoded, RZ-DPSK/ASK modulation format at 2x5 Gb/s and 3 dB better than 2x10 Gb/s RZ-DPSK/ASK. From a theoretical point of view, this is expected, as we will try to motivate. If the NRZ-OOK has a separation between the levels in signal space of Δ, it is reasonable to assume that the same separation between the ASK levels in the DPSK/ASK modulation (i.e., *b*-*a* = Δ) yields the same BER. We find that the average power required to get *a*/*b* = 0.398 (as was explained in Sec. 2.3) and *b*-*a* = Δ is (*b*
^{2} + *a*
^{2})/2 = 1.60Δ^{2}. Since the average power of NRZ-OOK is Δ^{2}/2, a power penalty of 1.60/0.5 = 3.2 = 5.0 dB can be expected for RZ-DPSK/ASK compared with RZ-ASK. The remaining 2 dB difference with respect to the observed 3 dB penalty, we attribute to the well-known performance difference between NRZ and RZ, which is in the order of 2 dB in favor of RZ, see, e.g., [22] and [23].

It should be mentioned that we compared the Monte Carlo-simulated back-to-back data with theoretical chi-2-distribution BER results based on the theory in [24], and found that the BER theory to be around 0.6 dB off from the Monte Carlo simulations. This is quite reasonable, and due to the idealized modeling of the optical and electrical filters in [24].

In Fig. 7, we show the results of using the two convolutional codes defined in 2.5. The ACG for the (133,171) code and hard decision is 4.0 dB according to Table 1, compared with the 4.8 dB for the (1167,1545) code. This agrees well with Fig. 7, where the (1167,1545) code yields a 4.8 dB improvement over the uncoded 2x5 Gb/s transmission, while the (133,171) code yields a 3.6 dB improvement. In the remaining parts of this paper, the (1167,1545) code is used.

These results can now be improved in two ways. First, the modulation format can be improved. The RZ-DQPSK is better in the sense that the separation between the points in signal space is larger for the same average power. Second, soft decision decoding can be used. The results of these improved systems are displayed in Fig. 8. Soft as well as hard decision Viterbi decoding is shown, and in addition, uncoded RZ-DPSK at 10 Gb/s is also shown, for reference. The hard decision DPSK/ASK curve is the same as the one shown in Fig. 7, and one can see that it performs around 1 dB better than the RZ-DPSK format. The separation between hard and soft decision decoding is around 3 dB for both modulation formats, in agreement with the ACG theory.

To quantify the influence of the bandwidth-efficient modulation and the convolutional coding, we use the RZ-DPSK system at a BER of 10^{-8} as a reference. This means that we measure the input power required to get a BER of 10^{-8} relative to -43.6 dBm. By using the RZ-ASK/DPSK transmission scheme, we find a gain of 0.7 dB for hard decision decoding, and about 3.4 dB for soft decision decoding. For the RZ-DQPSK transmission scheme, the coding gain is slightly better, being about 2.2 dB for hard decision and about 4.4 dB for soft decision, compared with DPSK. We can thus see that the RZ-DQPSK performs better than the RZ-ASK/DPSK, which we attribute to the larger separation between the modulation levels in signal space for DQPSK.

In the low-BER points we received at least 30 errors, which corresponds to an uncertainty in BER of 36 % (2 standard deviations), or 1/5 of a decade, which is around the size of the symbols in Figs 7 and 8. In the coded and dispersive cases the error may be larger, as some bit errors may have been correlated (due to the coding or dispersion), thereby reducing the number of uncorrelated errors. We did, however, not check the error correlation.

We may also note that the DPSK sensitivity (at a BER of 10^{-9}) we obtain in the simulations is -43 dBm, or only 3 dB from the quantum limit of 20 photons per bit (which corresponds to -46 dBm). This is not unrealistic, as experiments have reported sensitivities of 30 photons per bit at 10 Gb/s [25] and 38 photons per bit at 42.7 Gb/s [26].

### 3.2. Influence of dispersion

The coding gain obtained by using convolutional coding and bandwidth-efficient modulation can of course be used to increase system margins, amplifier spacings, etc., but we proceeded to investigate also the robustness to the intersymbol interference caused by chromatic dispersion, since intersymbol interference is not trivial to incorporate in a theoretical analysis of the transmission. We consider transmission through an SMF, as detailed in Sec. 1, with dispersion 17 ps/(nm km), and the amount of dispersion is discussed in terms of fiber length. Fiber nonlinearities are neglected, and also fiber losses, in order not to obscure the results.

In Fig. 9, the BER curves for fiber transmission of RZ-DPSK and coded RZ-DQPSK are shown with hard and soft decision. A bit error rate floor is clearly observed for the RZ-DPSK case and large amounts of dispersion. The transmission length with RZ-DPSK is limited to 62.5 km, since it was not possible to measure any useful (less than 10^{-4}) BER’s for RZ-DPSK over 75 km of fiber. This transmission limit applies also for coded transmission and hard decision Viterbi decoding. By using soft decision Viterbi decoding, the transmission length can be increased to 75 km.

The penalties induced by chromatic dispersion are defined relative to the back-to-back receiver sensitivity of RZ-DPSK at a BER of 10^{-5} (which is -45.8 dBm according to Fig. 7). As we can see from Fig. 9, the dispersion penalty at the 10^{-9}- level is the same for small distances, but for the longer distances the penalty is underestimated due to the BER floors. The penalties are plotted in Fig. 10.

First we may note that DPSK/ASK with hard decision decoding performs almost identically with uncoded DPSK. One reason for this is that we have limited the BER to 10^{-5}, where DPSK and DPSK/ASK(hard) have their crossover point (as shown in Fig. 8), and for lower BERs the coded transmission would be around 0.8 dB better. For soft decision decoding we can see that the power gain is 2 dB at 0 km and increases to 2.4 dB at 75 km.

The gain of DQPSK with convolutional coding and hard decision decoding compared with DPSK increases from 2.0 dB at 0 km transmission length to 2.6 dB at 62.5 km. In the case of soft decision Viterbi decoding, the coding gain also increases with the transmission length, from 3.9 dB for back-to-back transmission to 5.2 dB for 62.5 km. It can thus be seen that the coded transmission systems have an improved tolerance to dispersion. The reason for this is that the penalty of the uncoded DPSK reference system increases faster than that of the coded system. In fact, at a certain dispersion value, where the uncoded system has a BER floor at 10^{-5}, the penalty will approach infinity, and there the coding gain will also be infinite. We can observe that transmission lengths of 75 km can only be reached with the soft decision decoded systems, but then at fairly large transmission penalties.

### 3.3. Reed-Solomon and alternative coding

One might wonder how these systems perform compared with a conventional RS code. We can start by estimating the ACG for RS codes. For an RS(*n*,*k*) code, the rate is *R* = *k*/*n* and the minimum distance is *d* = *n*-*k* + 1. Thus *ACG* = 10log_{10}(*k*(*n*-*k* + 1)/2*n*) with hard decision decoding, and for the often used RS(255,239) code, we find *ACG* = 9.0 dB. This seems more impressive than the convolutional code, which has *ACG* = 7.8 dB with soft decision. The fact that the RS code is applied to DPSK and the convolutional code to DQPSK to maintain approximately the same bandwidth contributes an additional 2.3 dB to the advantage of RS [20, pp. 274–276]. To verify this, we performed simulations of RS(255,239)-coded data using the RZ-DPSK modulation format, at a 10 Gb/s information rate (i.e., a 10.7 Gb/s encoded data rate). The result is shown in Fig. 11. All optical and electrical filter bandwidths remained the same as in the previous simulations. Extrapolation of our data to BER around 10^{-9} in Fig. 11 nevertheless show that the convolutional code with soft decision performs similarly to the conventional RS coded system. The reason is that the asymptotic limits are reached faster for DQPSK with convolutional coding than for DPSK with RS coding. This shows clearly the importance of using both simulations and asymptotic theories when comparing different systems.

We should emphasize that if we would instead have compared different codes having similar code rates, it is well known that block codes such as Reed-Solomon and LDPC would significantly outperform the convolutional codes due to their superior distance properties. In fact, one way of improving the performance of the results we present here would be to replace the convolutional code in the DQPSK modulation format with a rate 1/2 block code. Studies along such lines have demonstrated that low-rate block codes together with DQPSK modulation can outperform high-rate codes with binary DPSK modulation, both with respect to bandwidth and receiver sensitivity [10].

The extra overhead required for a pure RS-coded system without the bandwidth reduction from the multilevel modulation may significantly affect the dispersion tolerance. However, only few studies of this issue have been published to date, e.g. by Agata et al. [27] in which the RS code overhead was optimized for a specific long-distance system configuration.

There is also a possibility to add soft decision decoding to the block codes. However, the potential 3 dB of improvement of that approach should be traded against a significantly increased level of complexity of the decoder. In fact, the encoder/decoder complexity and hence its price/availability is what ultimately determines what code to select. With the current rapid (and actually rather unpredictable) developments in high-speed electronics design it is difficult (and beyond our competence) to speculate in what constitutes high or low complexity encoders/decoders at 10 Gb/s. The results presented herein are timeless in the sense that they apply regardless of the available technologies.

#### 4. Conclusions and outlook

In this paper, we have evaluated the capability to increase the transmission quality over SMF’s by using convolutional coding for FEC. We argue that the large overhead of convolutional codes can be alleviated by using the codes together with bandwidth-efficient modulation. The performance can be comparable to that of RS codes, but with the benefit of simpler encoding electronics and the potential of using soft decision decoding.

From the results in Sec. 3 for back-to-back simulation at 10 Gb/s, we have found a sensitivity (at a BER of 10^{-8}) of -39.8 dBm for NRZ-OOK. This can be improved to - 43.6 dBm by using the RZ-DPSK modulation format. By using four-level modulation in form of RZ-DPSK/ASK and the (1167,1545)convolutional code, the sensitivity is improved to -44.4 dBm, and with the better RZ-DQPSK modulation format, -45.7 dBm can be reached. Finally, by using soft decision decoding on the convolutional code, we reach -48 dBm, which is an impressive 8.2 dB improvement over the conventional NRZ transmission, and a 4.4 dB improvement over RZ-DPSK.

We also evaluated the influence of intersymbol interference in the form of chromatic dispersion on the convolutional coded systems. We thus, for the first time, quantified dispersion robustness of combined coding and multilevel modulation for both soft and hard decoding. For uncoded RZ-DPSK transmission as well as for coded RZ-DQPSK transmission with hard decision Viterbi decoding, the SMF transmission distance (at a BER of 10^{-}5) is limited to 62.5 km. This is increased to 75 km by using coded RZ-DQPSK transmission and soft decision Viterbi decoding. In this case, the improvement over RZ-DPSK varies from 3.9 dB (back-to-back transmission) to 5.2 dB (62.5 km SMF).

Further improvements and some future research directions of these results could be for example: (i) the use of more powerful codes (e.g. low rate RS or LDPC codes) with the multilevel modulation format, (ii) coded modulation systems (such as e.g. trellis coded modulation), or (iii) concatenation of the convolutional coded modulation with a high-rate block codes [9]. Finally the inclusion of transmission nonlinearities is something that should be of great interest and suitable for future studies.

## References and links

**01. **O. Vassilieva, T. Hoshida, S. Choudhary, G. Castanon, H. Kuwahara, T. Terahara, and H. Onaka, “Numerical comparison of NRZ, CS-RZ and IM-DPSK formats in 43 Gbit/s WDM transmission”, *14th Annual Meeting of the IEEE Lasers and Electro-Optics Society (LEOS)*, 673–674, Nov. 12–13, 2001.

**02. **W. Kaiser, G. Mohs, T. Wuth, R. Neuhauser, W. Rosenkranz, and C. Glingener, “225 km repeaterless 10 Gb/s transmission over uncompensated SSMF using duobinary modulation and Raman amplification,” *14th Annual Meeting of the IEEE Lasers and Electro-Optics Society (LEOS)*, 155–156, Nov. 12–13, 2001.

**03. **T. Mizuochi, K. Kubo, H. Yoshida, H. Fujita, H. Tagami, M. Akita, and K. Motoshima, “Next generation FEC for optical networks,” *Proc. of Optical Fiber Communication Conference, OFC 2003*, paper ThN1, 2003.

**04. **I. B. Djordjevic, S. Sankaranarayanan, and B. Vasic, “Irregular Low-Density Parity-Check Codes for Long-Haul Optical Communications,” IEEE Photonics Technol. Lett., vol. **16**, pp. 338–340, 2004. [CrossRef]

**05. **E. Forestieri, R. Gangopadhyay, and G. Prati, “Performance of convolutional codes in a direct-detection PPM channel,” IEEE Transactions on Communications, **37**, 1303–1317, 1989. [CrossRef]

**06. **K. Seki, “Single-chip FEC codec LSI using iterative CSOC decoder for 10 Gb/s long-haul optical transmisison systems,” IEEE Custom Integrated Circuits Conference,Orlando, USA,155–158, May 12-15, 2002.

**07. **S. Lin and D. J. Costello Jr., *Error Control Coding*, 2nd ed., Prentice Hall, Inc., 2004.

**08. **H. Bülow, G. Thielecke, and F. Buchali, “Optical Trellis-Coded Modulation,” *Proc. of Optical Fiber Communication Conference, OFC 2004*, paper WM5, 2004.

**09. **P. Faraj, S. Schöllmann, J. Leibrich, and W. Rosenkranz, “8.4 dB net coding gain achieved with a serially concatenated coding scheme for differential quadrature phase shift keyed optical systems,” *Proc. of European Conference of Optical Communication, ECOC 2005*, paper Tu 3.2.4, 2005.

**10. **G. Kramer, A. Ashikhmin, A. J. van Wijngaarden, and X. Wei, “Spectral efficiency of coded phase-shift keying for fiber-optic communication,” J. Lightwave Technol. **21**, 2438–2445, 2003. [CrossRef]

**11. **G. Ungerboeck, “Channel coding with mulitlevel/phase signals,” IEEE Transactions of Information Theory **IT-28**, 55–67, 1982. [CrossRef]

**12. **M. Ohm and J. Speidel, “Quarternary optical ASK-DPSK and receivers with direct detection,” IEEE Photonics Technol. Lett. **15**, 159–161, 2003. [CrossRef]

**13. **X. Liu, X. Wei, Y.-H. Kao, J. Leuthold, C. R. Doerr, and L. F. Mollenauer, “Quartenary differential-phase amplitude-shift-keying for DWDM transmission,” *Proc. of European Conference on Optical Communication, ECOC 2003*, paper Th2.6.5, 2003.

**14. **X. Liu, X. Wei, Y.-H. Kao, J. Leuthold, C. R. Doerr, Y. Su, and L. F. Mollenauer, “Return-to-zero quaternary differential-phase amplitude-shift-keying for long-haul transmission,” *Proc. of Optical Fiber Communication Conference, OFC 2004*, paper FN2, 2004.

**15. **J. Hansryd, J. van Howe, and C. Xu, “Nonlinear crosstalk and compensation in QDPASK optical communication systems,” IEEE Photonics Technol. Lett. **17**, 232–234, 2005. [CrossRef]

**16. **C. Wree, J. Leibrich, and W. Rosenkranz, “RZ-DQPSK Format with High Spectral Efficiency and High Robustness Towards Fiber Nonlinearities,” *Proc. of European Conference on Optical Communication, ECOC 2002*, paper 9.6.6, 2002.

**17. **J.-X. Cai, C. R. Davidson, D. G. Foursa, L. Liu, Y. Cai, B. Bakhshi, G. Mohs, W. W. Patterson, P. C. Corbett, A. J. Lucero, W. Anderson, H. Li, M. Nissov, A. N. Pilipetski, and N. S. Bergano, “Experimental comparison of the RZ-DPSK and NRZ-DPSK modulation formats,” *Proc. of Optical Fiber Communication Conference, OFC 2005*, paper OThO1, 2005.

**18. **M. Ohm and J. Speidel, “Optimal receiver bandwidths, bit error probabilities and chromatic dispersion tolerance of 40 Gbit/s optical 8-DPSK with NRZ and RZ impulse shaping,” *Proc. of Optical Fiber Communication Conference, OFC 2005*, paper OFG5, 2005.

**19. **E. Desurvire, *Erbium Doped Fiber Amplifiers, Principles and Applications*, John Wiley&Sons, Inc., New York, 1994.

**20. **J. G. Proakis, *Digital Communications*, 4th ed., McGraw-Hill, NewYork, 2001.

**21. **R. A. Griffin and A. C. Carter, “Optical Differential Quadrature Phase-Shift Key (oDPSK) for High Capacity Optical Transmission,” *Proc. of Optical Fiber Communication Conference, OFC 2002*, paper WX6, 2002.

**22. **P. J. Winzer and A. Kalmár, “Sensitivity enhancement of optical receivers by impulsive coding,” J. Lightwave Technol. **17**, 171–177, 1999. [CrossRef]

**23. **M. Pauer and P. Winzer, “Impact of extinction ratio on return-to-zero coding gain in optical noise limited receivers,” IEEE Photonics Technol. Lett. **15**, 879–881, 2004. [CrossRef]

**24. **P. Humblet and M. Azioglou, “On the bit error rate of lightwave systems with optical amplifiers,” J. Lightwave Technol. , **9**, 1576–1582, 1991. [CrossRef]

**25. **W. A. Atia and R. S. Bondurant, “Demonstration of return-to-zero signaling in both OOK and DPSK formats to improve receiver sensitivity in an optically preamplified receiver,” *Proc. LEOS 12th Annual meeting*, 226–227, 1999.

**26. **J. S. Sinsky, A. Adamiecki, A. Gnauck, C. Burrus, J. Leuthold, O. Wohlgemuth, and A. Umbach, “A 42.7- Gb/s integrated balanced optical front end with record sensitivity,” *Proc. Conf. on Optical Fiber Communications, OFC’03*, paper PD39, 2003.

**27. **A. Agata, K. Tanaka, and N. Edagawa, “Study on the optimum ReedĐSolomon-based FEC codes for 40-Gb/s-based ultralong-distance WDM transmission,” J. Lightwave Technol. , **20**, 2189–2195, 2002. [CrossRef]