## Abstract

A new family of Nyquist pulses for coherent optical single carrier systems is introduced and is shown to increase the nonlinearity tolerance of dual-polarization (DP)-QPSK and DP-16-QAM systems. Numerical investigations for a single-channel 28 Gbaud DP-16-QAM long-haul system without optical dispersion compensation indicate that the proposed pulse can increase the reach distance by 26% and 19%, for roll-off factors of 1 and 2, respectively. In multi-channel transmissions and for a roll-off factor of 1, a reach distance increase of 20% is reported. Experimental results for DP-QPSK and DP-16-QAM systems at 10 Gbaud confirm the superior nonlinearity tolerance of the proposed pulse.

©2012 Optical Society of America

## 1. Introduction

Fiber nonlinearities pose a fundamental limit to the deployment of high speed long-haul optical systems. Essentially, by limiting the maximum launch power and thus, the optical signal-to-noise ratio (OSNR), nonlinear effects limit the transfer rates and the propagation distance that can be achieved.

In this paper, a family of Nyquist pulses for optical coherent communications is proposed. The new pulses are shown to improve the nonlinearity tolerance of coherent single carrier long-haul systems without optical dispersion compensation, to increase the maximal reach and to provide additional system margins.

## 2. Background

The effect of pulse shaping in optical coherent systems without optical dispersion compensation has recently been investigated in [1–3]. It was shown experimentally that for a 28 Gbaud dual-polarization (DP)-QPSK system, better nonlinear performance could be obtained by using a 50% return-to-zero (RZ) pulse instead of a non return-to-zero (NRZ) pulse. Reach increases of 18% for single-channel transmission and 24% for multi-channel transmission were reported [1]. For DP-16-QAM modulation, reach increases of 31% and 13% were obtained for single-channel and multi-channel experiments, respectively [2,3]. The better performance of the RZ pulse was explained by the fact that its wider spectrum reduces the phase-matching between adjacent frequency components as the signal propagates and experiences dispersion. However, as mentioned in [1], the larger signal bandwidth associated with the use of the RZ pulse implies a reduced spectral efficiency, and for Nyquist sampling at the receiver, higher analog-to-digital (ADC) clock speeds. Also, as stated in [1] and [2], due to its wider spectrum, the RZ pulse may be more sensitive to the effects of optical add/drop filter concatenation and it may increase WDM linear crosstalk.

In coherent optical communication systems using digital signal processing at the transmitter, more advanced pulse shapes than the NRZ and RZ can be considered. Using digital finite impulse response (FIR) filters and digital-to-analog converters (DACs), a wide variety of pulses can be generated. For example, by using the very well known root-raised cosine (RRC) pulse, the spectral efficiency of optical coherent single carrier systems can be improved compared to systems using the NRZ pulse [4–7], and better out-of-band attenuation can be achieved.

Other types of pulse shapes can be used to, for example, reduce the signal peak-to-average power ratio (PAPR) [8,9] or the system's sensitivity to timing jitter [10,11]. In wireless channels, nonlinear effects are mainly caused by the transmitter power amplifier and therefore any PAPR reduction of the transmitted waveform can improve the system performance. On the contrary, in optical channels, nonlinear effects are experienced all along the optical link, and accumulate as the signal propagates. Also, due to the dispersive nature of the fiber, the PAPR varies and usually increases as the optical signal travels along the link [12]. Therefore, in optical systems, reducing the PAPR by pulse shaping at the transmitter would not necessarily lead to PAPR reduction of the propagating signal. Furthermore, as experimentally shown in [1–3], the RZ pulse has better nonlinearity tolerance than the NRZ pulse despite the fact that it increases the PAPR of the transmitted signal by 3 dB. Clearly, in coherent systems without optical dispersion compensation, the PAPR of the transmitted waveform cannot be used as a figure of merit to predict a pulse shape performance under fiber nonlinearity. On the other hand, while jitter tolerance may be a concern for high-order QAM modulation formats, the effects of fiber nonlinearity are usually predominant. In high-speed optical links without optical dispersion compensation, fiber nonlinearity is the fundamental performance limiting factor, and system margins are most likely to be improved by mitigating nonlinear distortions.

A pulse shape optimized to reduce the effect of fiber nonlinearities was proposed in [13], and derived by taking into account the dispersive nature of optical channels. Compared to the RRC pulse, the optimized pulse was experimentally shown to improve both DP-QPSK [13] and DP-16-QAM [14] system performance and was found to significantly increase system margins. For DP-QPSK systems, numerical analysis further showed that the optimized pulse shape could increase the reach distance by more than 20%. The improved nonlinearity tolerance was achieved without increasing the signal bandwidth and thus without reducing the spectral efficiency. However, in [13] and [14], the optimized pulse shape was only derived and tested for an excess bandwidth of 100%. Furthermore, only single-channel transmission, where self-phase modulation (SPM) effects dominate, was studied. In practical wavelength division multiplexing (WDM) systems, many channels are transmitted simultaneously and the detrimental effect of cross-phase modulation (XPM) should also be considered.

In this work, a new family of Nyquist pulses specifically designed for WDM coherent optical communications is proposed. The proposed pulse is shown to improve the nonlinear tolerance of DP-QPSK and DP-16-QAM long-haul systems under the effects of SPM and XPM. Performance evaluations are provided for excess bandwidths between 0% and 200%, and while the optimized pulse in [13] and [14] was only compared to the RRC pulse, in this work, the performance of the proposed pulse is compared to the RRC, NRZ and RZ pulses.

The remainder of this paper is organized as follows. In section 3, analytical expressions of the proposed pulse transfer function are provided. In section 4, a description of the modeled long-haul coherent system is given. In section 5, the performance of the proposed pulse in a single-channel and multi-channel 28 Gbaud DP-16-QAM system is numerically compared to the RRC pulse in terms of maximum reach distance, sensitivity to truncation, sensitivity to quantization effects and sensitivity to timing jitter effects. Finally, in section 6, experimental results at 10 Gbaud are described in terms of bit error rate (BER).

## 3. Nyquist pulses for optical communication

Nyquist's first criterion [15] for zero ISI stipulates that the contributions at the sampling instants (*T*, 2*T*, 3*T*, …) of the received waveform must be zero. To obtain zero ISI, the impulse response of a Nyquist pulse shaping filter should meet the following condition:

In the frequency domain, this criterion can be expressed as:

As indicated by Eq. (2), the ISI condition is satisfied if the sum of the delayed pulse spectra by *k*/*T* equals to *T*, or more generally, to a constant level.

In [15], Nyquist also demonstrated that the maximal transfer rate for a channel having a bandwidth of *W* is 2*W* independent symbols per second. The pulse shaping filter having minimum bandwidth while satisfying the ISI condition expressed by Eq. (2) is thus given by:

The pulse shaping filter expressed by Eq. (3) corresponds to a raised cosine (RC) pulse with an excess bandwidth of 0%, or equivalently, having a roll-off factor (*α*) of zero. Its frequency response is represented in Fig. 1
for *T* = 1. It can be seen that it satisfies the ISI criterion expressed by Eq. (2), here shown for *k* ranging between –1 and 1. Figure 1 also shows that a RC filter with *α* = 1 satisfies the ISI criterion, since adding all the delayed spectra would produce a constant level of one. Furthermore, it can be observed that the bandwidth of a RC filter with *α* = 1 is twice the bandwidth of the minimum-bandwidth Nyquist pulse (*α* = 0), hence corresponding to an excess bandwidth of 100%.

Due to its extremely steep transition band, the impulse response decay rate of a RC filter with *α* = 0 is very slow. This translates in a waveform with a high PAPR that is also more sensitive to synchronization errors. Nevertheless, recent experiments on optical coherent systems [5–7] have demonstrated that it is possible to generate pulses with low roll-off factors and steep transition bands, while maintaining good system performance.

By increasing the roll-off factor, a more gradual transition band resulting in an impulse response with a faster decay rate is obtained, at the expense of a larger bandwidth. Using larger roll-off factors also allows the resulting digital filter to be more easily windowed and truncated. The RC response is given by:

It is well known that the signal-to-noise ratio is maximized when the transmitter and receiver filters are matched [16]. Therefore, in a system with zero pulse-induced ISI, their frequency responses correspond to $\sqrt{P\left(f\right)}$. Hence, if the RC pulse is chosen as the overall system response, the frequency response of the transmitter and receiver filters is usually given by the RRC filter.

As stated above, the main difference between an optical fiber channel without optical dispersion compensation and a wireless channel is the high amount of dispersion that the signal incurs in the optical case. Furthermore, while nonlinearities in a wireless channel are mainly due to the saturation of the transmitter power amplifier, in optical channels, the nonlinearities are mainly due to SPM and XPM, which are respectively a function of the instantaneous power of the propagating signal and the instantaneous power of the signals in the adjacent channels.

It is well documented that the nonlinear tolerance of high speed long-haul systems without optical dispersion compensation can be increased by using pulse shapes with broad spectra. For pre-compensated systems, it was shown that the use of the RZ pulse instead of the NRZ pulse leads to performance gains of more than 3 dB [17,18]. For post-compensated systems, it was shown that decreasing the duty cycle of the pulse, and hence increasing its energy content at high frequencies, leads to significant performance gains [19]. As previously mentioned, the better nonlinear performance of low duty cycle pulses was also experimentally reported in [1–3] for coherent systems. The increased nonlinear tolerance of pulse shapes with broad spectra can be explained by the fact that the evolution of these pulses with dispersion decorrelates the effect of SPM so that it does not add linearly. Similarly for XPM, the increased dispersion associated with pulse shapes having broad spectra leads to a fast evolution of the waveform profile and reduced nonlinear interactions between neighboring channels.

The proposed pulse is therefore designed to maximize the high frequency content of its spectrum, thereby increasing dispersion and maximizing its nonlinearity tolerance. It is also designed to have a finite bandwidth and to meet Nyquist's first criterion. While an infinite number of filter shapes can satisfy Nyquist's first criterion for *α* > 0, the rectangular shape is the low-pass filter shape that maximizes the most the energy content in the higher portion of the spectrum. The design of the pulse is thus achieved first, by specifying a frequency response that is a rectangular function, and then by manually setting its passband shape, so that the ISI criterion expressed by Eq. (2) is fulfilled. For excess bandwidth ranging from 0% to 100%, the proposed pulse response is given by:

For excess bandwidth between 100% and 200% it is given by:

It should be clear that the proposed pulse transfer functions expressed by Eq. (5) and Eq. (6) satisfy the ISI criterion defined by Eq. (2). Figure 2 depicts the frequency response of the RRC and proposed pulse for excess bandwidths of 50%, 100%, 150% and 200%.

The impulse response of the proposed Nyquist pulses is obtained by taking the inverse Fourier transform of $\sqrt{P\left(f\right)}$. For excess bandwidth between 0% and 100%, it is:

For excess bandwidth between 100% and 200% it is given by:

Figure 3 shows the impulse response of the proposed pulse compared to the impulse response of the RRC pulse for excess bandwidths of 50%, 100%, 150%, and 200%. It can be seen that in all cases the pulse width of the proposed pulse is shorter, corresponding as highlighted in Fig. 2, to a spectrum with more energy content in the higher frequencies. Due to the fact that these types of pulses experience more dispersion, it is expected that they will provide increased nonlinearity tolerance.

## 4. System model

Figure 4
illustrates the simulation setup. The system under study is a DP-16-QAM system operating at 28 Gbaud, resulting in a bit rate of 224 Gb/s. The transmitter model mainly consists in a random symbol generator of length 2^{15}, pulse shaping finite impulse response (FIR) filters, 6-bit digital-to-analog converters (DACs), in-phase-quadrature (I-Q) modulators and a polarization beam combiner (PBC). As in [3], random noise is added to the electrical driving signals to represent residual implementation penalties. The level of the noise source is adjusted in order to reproduce the experimental back-to-back transmission results presented in [3]. Five channels, each having independent symbol sources, are modeled. The emitting wavelength of the center channel (TX3) is set to 193.1 THz. Two types of filter responses are considered for the WDM multiplexer (Mux): a rectangular response and a Gaussian response. For the rectangular transfer function, the bandwidth of each channel corresponds to the channel spacing. For the Gaussian response, a 4th-order bandpass filter is used with its 3-dB cutoff point corresponding to 75% of the channel spacing. A fiber link consisting of 80 km spans of G.652 fiber is modeled, together with an EDFA at each span. The fiber and link parameters correspond to those described in [3]: the fiber attenuation is set to 0.19 dB/km, the dispersion to 16.87 ps/nm/km, the dispersion slope to 0.06 ps/nm^{2}/km, the effective area to 80 µm^{2}, the nonlinear coefficient to 1.2 W^{−1}km^{−1}, the PMD coefficient to 0.1 ps/km^{0.5}, and the EDFA noise figure to 4.5 dB. A linewidth of 100 kHz is specified for the transmitter and receiver lasers, and the frequency offset between the transmitter and receiver laser is assumed to be negligible. The value *L* corresponds to the number of spans. At the receiver, the WDM demultiplexer (Demux) is modeled with the same bandpass filters as the transmitter Mux. Following the Demux, a coherent front-end integrates polarization beam splitters, optical hybrids, a local oscillator and the photodetectors [12]. It provides four signals corresponding to the in-phase and quadrature components of the two polarizations. Analog-to-digital converters (ADCs) with 8-bit resolution are modeled. Matched filtering and chromatic dispersion compensation (CD^{−1}) are implemented by FIR filters. The final stages of the receiver digital signal processing functions include a 13-tap fractionally-spaced decision-directed butterfly equalizer (EQ) and a decision-directed second order phase-lock loop (PLL) [12]. For SPM simulations, only the center channel transmits; channels 1, 2, 4 and 5 are turned off. For XPM simulations, all channels transmit simultaneously.

All simulations are done using Matlab (R2010a) for the transmitter and receiver digital signal processing blocks and OptiSystem (9.0) for the analog front-end blocks, and for fiber transmission. The oversampling rate of the transmitter and receiver digital signal processing functions is chosen to avoid aliasing. It is set to 2 for roll-off factors between 0 and 1, and to 4 for roll-off factors between 1 and 2. The oversampling rate of the analog portion of the model is set to 32.

## 5. Numerical analysis

In this section, the system performance is evaluated for different pulse shapes, considering single-channel and multi-channel transmission, pulse truncation, quantization effects and timing jitter. The RRC and proposed pulse lengths are 64 symbol periods, which provides a good compromise between performance and complexity. Their impulse responses are obtained using the procedure detailed in section 5.3.

#### 5.1 System performance in single-channel transmission

The pulse performance in a single-channel transmission scenario is first analyzed using the NRZ, RRC and proposed pulse. It should be noted that the NRZ and RZ pulses are both Nyquist and root-Nyquist pulses, and that they can therefore be directly used in a matched filter configuration. To highlight the effects of single-channel nonlinearity, the Mux and Demux blocks are in this case bypassed. The influence of the Mux/Demux response is studied in section 5.2.

Figure 5(a)
shows the spectrum of the 28 Gbaud DP-16-QAM signal when using the NRZ, RRC and proposed pulses for an excess bandwidth of 100%. It can be observed that the null-to-null bandwidth of the NRZ, RRC, and proposed pulses is the same and that it corresponds to 2/*T*, or in this case to 56 GHz. But has can be seen in Fig. 5(a), the out-of-band attenuation of the NRZ pulse is lower than it is for the two other pulses. The RRC and proposed pulses have an out-of-band attenuation of more than 40 dB. The out-of-band attenuation of the RRC and proposed pulse is limited in this system by the finite resolution of the DACs.

Figure 6(a)
shows the achievable reach distance against the launch power of the modeled 28 Gbaud DP-16-QAM system, for single-channel transmission. To allow direct comparisons with the simulation and experimental results reported in [3], a forward error correction (FEC) threshold of 3 × 10^{−3} is considered. It can be seen that in the linear regime, for launch powers between –7 dBm and –3 dBm, the system performance when using the NRZ, RRC and proposed pulses is practically the same. This is explained by the fact that these pulses are all matched root-Nyquist pulses satisfying the ISI criterion. It can also be seen that the performance of the NRZ and RRC pulses is almost the same in the nonlinear regime, from –3 dBm to 4 dBm, resulting in a maximum reach of around 1530 km. This can be understood by the fact that they have similar frequency characteristics in the passband region, as shown in Fig. 5(a). Therefore, when the NRZ and RRC pulses propagate, they experience similar dispersion, resulting in comparable sensitivity to SPM effects. It can also be seen in Fig. 6(a) that the maximal reach obtained with the proposed pulse is 1963 km, corresponding to a 26% reach increase.

Next, the performance of the RZ pulse is compared to the performance of the RRC and optimized pulses. An excess bandwidth of 140% is chosen to obtain similar nonlinearity tolerance between the RZ and the proposed pulse. Figure 5(b) shows that the null-to-null bandwidth is 112 GHz when using the RZ pulse, corresponding to an excess bandwidth of 300%. When using the RRC and proposed pulses with *α* = 1.4, the null-to-null bandwidth of the 28 Gbaud DP-16-QAM signal is 67.2 GHz. Figure 5(b) indicates that the out-of-band attenuation provided by the RZ pulse is similar to the out-of-band attenuation obtained when using the NRZ pulse, and that better out-of-band rejection can be achieved by the RRC and proposed pulses.

Figure 6(b) compares the achievable reach distance for a system using the RZ, RRC and proposed pulses for single-channel transmission. It can be seen that the use of the RZ and proposed pulse provides an increase in reach distance of 411 km and 465 km, representing a 25% and 29% increase compared to the RRC pulse, respectively. The similar performance of the RZ and proposed pulse with an excess bandwidth of 140% can be explained by the fact that they have similar pulse widths, resulting in similar sensitivity to dispersion and nonlinear effects. Although the RZ and proposed pulse with an excess bandwidth of 140% perform similarly, the proposed pulse occupies much less bandwidth, as indicated in Fig. 5(b), and is therefore more spectrally efficient. In terms of optimal launch power levels and maximal reach distances, the simulation results obtained for the NRZ and RZ pulse are in very good accordance with the experimental results presented in [2,3], therefore validating the presented numerical analysis.

Figure 7
shows the maximum reach distance that can be obtained using the RRC and the proposed pulses for roll-off factors between 0.1 and 2. It can be seen that the proposed pulse outperforms the RRC pulse for *α* > 0.3. For lower roll-off factors, the frequency characteristics of the proposed pulse is very similar to the frequency characteristics of the RRC pulse, explaining the absence of significant improvements. As the roll-off factor is increased, the maximum reach of the system for both pulses is increased, due to the larger bandwidth that is occupied and to the increased dispersion. For roll-off factors of 0.5, 1.0, 1.5 and 2.0, the increase in reach distance when using the proposed pulse is respectively 9%, 26%, 26%, and 19%.

Since the amount of dispersion experienced by the signal is directly proportional to the energy content of the pulse in the high frequency region of its spectrum, a simple metric is proposed in order to predict a pulse shape tolerance to SPM effects. It is calculated by taking the ratio of the root-Nyquist pulse energy between 1/(2*T*) and infinity, to its total energy:

The energy ratio (*E _{R}*) obtained using Eq. (9) as a function of the roll-off factor is displayed in Fig. 8
for the RRC and proposed pulses. It can be seen that the proposed metric is directly proportional to the system maximum reach for single-channel transmission, presented in Fig. 7. For the RRC pulse, the energy ratio increases almost constantly as the roll-off factor is increased. In the case of the proposed pulse, the energy ratio increase rate is higher for roll-off factors between 0 and 1, than it is for roll-off factors between 1 and 2. This can be explained by considering the proposed pulse transfer function expressed by Eq. (5) and Eq. (6). For 0 <

*α*≤ 1, the magnitude of the spectrum for frequencies higher than 1/(2

*T*) is $\sqrt{T/2}$, but for 1 <

*α*≤ 2, the magnitude varies as a function of the roll-off factor and is only $\sqrt{T/3}$ for $\left(3-\alpha \right)/2T\le \left|f\right|<\left(1+\alpha \right)/2T$. Therefore, integrating the proposed pulse frequency response from 1/(2

*T*) to infinity in the case of 1 <

*α*≤ 2 results in a lower energy ratio increase rate. The excellent correspondence between the energy ratio and the maximum reach further indicates that for coherent systems without optical dispersion compensation, the energy content of the pulse spectrum in its higher portion determines its tolerance to single-channel fiber nonlinearity, as previously suggested in section 3.

#### 5.2 System performance in multi-channel transmission

Figure 9
shows the maximum reach that can be attained in the presence of XPM effects, for multi-channel transmission, when a rectangular Mux/Demux frequency transfer function is specified. For a channel spacing of 50 GHz, Fig. 9(a) indicates that reach increase of 2%, 9%, and 13% are obtained for roll-off factors of 0.4, 0.6, and 0.8. For a channel spacing of 100 GHz, Fig. 9(b) shows that reach increase of 13%, 18%, 23%, and 17% are possible, for roll-off factors of 0.6, 1.0, 1.4, and 1.8. For a channel spacing of 50 GHz and *α* = 0.9, the use of the proposed pulse decreases the maximum reach of the system. This is due to the fact that the bandwidth of the signal is then 53.2 GHz, and that it exceeds the 50 GHz Mux/Demux bandwidth. The Mux and Demux alter the spectrum of the signal, breaking both the matched filter and the ISI conditions. Although the same phenomenon occurs with the RRC pulse, it has less energy content at high frequencies and is therefore less affected by the Mux and Demux responses. For a channel spacing of 100 GHz, a similar effect is observed in Fig. 9(b) reducing the maximal reach at a roll-off factor of 2.0.

Figures 10(a) , 10(b) and 10(c) show the maximum reach distance of the system when a Gaussian filter is specified for the Mux and Demux responses, and for varying channel spacing. For a roll-off factor of 0.5, the proposed pulse improves the performance when the channel spacing is greater than 60 GHz, and the reach distance is increased by at least 7% for channel spacing higher than 62.5 GHz. For a roll-off factor of 1, the system performance using the proposed pulse is improved for a channel spacing greater than 69 GHz, and for channel spacing of 75 GHz and 100 GHz, the maximum reach is increased by 20% and 18%, respectively. For a roll-off factor of 1.5, the proposed pulse performs better than the RRC pulse for channel spacing higher than 88 GHz. An increased in reach distance of more than 14% is reported for channel spacing higher than 100 GHz.

Globally, these results indicate that, due to its rectangular spectrum, the proposed pulse is more sensitive to the filter characteristics of the Mux and Demux. In the case of a Gaussian frequency response, Figs. 10(a), 10(b) and 10(c) suggest that for improved system performance, the bandwidth of the signal when using the proposed pulse should be approximately less than 75% of the channel spacing, corresponding in this case to the 3 dB cutoff point of the Gaussian filter frequency response. The tolerance of the proposed pulse to filtering effects could be improved by pre-compensating the signal for the Mux/Demux response, as experimentally demonstrated in [20].

#### 5.3 Effect of pulse truncation

In traditional pulse shape design, windowing is often used to reduce the impulse response length and to preserve good out-of-band attenuation. Although this method can be efficient, it usually modifies the frequency response of the pulse and increases its bandwidth. The application of a window therefore modifies the spectral characteristics of the pulse and may lead to the violation of Nyquist's first criterion. An alternative approach for impulse response truncation consists in relying on numerical optimization to explicitly formulate the ISI constraint. The optimization function used to obtain the truncated version of the proposed pulse in this work is similar to the one suggested in [13], except that the bandwidth and stopband constraints are incorporated in the objective function. The optimization problem is described as:

*s*(

*n*) is the impulse response of the optimized FIR pulse shaping filter,

*M*is the impulse response length in symbol periods,

*N*corresponds to the oversampling rate,

*n*is the discrete time index (

*n*= –

*MT*/2, ..., –2

*T*/

*N*, –

*T*/

*N*, 0,

*T*/

*N*, 2

*T*/

*N*, ...,

*MT*/2),

*S*(

*f*) is the Fourier transform of

*s*(

*n*), and

*f*is the frequency index. The number of coefficients of the FIR filter

*s*(

*n*) is thus given by

*M*∙

*N*+ 1. By maximizing the pulse energy of the center coefficient

*s*(0), the first term of the objective function seeks to minimize the pulse width, and consequently the energy content in the higher part of its spectrum. By maximizing the energy of

*S*(

*f*) between 0 and $\left(1+\alpha \right)/2T$, the second term allows to control the bandwidth of the pulse. And by adjusting the optimization weight

*γ*relative to the optimization weight

*µ*, the stopband attenuation can be set. To assure zero ISI, or equivalently that Nyquist's first criterion is met, the following constraint is formulated:

This constraint corresponds exactly to the ISI criterion expressed by Eq. (1), where in this case *r*(*n*) is a Nyquist pulse given by $r\left(n\right)=s\left(n\right)\otimes s\left(n\right)$, with $\otimes $ denoting the convolution operator. In this problem formulation, *s*(*n*) corresponds to a root-Nyquist pulse and matched filtering is achieved since *s*(*n*) is used both at the transmitter and receiver. A final constraint ensures that the filter impulse response is symmetric:

The optimization problem is solved using the sequential quadratic programming (SQP) algorithm [21]. Figure 11
shows the optimized frequency responses of the proposed pulse for a roll-off factor of 0.5 and for impulse responses truncated to 16, 32 and 64 symbol periods, compared to the ideal frequency response that would be obtained using Eq. (7) and a length *M* of 2048 symbol periods. It can be seen that the first-null bandwidth of the optimized responses is exactly the same as of the ideal response, but that as the length of the impulse response is reduced, the amount of energy in the higher part of the passband (from 0.65 Hz to 0.75 Hz) is also reduced. Figure 11 also shows the frequency response of the proposed pulse using a rectangular window and *M* = 64. In this case, the side-lobe attenuation is reduced to only 24 dB, while a side-lobe attenuation of at least 40 dB is achieved by all the optimized truncated versions of the pulse, thereby showing the effectiveness of the proposed truncation procedure.

Figures 12(a)
, 12(b) and 12(c) show the theoretical eye diagrams at the receiver (after matched filtering) using the RRC pulse with rectangular windowing, the proposed pulse with rectangular windowing, and the proposed pulse with optimized truncation, respectively, for a 16-QAM sequence with *α* = 0.5, *T* = 1, *N* = 16 and *M* = 64. To produce these eye diagrams, 2^{18} symbols were used and no noise was added to the signal. It can be seen in Fig. 12(a) that truncating the RRC pulse response to 64 symbol periods with a rectangular window results in a 16-QAM signal with zero ISI. On the contrary, when using a rectangular window and truncating the proposed pulse response to 64 symbol periods, Fig. 12(b) shows that the level of ISI is increased, since the vertical eye opening is reduced by 20%, from 2.0 to 1.6. When using the truncation procedure described above for the proposed pulse and *M* = 64, Fig. 13(c)
demonstrates that the vertical eye opening is optimal and that the proposed pulse obtained by numerical optimization meets the zero ISI constraint.

Table 1
indicates the maximum reach that can be achieved for a 28 Gbaud DP-16-QAM system for single-channel transmission, using truncated versions of the RRC and proposed pulses. The RRC pulse truncation was done using a rectangular window. To obtain the truncated version of the proposed pulse, the optimization problem described by Eq. (10), Eq. (11) and Eq. (12) was solved for each case. For a roll-off factor of 0.5, Table 1 indicates that the effect of truncation is not negligible. For *M* = 16 the reach increase is only 3.7%, while it is 7.6% and 8.9% when *M* = 32 and *M* = 64, respectively. This variation of performance can be explained by comparing the frequency responses of the truncated pulses in Fig. 11, that show less energy content in the high frequencies, and therefore less nonlinearity tolerance for reduced pulse lengths. For a roll-off factor of 1.0, the effect of truncation is less important since the truncation to 16 symbol periods does not bring significant penalties compared to pulse lengths of 32 and 64 symbol periods. For a roll-off factor of 1.5, the proposed pulse truncation method has no effect on system performance. Since the RRC pulse has a smoother transition band and a faster decay rate, its truncation does not lead to any performance variations, for the considered roll-off factor values and impulse response lengths.

#### 5.4 Effect of DAC resolution

As it can be observed in Fig. 3, the proposed pulse has a slower decay rate in comparison to the RRC pulse, which leads to a modulated signal with a higher PAPR. For instance, the PAPR of a baseband 16-QAM signal with an excess bandwidth of 50%, 100% and 150% is increased by 0.8 dB, 1.5 dB, and 2.1 dB, respectively, when using the proposed pulse instead of the RRC pulse.

To study the impact of this PAPR increase on system performance, the achievable reach of the 28 Gbaud DP-16-QAM system described in section 4 is evaluated for a DAC resolution of 3 bits to 8 bits. The impulse response of the pulse shaping filters is truncated to 64 symbol periods and is obtained using the procedure detailed in section 5.3. For an excess bandwidth of 50%, Fig. 13(a) indicates that a minimum of 4 bits are required in order to obtain a reach increase of 12%. When using 6 bits, Fig. 13(a) shows that the reach distance of the system using the RRC pulse is increased, resulting in a reach distance improvement of 9%. Figure 13(b) indicates that a reach increase of at least 24% can be obtained for an excess bandwidth of 100%, when using the proposed pulse and a resolution of 4 bits or more. For an excess bandwidth of 150%, Fig. 13(c) shows that DACs with 4 bits of resolution allow a reach increase of 18%. But, in order to obtain a 24% reach increase or more, a minimum of 5 bits are required. This comes from the fact that, as discussed above, the PAPR increase associated with the use of the proposed pulse with 150% excess bandwidth is higher than it is for excess bandwidths of 50% and 100%. This translates into a transmitted signal that has a larger dynamic range and that is therefore slightly more sensitive to quantization effects.

In terms of absolute performance, for all the considered excess bandwidths and both for the RRC and proposed pulses, Figs. 13(a), 13(b) and 13(c) show that a minimum of 4 bits are required to obtain long-haul transmission distances. For all cases, maximum reach distances are obtained with 7 bits of DAC resolution.

It should be noted that reducing the RRC and proposed pulse length to 32 or 16 symbol periods does not significantly changes the PAPR of the transmitted signal and consequently does not affect the DAC resolution requirements.

#### 5.5 Performance under timing jitter

The simulation setup used to study the pulse sensitivity to timing jitter is detailed in Fig. 14
. It consists of a complex random integer source of length 2^{15}, followed by a 16-QAM mapper and by a FIR pulse shaping filter. The channel is modeled by an additive white gaussian noise (AWGN) channel. At the receiver, a sampler acquires the signal. In practical systems, the sampler function is realized by ADCs. Timing detection is accomplished by Gardner's timing error detector (TED) [22]. The input of the TED is taken either at the ouptut of the matched filter or at the output of a prefilter. Following the TED, a second order loop filter and a numerically controlled oscillator (NCO) are used to control the sampling instant. A noise loop bandwidth of *T*/1000 is chosen and the length of the FIR pulse shaper, matched filter and prefilter is set to 64 symbol periods. The oversampling rate of the matched filter, prefilter, TED and loop filter is set to *N* = 2. The model was implemented in Simulink (R2010a).

The prefilter task is to enhance the performance of the TED, by minimizing the variance of the error signal. This is achieved by imposing [23]:

andwhere*g*(

*t*) corresponds to the convolution of the channel impulse response

*p*(

*t*) and the prefilter

*h*(

_{p}*t*). It is shown in [23] that the prefilter impulse response satisfying Eq. (13) and Eq. (14) is obtained by translating the overall channel function

*P*(

*f*) by ± 1/

*T*. In the time domain, this corresponds to:

The 16-QAM theoretical eye diagrams obtained after the transmitter pulse shaping filter using the RRC pulse and the proposed pulse for *α* = 1 are shown in Figs. 15(a)
and 15(d), respectively. Some level of ISI is present, since the signal at the output of a root-Nyquist pulse shaping filter is not necessarily ISI free. Figures 15(b) and 15(e) show the eye diagrams obtained after the receiver matched filter for the RRC and proposed pulse, for a channel without noise. As expected, both signals are free from any ISI. Figures 15(c) and 15(f) show the eye diagrams at the output of the prefilter for a sequence filtered by the RRC and the proposed pulse. As it can be seen, the output of the prefilter has regular *T*-spaced zero-crossings.

A lower bound on the jitter variance is given by the modified Cramer-Rao bound (MCRB) which can be expressed as [24]:

*B*is the noise loop bandwidth, E

_{L}_{b}/N

_{0}is the energy per bit and

*M*is the order of the QAM modulation. Figures 16(a) and 16(b) show the normalized jitter variance against the E

_{QAM}_{b}/N

_{0}, for roll-off factors of 0.5 and 1.0. The MCRB of the proposed pulse is lower than the MCRB of the RRC pulse, suggesting that the former has better synchronization properties. Figures 16(a) and 16(b) confirm this expectation, and show that when the input of the TED is the matched filter (i.e. no prefilter is used), a lower jitter floor is reached. For a roll-off factor of 0.5, the minimum jitter variance is 5.5 × 10

^{−6}and 1.8 × 10

^{−5}for the proposed and RRC pulses, respectively. For a roll-off factor of 1, the minimum jitter variance is 3.5 × 10

^{−7}and 3.5 × 10

^{−6}for the proposed and RRC pulses, respectively. When using a prefilter, Figs. 16(a) and 16(b) show that the effect of self-noise is practically removed and that the proposed pulse has better jitter tolerance. For a jitter variance of 10

^{−5}, performance gains of 4.0 dB and 3.9 dB are obtained for roll-off factors of 0.5 and 1.0, respectively.

In clock recovery schemes for optical long-haul transmissions, the effects of CD and more particularly PMD should be taken into account. For proper operation of the timing recovery circuit depicted in Fig. 14, the signal at the input of the prefilter should already be compensated for CD and PMD. The prefilter could therefore be placed after the butterfly equalizer, in a system performing joint adaptive equalization and clock recovery such as the one proposed in [25] for optical coherent systems.

## 6. Experimental results

The experimental analysis was conducted for single-channel DP-QPSK and DP-16-QAM systems without optical dispersion compensation, operating at 10 Gbaud. The experimental setup is described in Fig. 17
. The dual-polarization transmitter (DP-Tx) architecture was the same as the one described in Fig. 4. In this case, the random symbol patterns were of length 2^{14}, but two symbol pattern periods, totaling 2^{15} symbols for each polarization, were used to produce BER statistics. Prior to launching the signal in the fiber, a polarization scrambler (PS) was added to continuously vary the polarization state. The optical link consisted of *L* G.652 fiber spans of 80 km. Erbium doped fiber amplifiers (EDFAs) were used to amplify the signal. At the receiver a broadband noise source was inserted in order to adjust the OSNR and an optical spectrum analyzer (OSA) was used to determine the OSNR. Prior to demodulation, a 50 GHz Demux was inserted. The receiver architecture is also similar to the one described by Fig. 4. The received signal first went through a coherent front end. An 8-bit high speed real-time oscilloscope operated at 50 GSamples/s and having a 3 dB bandwidth of 16 GHz was then used to digitize the signal. The same clock source was driving the DACs and the oscilloscope. Off-line processing of the remaining digital signal processing functions was done on a personal computer. A quadrature-imbalance compensation (QIC) circuit [26] was followed by matched filters and chromatic dispersion compensation, which were both implemented by FIR filters. A 15 taps *T*/2 fractional equalizer (EQ) was then used to recover the signal from the two polarizations. For the DP-QPSK experiment, a feedforward Viterbi-Viterbi [27] carrier recovery (CR) and a blind constant modulus algorithm [28] were implemented. For the DP-16-QAM experiment, equalization was performed by a decision directed least-mean square *T*/2 equalizer, while carrier recovery was achieved by a second order PLL. The dashed lines in Fig. 17 between the decision circuit, the CR and the *T*/2 EQ represent the feedback paths for the DP-16-QAM implementation.

As for the numerical analysis presented in section 5, the system performance was evaluated by selecting either the RRC pulse or the proposed pulse, for *α* = 1, as transmit pulse shaping and receive matched filters. Note that the proposed pulse response was in this case obtained by the optimization problem defined in [13], with *M* = 16. For a roll-off factor of 1, the formulation in [13] practically gives the same result as the formulation expressed by Eq. (10), Eq. (11), and Eq. (12).

Figure 18
summarizes the DP-QPSK experiment results. In the back-to-back configuration, the simulation and experimental results presented in Fig. 18(a) show that both pulses yield similar performance. Considering as in [12] and as in the authors' previous work on optimized pulse shaping [13,14] a FEC threshold of 3.8 × 10^{−3}, the measured BER penalty is only 0.2 dB and 0.4 dB for the RRC and proposed pulse, respectively. Figure 18(b) shows that for a 1200 km transmission and a launch power of 2 dBm, the measured and simulated nonlinear tolerance improvement provided by the optimized pulse is 1.0 dB and 0.9 dB, respectively. When the launch power is increased to 4 dBm, Fig. 18(c) shows that the system with the RRC pulse does not reach the FEC threshold. At this power level, the measured and simulated added margins are at least 4.7 dB and 4.8 dB, respectively. The simulation results presented in Fig. 18 are in good accordance with the measured data.

The back-to-back result for the 10 Gbaud DP-16-QAM experiment is presented in Fig. 19(a)
. Simulations and measurements show that both pulses perform similarly but in this case, the result is further away from the theoretical prediction. This can be explained by the fact that the four hand-crafted 6-bit high-speed DACs that were used in the experiment had an effective number of bits (ENOB) of 3.6, limiting the performance of the DP-16-QAM experiment in back-to-back transmission. Figure 19(b) indicates that at 1200 km, for a FEC threshold of 8 × 10^{−3} and for a launch power of –2 dBm, an increased tolerance to fiber nonlinearity of 1.2 dB is provided by the proposed pulse. At this power level, simulation results indicate a 0.9 dB of added tolerance to fiber nonlinearity. At 1200 km and for a launch power of 0 dBm, Fig. 19(c) shows that the system with the proposed pulse almost reaches the BER threshold, while the system using RRC pulses is limited to a BER of 1.8 × 10^{−2}. At this BER level, the optimized pulse outperforms the RRC pulse by 4.3 dB. Figures 19(a), 19(b) and 19(c) show good general agreement between the experimental results and the simulation model. It should be noted that the introduction of soft-decision forward error correction (FEC) [29] would be required to be able to operate at a FEC threshold of 8 × 10^{−3}.

## 7. Conclusion

A new class of Nyquist pulses has been proposed for coherent optical communications. The new pulses were shown to significantly increase the nonlinear tolerance of DP-QPSK and DP-16-QAM systems without optical dispersion compensation, in the presence of both SPM and XPM. The performed numerical analysis indicated that for single-channel transmission, the proposed pulse outperforms the RRC pulse for roll-off factors greater than 0.3, and that reach increase of more than 9% and 26% can be obtained for roll-off factors of 0.5 and 1, respectively. Despite the fact that the proposed pulse is more sensitive to the Mux/Demux response, the presented numerical analysis showed that for multi-channel transmission, the reach distance could be increased by 20% and 18%, for a roll-off factor of 1 and channel spacing of 75 GHz and 100 GHz, respectively. Overall, it was shown that the improved nonlinear tolerance of the proposed pulse increases with its bandwidth. A metric based on this relationship was shown to be directly proportional to the pulses performance in single-channel transmission.

Due to its rectangular-shaped spectrum and steep transition band, the proposed pulse was shown to be more sensitive to truncation for low roll-off factors and to be slightly more sensitive to quantization effects, but to surpass the RRC pulse performance under timing jitter. Experimental results revealed that performance gains of more than 4 dB could be obtained for 10 Gbaud DP-QPSK and DP-16-QAM systems, demonstrating the improved nonlinearity tolerance of the proposed pulse.

## Acknowledgments

The authors would like to acknowledge the NSERC/Bell Canada Industrial Research Chair Program, and thank Prof. John C. Cartledge for providing most valuable comments and suggestions.

## References and links

**1. **E. Torrengo, S. Makovejs, D. S. Millar, I. Fatadin, R. I. Killey, S. J. Savory, and P. Bayvel, “Influence of pulse shape in 112-Gbit/s WDM PDM-QPSK transmission,” IEEE Photon. Technol. Lett. **22**(23), 1714–1716 (2010). [CrossRef]

**2. **S. Makovejs, E. Torrengo, D. S. Millar, R. I. Killey, S. J. Savory, and P. Bayvel, “Comparison of pulse shapes in a 224 Gbit/s (28 Gbaud) PDM-QAM16 long-haul transmission experiment,” in *Proceedings of OFC 2011*, paper OMR5 (2011).

**3. **C. Behrens, S. Makovejs, R. I. Killey, S. J. Savory, M. Chen, and P. Bayvel, “Pulse-shaping versus digital backpropagation in 224Gbit/s PDM-16QAM transmission,” Opt. Express **19**(14), 12879–12884 (2011). [CrossRef] [PubMed]

**4. **M. Nakazawa, S. Okamoto, T. Omiya, K. Kasai, and M. Yoshida, “256-QAM (64 Gb/s) coherent optical transmission over 160 km with an optical bandwidth of 5.4 GHz,” IEEE Photon. Technol. Lett. **22**(3), 185–187 (2010). [CrossRef]

**5. **X. Zhou, L. Nelson, P. Magill, R. Isaac, B. Zhu, D. W. Peckham, P. Borel, and K. Carlson, “8x450-Gb/s, 50-GHz-spaced, PDM-32QAM transmission over 400km and one 50GHz-grid ROADM,” in *Proceedings of OFC 2011*, paper PDPB3 (2011).

**6. **R. Schmogrow, M. Winter, M. Meyer, D. Hillerkuss, B. Nebendahl, J. Meyer, M. Dreschmann, M. Huebner, J. Becker, C. Koos, W. Freude, and J. Leuthold, “Real-time Nyquist pulse modulation transmitter generating rectangular shaped spectra of 112 Gbit/s 16QAM signals,” in *Proceedings of SPPCom 2011*, paper SPMA5 (2011).

**7. **K. Roberts, A. Borowiec, and C. Laperle, “Technologies for optical systems beyond 100G,” Opt. Fiber Technol. **17**(5), 387–394 (2011). [CrossRef]

**8. **B. Farhang-Boroujeny, “A square-root Nyquist (M) filter design for digital communication systems,” IEEE Trans. Signal Process. **56**(5), 2127–2132 (2008). [CrossRef]

**9. **P. S. Rha and S. Hsu, “Peak-to-average ratio (PAR) reduction by pulse shaping using a new family of generalized raised cosine filters,” in *Proceedings of VTC 2003* (2003).

**10. **B. Farhang-Boroujeny and G. Mathew, “Nyquist filters with robust performance against timing jitter,” IEEE Trans. Signal Process. **46**(12), 3427–3431 (1998). [CrossRef]

**11. **A. Assalini and A. M. Tonello, “Improved Nyquist pulses,” IEEE Commun. Lett. **8**(2), 87–89 (2004). [CrossRef]

**12. **K. Roberts, M. O’Sullivan, K.-T. Wu, H. Sun, A. Awadalla, D. J. Krause, and C. Laperle, “Performance of dual-polarization QPSK for optical transport systems,” J. Lightwave Technol. **27**(16), 3546–3559 (2009). [CrossRef]

**13. **B. Châtelain, C. Laperle, D. Krause, K. Roberts, M. Chagnon, X. Xu, A. Borowiec, F. Gagnon, J. C. Cartledge, and D. V. Plant, “SPM-tolerant pulse shaping for 40- and 100-Gb/s dual-polarization QPSK systems,” IEEE Photon. Technol. Lett. **22**(22), 1641–1643 (2010).

**14. **B. Châtelain, C. Laperle, D. Krause, K. Roberts, M. Chagnon, X. Xu, A. Borowiec, F. Gagnon, J. C. Cartledge, and D. V. Plant, “Optimized pulse shaping for intra-channel nonlinearities mitigation in a 10 Gbaud dual-polarization 16-QAM system,” in *Proceedings of OFC 2011*, paper OWO5 (2011).

**15. **H. Nyquist, “Certain topics in telegraph transmission theory,” AIEE Trans. **47**(2), 617–644 (1928).

**16. **J. G. Proakis, *Digital Communications*, 4th ed. (McGraw Hill, New-York, 2001).

**17. **F. Zhang, “XPM Statistics in 100% Precompensated WDM Transmission for OOK and DPSK Formats,” IEEE Photon. Technol. Lett. **21**(22), 1707–1709 (2009). [CrossRef]

**18. **R. J. Essiambre and P. J. Winzer, “Impact of fiber nonlinearities on advanced modulation formats using electronic pre-distortion,” in *Proceedings of OFC 2006*, paper OWB1 (2006).

**19. **C. Behrens, R. I. Killey, S. J. Savory, M. Chen, and P. Bayvel, “Nonlinear Distortion in Transmission of Higher Order Modulation Formats,” IEEE Photon. Technol. Lett. **22**(15), 1111–1113 (2010). [CrossRef]

**20. **Y. Jiang, X. Tang, J. C. Cartledge, and K. Roberts, “Electronic Pre-Compensation of Narrow Optical Filtering for OOK, DPSK and DQPSK Modulation Formats,” J. Lightwave Technol. **27**(16), 3689–3698 (2009). [CrossRef]

**21. **M. J. D. Powell, “A fast algorithm for nonlinearly constrained optimization calculations,” Lect. Notes Math. **630**, 144–157 (1978). [CrossRef]

**22. **F. M. Gardner, “A BPSK/QPSK timing-error detector for sampled receivers,” IEEE Trans. Commun. **34**(5), 423–429 (1986). [CrossRef]

**23. **A. N. D’Andrea and M. Luise, “Design and analysis of a jitter-free clock recovery scheme for QAM systems,” IEEE Trans. Commun. **41**(9), 1296–1299 (1993). [CrossRef]

**24. **A. N. D’Andrea, U. Mengali, and R. Reggiannini, “The modified Cramer-Rao bound and its application to synchronization problems,” IEEE Trans. Commun. **42**(234), 1391–1399 (1994). [CrossRef]

**25. **X. Zhou, X. Chen, W. Zhou, Y. Fan, H. Zhu, and Z. Li, “All-digital timing recovery and adaptive equalization for 112 Gbit/s POLMUX-NRZ-DQPSK optical coherent receivers,” J. Opt. Commun. Netw. **2**(11), 984–990 (2010). [CrossRef]

**26. **A. Leven, N. Kaneda, and S. Corteselli, “Real-time implementation of digital signal processing for coherent optical digital communication systems,” IEEE J. Sel. Top. Quantum Eelectron. **16**(5), 1227–1234 (2010). [CrossRef]

**27. **A. J. Viterbi and A. M. Viterbi, “Nonlinear estimation of PSK-modulated carrier phase with applications to burst digital transmission,” IEEE Trans. Inf. Theory **29**(4), 543–551 (1983). [CrossRef]

**28. **D. Godard, “Self-recovering equalization and carrier tracking in two-dimensional data communication systems,” IEEE Trans. Commun. **28**(11), 1867–1875 (1980). [CrossRef]

**29. **K. Onohara, Y. Miyata, T. Sugihara, K. Kubo, H. Yoshida, and T. Mizuochi, “Soft decision FEC for 100G transport systems,” in *Proceedings of OFC 2010*, paper OThL1 (2010).