## Abstract

In this paper, we examine the performance of several modulation formats in more than four dimensions for coherent optical communications systems. We compare two high-dimensional modulation design methodologies based on spherical cutting of lattices and block coding of a ‘base constellation’ of binary phase shift keying (BPSK) on each dimension. The performances of modulation formats generated with these methodologies is analyzed in the asymptotic signal-to-noise ratio regime and for an additive white Gaussian noise (AWGN) channel. We then study the application of both types of high-dimensional modulation formats to standard single-mode fiber (SSMF) transmission systems. For modulation with spectral efficiencies comparable to dual-polarization (DP-) BPSK, polarization-switched quaternary phase shift keying (PS-QPSK) and DP-QPSK, we demonstrate SNR gains of up to 3 dB, 0.9 dB and 1 dB respectively, at a BER of 10^{−3}.

© 2014 Optical Society of America

## 1. Introduction

The introduction of phase- and polarization-diverse digital coherent receivers in recent years has led to gains in capacity both from the mitigation of previously limiting transmission impairments such as chromatic dispersion (CD) and polarization mode dispersion (PMD) [1], and the use of all four optical carrier dimensions for modulation. This has led to the standardization of coherent systems that detect all four dimensions of the received optical field: the in-phase and quadrature-phase components of two orthogonal polarizations [2]. More recently, efforts have been made to improve the performance of coherent optical systems by considering joint modulation of all four dimensions of the carrier. Although utilization of this concept in optical communications may be traced back to the first generation of coherent optical communications research in the 1980s [3], the concept was more recently re-introduced by Karlsson and Agrell [4, 5] with polarization-switched quadrature phase shift keying (PS-QPSK) [6–8], set-partitioned quadrature amplitude modulation (SP-QAM) [9–11] and 6-polarization QPSK (6Pol-QPSK) [12] attracting particular interest from the research community. Research has also been conducted into the effects of expanding the number of modulation dimensions beyond 4, either by using multiple wavelengths [13,14], or time-slots [15–21] to map symbols in a modulation space of more than 4-D onto a 4-D carrier. Modulation techniques for many dimensions have also been investigated for carriers with more than four dimensions, particularly for systems which utilize spatial division multiplexing [20, 22, 23]. Bit-labeling across several consecutive 4-D symbols for constellations that do not have a cardinality (i.e. number of possible symbols) of a power of two has also been a topic of investigation [24].

As the in-phase and quadrature-phase components of passband signals are the two dimensions for which joint processing is the simplest, 2-D discrete constellations have been of strong interest for coherent optical communications. The most recent deployed long-haul systems use 2-D dual-polarization (DP) quadrature-phase shift keying (QPSK) modulation (which can essentially be considered as 1-D modulation because of the independent modulation of in-phase and quadrature-phase components when Gray coding is used) and experiments demonstrate modulation with very dense 2-D constellations [25]. In principle, spectral efficiency of a 2-D modulation scheme can be increased arbitrarily by employing denser constellations. However, besides the limitations of noise regimes including amplified spontaneous emission (ASE) noise, RF amplifier noise, laser phase noise and nonlinear distortion, there are also several other technological bottlenecks including ADC resolution, DAC resolution and DSP complexity, which limit the practicality of dense constellations in 2-D [26].

In this paper, we will focus on cases where the number of constellation dimensions is greater than the number of parallel physical dimensions in the carrier. While modulation using multiple time-slots has been considered in other contexts such as elastic networks with hybrid modulation schemes [27] or rate adaptive transmission [28], here we will consider optimization of the high-dimensional constellations to improve performance. It is important to note that modulation techniques such as time-domain hybrid QAM are not high-dimensional as each dimension (or time-slot) may be detected independently, and as such may be considered as time division multiplexing. The modulation techniques described in this paper as *N*-dimensional require all *N* components of the signal field to be jointly considered when performing symbol estimation. For highly dispersive channels, we may consider the combined effects of ASE and fiber nonlinearity as AWGN [29]. Therefore, we will simplify our modulation design procedure by simply optimizing performance for an AWGN channel. A basic schematic of the type of system considered is shown in Fig. 1. Firstly, our serial data is broken up into *p*-bit words to be modulated. These *p*-bit words are then encoded with a block encoder to form *N*-bit codewords, which are then mapped to an *N*-dimensional field. The field is then partially serialized to form a 4-D signal where a single high-dimensional symbol is transmitted over several time-slots. Modulation, transmission and coherent detection of these 4-D signals are carried out in the conventional manner. After coherent detection and equalization (not shown), the 4-D signal is then deserialized into an *N*-dimensional field. This field is then used for minimum Euclidean distance symbol detection (this may be considered as joint demapping and decoding), resulting in a *p*-bit hard decision. This *p*-bit hard decision is then serialized, and sent to the sink.

It should be noted that this schematic is for the case described below as “linear binary block coded” high-dimensional modulation. While “sphere-cut” high-dimensional modulation [30] may theoretically take the same structure (i.e. separate high-dimensional block encoder and mapper), we considered joint encoding and mapping for lattice-cut modulation. We also note that although in this paper and our previous work on high-dimensional modulation [16–20] we have focussed on modulation with numbers of dimensions which are multiples of four, any number of dimensions may be used. This may be achieved by simultaneously generating four high-dimensional symbols, and serializing them independently onto each of the four carrier quadratures.

The remainder of the paper is organized as follows. In Section 2, we explain the concept of high-dimensional modulation and describe two design methodologies: one based on spherical cutting of optimal lattices, and one based on the use of linear block codes over a base constellation. The performance of several high-dimensional formats generated using the previously described methods is analyzed in Section 3, in terms of asymptotic performance and AWGN channel performance. Section 4 provides simulated results of high-dimensional modulation transmissions over an SMF link. Conclusions are presented in Section 5.

## 2. High-dimensional modulation formats

Let *p* denote the number of bits per high-dimensional symbol, and *N* denote the number of modulation dimensions. A high-dimensional modulation format maps each *p*-bit symbol to one of *M* = 2* ^{p}* constellation points in the

*N*dimensional signal space (see Fig. 1). High-dimensional modulation designs must therefore optimize (i) locations of

*M*constellation points and (ii) corresponding

*p*-bit labelings for each of the constellation points. Throughout the paper, we use the notation ‘

*p*b-

*N*D’ for modulation formats of constellations generated by sphere-cutting of lattices, where

*p*is the number of bits per symbol and

*N*is the dimensionality of the constellation space. For block-coded modulation, we use the notation [

*N*,

*p*,

*h*] - consistent with the coding literature - where

*h*is the minimum Hamming distance of the code. Although this is not consistent with the notation used in our previous work [16–20], the large variety of codes used in this work necessitates this more generalized notation. In this section, we describe the design of such high-dimensional modulation formats using two approaches: spherical cutting of optimal lattices, and linear block codes over a base constellation of BPSK.

#### 2.1. Sphere cutting of densest lattices

This design methodology may be considered as having a geometric basis. By considering constellations as collections of *N*-dimensional hyperspheres which are optimized by finding the densest possible non-overlapping arrangement in *N*-dimensional space [5], we may view lattices as constellations with infinitely many points. Optimal solutions to this “sphere packing problem” are known for certain numbers of dimensions [31], but this remains an active topic for research. A subset of points (the constellation) is then taken by cutting an *N* dimensional hypersphere of points from the lattice – optimization of this cutting procedure is described in detail in [30]. We then have a symbol alphabet which is optimized in terms of minimum separation between constellation points, but has no bit labeling.

Generation of sphere-cut high-dimensional modulated signals uses the following algorithm:

- An
*M*= 2element look-up-table (LUT) is used to determine the^{p}*N*-dimensional signal vector**s**= (*s*_{1},...,*s*)_{N} - The symbol is then partially serialized to a 4-D signal occupying
*N*/4 time slots:$$\mathbf{E}=\left(\begin{array}{c}\text{real}\left({E}_{x}\left(1,\dots ,\frac{N}{4}\right)\right)\\ \text{imag}\left({E}_{x}\left(1,\dots ,\frac{N}{4}\right)\right)\\ \text{real}\left({E}_{y}\left(1,\dots ,\frac{N}{4}\right)\right)\\ \text{imag}\left({E}_{y}\left(1,\dots ,\frac{N}{4}\right)\right)\end{array}\right)=\left(\begin{array}{cccc}{s}_{1}& {s}_{5}& \dots & {s}_{N-3}\\ {s}_{2}& {s}_{6}& \dots & {s}_{N-2}\\ {s}_{3}& {s}_{7}& \dots & {s}_{N-1}\\ {s}_{4}& {s}_{8}& \cdots & {s}_{N}\end{array}\right)$$

For *N* = 2, the densest lattice is the hexagonal lattice [32]. The optimized constellation by cutting the 2-D hexagonal lattice reduces to BPSK constellations for *M* = 2 (*p* = 1) and rhombus constellation for *M* = 4 (*p* = 2) which has the same minimum Euclidean distance with the QPSK constellation. For higher *p*, comparing the optimized constellations cut from 2-D hexagonal lattice to the constellations cut from 2-D rectangular lattice, 0.6, 0.76 and 0.81 dB increase in asymptotic power efficiency [5] can be obtained for *M* = 16, 64 and 256 (*p* = 4, 6 and 8) [33]. The asymptotic power efficiency is defined as
$\gamma ={d}_{\text{min}}^{2}/4{E}_{\text{b}}$, where *d*_{min} is the minimum Euclidean distance between constellation points, and *E*_{b} is the energy per bit.

Although optimal lattices are not known for arbitrary numbers of dimensions, optimal or near-optimal solutions are known for many cases. Some examples are the checkerboard lattice in 4-D; the diamond lattice in 8-D; the Barnes-Wall lattice in 16-D and the Leech lattice in 24-D [32]. Among these, the 8-D and 24-D lattices are particularly important, as they are close to the upper bound on the packing density of a lattice (the Rogers’ bound) [30].

As a representative example, the sphere cutting 8-D constellation with 0.5 bit/symbol/dimension spectral efficiency, i.e. 4b-8D constellation, is shown in Fig. 2. In addition to the full constellation, we have highlighted the constellation points which have maximum and minimum Euclidean distance relative to a reference constellation point. Note that a unique constellation point requires a marker in each of the four constellations, and not every combination of points marked as the full constellation is possible. We also note that there are at most 4 possible levels in each of the 8 dimensions for this case, and for many constellations — even on higher dimensional lattices — the number of signal levels remains typically much lower than *M* because of the regular lattice structure. An interesting feature of this constellation is that the maximum and minimum distance points both have the same distance from the reference point. We may consider the reference point as a high-dimensional sphere, with additional high-dimensional spheres (representing the other constellation points) of the same size touching it. As long as the total number of neighboring constellation points is lower than the kissing number (240 for the *E*_{8} lattice), each neighboring sphere can be touching the reference sphere. The centers of all neighboring spheres may therefore be the same distance from the center of the reference sphere.

When designing high-dimensional sphere-cut lattice modulation formats, a significant challenge is the bit-labeling of the constellation points. For even moderate spectral efficiencies with many dimensions, optimal labeling can be prohibitively complex, as it requires a search among (*M* −1)! possibilities. As will be discussed in Section 3, the increased number of near neighbors for high dimensional constellations means that although sphere-cut high-dimensional formats can maximize the minimum Euclidean distance between constellation points and provide significant improvements in symbol error ratio (SER), the gain for uncoded bit error ratio (BER) may not be so significant in the low SNR regimes. In this paper, we have employed a random search algorithm for bit-labeling. For a fixed SNR, we tested the BER of many randomly selected labelings. The best labeling was selected when either the improvement expected from testing additional labelings approached zero, or the maximum number of test labelings was reached. Although this approach does not determine the optimal labeling, it was considered to be computationally tractable, unlike an exhaustive search of all possible labelings.

The extrinsic information transfer characteristics of some sphere-cut high-dimensional modulation formats are studied in [30] and it is shown that dimensionality increase can yield to higher correlation between *a-priori* and extrinsic mutual information and also higher total extrinsic mutual information. Iterative decoding of high-dimensional formats may therefore provide additional benefits in performance not revealed by the investigation of uncoded performance presented in this paper. We also note that this technique may be prohibitively computationally intensive for very large numbers of dimensions.

#### 2.2. Linear binary block codes

An alternative approach to high-dimensional constellation designs is based on inner block codes, and independent modulation of each component of the resulting codeword [34, 35]. Serial concatenation of two or more codes is a well developed concept in digital communications, and often takes the form of one or more hard decision outer codes to eliminate the error floor present in an inner soft decision code [36]. Here, we consider the use of short linear binary block codes for high-dimensional modulation design: effectively using these codes as inner codes with soft input soft output (SISO) decoding which improve the input BER to an outer FEC code.

This method for designing modulation formats was motivated by the congruence of the mathematics of codes and of lattices. The design of the code aims to separate the codewords as much as possible in the Hamming space. Codes were chosen with short length (to enable minimum Euclidean distance SISO decoding), and with the best possible minimum Hamming distance for the code rate and word length. Generator matrices of the codes used in this manuscript may be found in many places in the coding theory literature, [37] being particularly useful.

When each bit of the inner codeword is mapped to a BPSK “base constellation” on each modulation dimension, the minimum squared Euclidean distance between the constellation points is proportional to the minimum Hamming distance of the block code. Therefore, the use of a binary block code with block length *N*, message length *p* and minimum Hamming distance between codewords *h*, with BSPK mapping for each of the *N* binary outputs, yields *M* = 2* ^{p}* constellation points in

*N*dimensional space and corresponding binary labelings. For example, for

*N*= 4, DP-QPSK modulation can be considered as two independent uncoded modulation dimensions with

*h*= 1. Using an inner [4,3,2] single-parity check code results in PS-QPSK (also equivalent to 3b-4D sphere-cut modulation) [4]. Our procedure for generating high-dimensional modulation for transmission over a 4-D channel may therefore be described by the following steps (assuming that

*N*is a multiple of 4 while it is not necessary condition):

- The symbol is then partially serialized to a 4-D signal occupying
*N*/4 time slots:$$\mathbf{E}=\left(\begin{array}{c}\text{real}\left({E}_{x}\left(1,\dots ,\frac{N}{4}\right)\right)\\ \text{imag}\left({E}_{x}\left(1,\dots ,\frac{N}{4}\right)\right)\\ \text{real}\left({E}_{y}\left(1,\dots ,\frac{N}{4}\right)\right)\\ \text{imag}\left({E}_{y}\left(1,\dots ,\frac{N}{4}\right)\right)\end{array}\right)=\left(\begin{array}{cccc}{s}_{1}& {s}_{5}& \dots & {s}_{N-3}\\ {s}_{2}& {s}_{6}& \dots & {s}_{N-2}\\ {s}_{3}& {s}_{7}& \dots & {s}_{N-1}\\ {s}_{4}& {s}_{8}& \cdots & {s}_{N}\end{array}\right)$$

Throughout the paper we use the notation [*N*,*p*, *h*] with the corresponding block code name for block-coded modulation formats with block length *N*, message length *p*, and the minimum Hamming distance *h*. As an illustration, the extended Hamming [8,4,4] code constellation is shown in Fig. 3, which has the same spectral efficiency and minimum Euclidean distance with the 4b-8D constellation in Fig. 2. Again, we have plotted both the full constellation, and the minimum and maximum Euclidean distance points relative to a reference point. For this constellation, we can clearly see that the minimum distance constellation point requires a total of 4 changes of signal level (either I or Q) for the 8 signal dimensions. Similarly, the maximum distance point is achieved when the signal level changes on all 8 dimensions (maximum distance corresponding to a change of signal on all dimensions requires the use of a self-dual code such as the extended Hamming [8,4,4] code).

The codes that belong to the family of perfect codes are particularly of interest for high-dimensional modulation design, as they attain the sphere packing bound [31]. The sphere packing bound (or Hamming bound) is an upper bound on the separation in Hamming space that is possible for a given codeword length and code rate, this may be considered as the Hamming space equivalent of the Rogers’ bound in Euclidean space. There are relatively few known perfect codes [37]: [2* ^{r}* − 1,2

*−*

^{r}*r*− 1,3] Hamming codes and the [23,12,7] binary Golay code are the only two (non-trivial) perfect binary codes. Indeed, the Golay code is the only binary quasi-perfect code with

*h*> 4. The quasi-perfect codes can be obtained by adding a single parity bit to the perfect codes to ensure even parity over the set of all codewords, thus increasing Hamming distance by 1. Extended Hamming [2

*, 2*

^{r}*−*

^{r}*r*− 1,4] codes and the extended Golay [24,12,8] code belong to this family.

Block-coded high-dimensional modulation is limited to spectral efficiency of less than 1 bit/symbol/dimension when used with BPSK mapping per dimension. While it is possible to use block coding with higher-order PAM mapper per dimension, these mappings do not result in a linear relationship between Hamming distance and squared Euclidean distance for all constellation points.

## 3. Asymptotic power efficiency and noise sensitivity

In this section, we analyze and compare the minimum Euclidean distance and AWGN channel characteristics of the designed sphere-cut constellations and the block-coded constellations.

In the very high SNR regimes, the minimum Euclidean distance *d*_{min} between constellation points determines the SER as the nearest neighbor errors become dominant. One commonly used parameter to quantify the asymptotic noise performance is the sensitivity penalty 1/*γ* where *γ* is the asymptotic power efficiency. Note that for 1 bit/symbol/dimension case of DPQPSK, *γ* = 1, hence sensitivity penalty can also be considered as the performance penalty with respect to DP-QPSK for asymptotically high SNR.

In Fig. 4, sensitivity penalty and spectral efficiency of various high-dimensional modulation formats obtained by sphere-cutting and block-coding approaches are shown. It can be observed that optimized sphere-cut constellations with 1 bit/symbol/dimension spectral efficiency yield 0, −0.82, −1.87, −2.80 and −4.25 dB sensitivity penalties for *N* = 2, 4, 6, 8 and 16, respectively. In [5], it is shown that highest power efficiency for *N* = 2 is obtained for *M* = 3 and for *N* = 3 for *M* = 4, both using simplex configurations, i.e. same length vectors with same angular separation. For *N* > 3, simplex configurations are not optimal constellations in terms of power efficiency. For *N* = 4, the highest power efficiency is obtained for *M* = 8, with the 3b-4D PS-QPSK constellation. Among the values we consider for sphere-cut constellation design, for *N* = 6, 8 and 16, we observe the highest power efficiencies for *M* = 2^{4}, 2^{4} and 2^{11} (4b-6D, 4b-8D and 11b-16D modulation formats).

Besides the minimum Euclidean distance *d*_{min}, the total number of nearest neighbors at that distance increases the SER proportionally. For reference, the average number of neighbors within 1.01 · *d*_{min} distance is listed in Table 1. Although the maximum number of spheres which can touch an adjacent sphere (the so-called kissing number problem) is unknown for an arbitrary number of dimensions, upper and lower bounds (including lattice arrangements) indicate that the kissing number increases exponentially with number of dimensions [32]. From this we may infer that the number of nearest neighbors in the high spectral efficiency region increases exponentially for sphere-cut high-dimensional modulation. Of course, for finite constellations, the maximum number of nearest neighbors is at most the number of points in the constellation.

Now, we consider the AWGN performance of high-dimensional modulation formats for the low SNR regime, focusing on typical threshold targets of modern forward error correction codes. SNR is defined as *E*_{b}/*N*_{0} where *N*_{0} is the unilateral power spectral density of the noise per dimension. Monte Carlo simulations were performed for an ideal transmitter and receiver using minimum Euclidean distance decisions, and more than 10^{7} bits for each SNR level.

The performance of high-dimensional sphere-cut modulation formats is shown in Figs. 5 –7(a). As expected, SNR improvements with respect to DP-QPSK for high target BER values are considerably lower than asymptotic gains (Fig. 4). In the extreme case of very low SNR, DPQPSK (equivalent to 1-D BPSK) demonstrates the best performance as the very high number of near neighbors for higher dimensions become dominant. It may be observed from Fig. 5(a) that for 0.5 bit/symbol/dimension spectral efficiency, 4b-8D and 12b-24D sphere-cut modulation provide gains of 1.7 and 2.8 dB, respectively, compared with DP-QPSK at a BER of 10^{−3}. This gain is reduced to 1 dB and 1.5 dB respectively at a BER of 10^{−2}. For the sphere-cut modulation formats presented in Fig. 6(a) with spectral efficiency of 0.687 and 0.75 bit/symbol/dimension (3b-4D – also known as PS-QPSK – and 11b-16D), we note a gain compared with DP-QPSK at a BER of 10^{−3} of 1 dB and 1.9 dB respectively. Increasing BER to 10^{−2} reduces the gain to 0.6 and 0.8 dB respectively. The formats presented in Fig. 7(a) with 0.875 bit/symbol/dimension (7b-8D) and 1 bit/symbol/dimension (6b-6D and 8b-8D) spectral efficiency modulation formats to DP-QPSK, result in gains of 1.1, 0.5 and 0.9 dB respectively when compared with DP-QPSK at a BER of 10^{−3}. These gains are reduced to 0.3, −0.1 and 0.1 dB at a BER of 10^{−2}.

The performance of high-dimensional modulation formats with block coding is shown in Figs. 5 – 7(b). Although asymptotic performance of these formats is typically close to sphere-cut formats for equal spectral efficiency and dimensionality, we note that they generally exhibit better performance at high BERs. We attribute this to the correspondence between the code’s weight distribution and the distribution of squared Euclidean distance between constellation points. Figure 5(b) presents results for [7,4,3], [8,4,4], [23,12,7] and [24,12,8] codes, resulting in spectral efficiency of 0.571, 0.5, 0.523 and 0.5, respectively. Compared with DPQPSK at a BER of 10^{−3}, we observe improvements in sensitivity of 1.5, 1.7, 2.9 and 3 dB. For a BER of 10^{−2}, we note improvements of 1, 1.1, 1.8 and 1.9 dB. Modulation formats presented in Fig. 6(b) are [4,3,2] (identical to PS-QPSK), [15,11,3] and 16,11,4], with spectral efficiency of 0.75, 0.733 and 0.688 bit/symbol/dimension. At a BER of 10^{−3}, we can see improvements in sensitivity of 1, 1.9 and 2.2 dB, respectively. These gains are reduced to 0.6, 1.2 and 1.3 dB at a BER of 10^{−2}. At higher spectral efficiency, we are asymptotically limited to 1 bit/symbol/dimension. Therefore in Fig. 7(b), we examine two single party check coded formats: [8,7,2] and [16,15,2], with respective spectral efficiencies of 0.875 and 0.938 bit/symbol/dimension. These formats offer a gain of 1.1 and 1 dB over DP-QPSK at a BER of 10^{−3}, and a gain of 0.6 and 0.4 dB at a BER of 10^{−2}.

## 4. Transmission over single-mode fiber

In this section, we simulate transmission performance of several high-dimensional modulation formats over an ultra long-haul SMF link with Erbium doped fiber amplification (EDFA). The schematics of the overall transmission system is depicted in Fig. 8.

We consider a 5-channel wavelength-division multiplexed (WDM) system, and examine two transmission cases: (i) per channel data rate 112 Gb/s with 7% FEC overhead and a BER threshold of 10^{−3}; (ii) 125 Gb/s with 20% FEC overhead and a BER threshold of 10^{−2}. For each case, we consider three groups of modulation formats: low, intermediate and high spectral efficiencies, that are listed in Table 2. The wavelength spacings are chosen to be wide enough to account for the different spectral widths of the modulated signals, and are also listed in Table 2. The total transmission link of 6000 km consists of 75 spans of 80 km SMF, with optical loss compensated by EDFA, each with 5 dB of noise figure. To quantify the transmission performance in terms of span loss budget for a target BER, variable optical attenuators are used [6]. Fiber propagation is modeled using adaptive step-size split-step Fourier method with the Manakov model and a nonlinearity coefficient of 1.2 /W/km, chromatic dispersion of 17 ps/nm/km and attenuation of 0.2 dB/km. Dispersion slope and PMD are not simulated.

There are many possible mappings with identical performance for the linear channel. However, we assume that the modulated symbol dimensions are mapped to the available 4 parallel dimensions by parallelizing consecutive high-dimensional serial blocks as shown in Fig. 8. This is possible as all formats considered for transmission have a number of dimensions which is a multiple of 4 (although this is not strictly necessary). The electrical drive waveforms are formed using ideal rectangular waveforms with level of the symbol value in corresponding dimension, filtered by a 5* ^{th}* order Bessel filter with (single sided) −3 dB cut-off frequency of 0.35 times the (double sided) Nyquist bandwidth. Optical to electrical conversion is performed by ideal, linear dual-polarization I/Q modulators for each wavelength.

At the receiver, wavelength demultiplexing is performed with an ideal optical filter, followed by ideal homodyne coherent detection. The complex baseband signals are filtered by a low-pass 5* ^{th}* order Bessel filter with −3 dB cut-off frequency of 0.35 times the Nyquist bandwidth to emulate limited receiver bandwidth, and then sampled at the rate of twice the symbol rate. Digital signal processing (DSP) is then performed on the received signals in the following order prior to demodulation: normalization, chromatic dispersion compensation by frequency-domain equalization, time-domain data-aided equalization by least-mean-square (LMS) algorithm. Demodulation is then performed by selecting the minimum Euclidean distance between the received signal and each possible transmitted symbol in the signal alphabet. The transmitted bits were then compared to the received bits to calculate the BER.

Figure 9 shows the span loss budget for the modulation formats with low spectral efficiencies. In the linear propagation regime, where signal launch power is low enough so that nonlinearity is negligible and equalized SMF channel resembles an AWGN channel, relative performances of the modulation formats are very similar to the relative performances corresponding to BER of 10^{−2} and 10^{−3} in Fig. 5 as expected. As the launch power increases and nonlinearity becomes significant, relative performances of the modulation formats alter differently. For both cases of 7% FEC and 20% FEC, block-coded modulation formats outperform sphere-cut modulation formats in the highly nonlinear regime, which we attribute to the increased peak-to-average power ratio in sphere-cut constellations. For BER of 10^{−3}, extended Golay [24,12,8] coded modulation yields span loss budget improvement around 3.2 dB gain with respect to DP-BPSK in the linear propagation regime, whereas the difference between the peak span loss budget values is around 4.8 dB. Although 12b-24D sphere-cut modulation demonstrates similar performance to extended Golay [24,12,8] coded modulation in the linear propagation regime, the peak span loss budget difference is around 1.2 dB between these two 24-D modulation formats for BER of 10^{−3}. The 8-D modulation formats have more similar performances. In the linear propagation regime, 4b-8D sphere-cut modulation yields 1.6 dB improvement and extended Hamming [8,4,4] coded modulation yields 1.8 dB improvement with respect to DP-BPSK. Between the peak points, this improvement increases to 2.7 dB and 2.8 dB for 4b-8D sphere-cut modulation and extended Hamming [8,4,4] coded modulation. The SNR improvements are lower for BER of 10^{−2} in all cases. The difference between peak points with respect to DPBPSK ares 1.5 dB for 4b-8D sphere-cut modulation, 1.7 dB for extended Hamming [8,4,4] coded modulation, 1.7 dB for 12b-24D sphere-cut modulation and 2.7 dB for extended Golay [24,12,8] coded modulation.

Figure 10 shows the span loss budget for the modulation formats with intermediate spectral efficiencies. In the linear propagation regime, relative gain of the modulation formats are again very similar to the relative performances corresponding to BER of 10^{−2} and 10^{−3} in Fig. 6 for the AWGN channel. Moreover, robustness against nonlinearity is again considerably higher for the block-coded modulation format comparing to the sphere-cut modulation format. In the linear propagation regime, span loss budget improvement with respect to PS-QPSK is 0.9 dB for 11b-16D sphere-cut modulation and 1.2 dB for extended Hamming [16,11,4] coded modulation for BER of 10^{−3}. The improvement between the peak points become 1 dB for 11b-16D sphere-cut modulation and 2.2 dB for extended Hamming [16,11,4] coded modulation for the same BER of 10^{−3}. For BER of 10^{−2}, performances of 11b–16D sphere-cut modulation and PSQPSK become very similar, and extended Hamming [16,11,4] coded modulation results in span loss budget improvement up to 1.5 dB with respect to other two.

Figure 11 shows the span loss budget for the modulation formats with high spectral efficiencies. For BER of 10^{−3}, 8b-8D sphere-cut modulation provides 0.9 dB improvement with respect to DP-QPSK in the linear region, while performance degrades above the optimum launch power. The performance deterioration of sphere-cut modulation formats in highly nonlinear regime can also be clearly observed for 8b-8D modulation format. For BER of 10^{−2}, as expected from the additive white Gaussian channel characteristics, span loss budgets of 8b-8D sphere-cut modulation and DP-QPSK are almost same in the linear propagation regime and 8b-8D modulation underperforms when nonlinearity becomes significant.

## 5. Conclusions

We have proposed and analyzed the performance of several high-dimensional modulation formats for coherent optical fiber communications. We introduced two methodologies to design such high-dimensional modulation: spherical cutting of optimal lattices, and using block codes with a ‘base constellation’ of BPSK on each dimension. Performance of these modulation formats was simulated for both an AWGN channel, and a long-haul link of SSMF with EDFA.

For spectral efficiency of 0.5 bit/symbol/dimension - equivalent to DP-BPSK, the best performing modulation format in the AWGN channel was block coded modulation using the [24,12,8] extended Golay code, as previously reported in [16, 18]. This format provides a gain over DP-BPSK of 3 dB at a BER of 10^{−3}, and 1.9 dB for a BER of 10^{−2}. At a spectral efficiency of 0.73 bit/symbol/dimension, the block coded [15,11,3] format provides gains over PS-QPSK of 0.9 dB and 0.6 dB at BERs of 10^{−3} and 10^{−2} respectively, with a reduction in spectral efficiency of just 2.2%. Using a [16,15,2] single parity check code, gains over DP-QPSK of 1 dB and 0.4 dB were demonstrated at BERs of 10^{−3} and 10^{−2} respectively.

We have consistently observed that sphere-cut modulation formats have impaired nonlinear performance compared with constant modulus modulation formats generated with block coding. This indicates that in general, gains in noise tolerance may be offset by poorer nonlinear performance for novel formats, and simulation of nonlinear performance is essential. Also, for all modulation formats considered in this paper, below a certain threshold SNR, BPSK/QPSK has the best performance. This threshold SNR increases with spectral efficiency, and seems to be consistently higher when comparing sphere-cut with block-coded constellations. While the performance of many of the high-dimensional modulation formats examined here is superior to conventional formats at BERs in the region of 10^{−3} ∼ 10^{−2}, overall performance comparisons for the joint use of such high-dimensional modulation and strong modern FEC codes remains a topic for further investigation.

## Acknowledgements

The authors would like to thank Kazuyuki Ishida and Takashi Mizuochi for useful discussions in developing this paper.

## References and links

**1. **S. J. Savory, “Digital filters for coherent optical receivers,” Opt. Express **16**, 804–817 (2008). [CrossRef] [PubMed]

**2. **“Implementation Agreement for Integrated Dual Polarization Intradyne Coherent Receivers,” Optical Internetworking Forum (2010). http://www.oiforum.com/public/documents/OIF_DPC_RX-01.0.pdf.

**3. **S. Betti, F. Curti, G. De Marchis, and E. Iannone, “Exploiting fibre optics transmission capacity: 4-quadrature multilevel signalling,” Electron. Lett. **26**, 992–993 (1990). [CrossRef]

**4. **M. Karlsson and E. Agrell, “Which is the most power-efficient modulation format in optical links?” Opt. Express **17**, 10814–10819 (2009). [CrossRef] [PubMed]

**5. **E. Agrell and M. Karlsson, “Power-efficient modulation formats in coherent transmission systems,” J. Lightwave Technol. **27**, 5115–5126 (2009). [CrossRef]

**6. **P. Poggiolini, G. Bosco, A. Carena, V. Curri, and F. Forghieri, “Performance evaluation of coherent WDM PSQPSK (HEXA) accounting for non-linear fiber propagation effects,” Opt. Express **18**, 11360–11371 (2010). [CrossRef] [PubMed]

**7. **P. Serena, A. Vannucci, and A. Bononi, “The performance of polarization switched-QPSK (PS-QPSK) in dispersion managed WDM transmissions,” Proc. ECOC, Th.10.E.2, (2010).

**8. **D. S. Millar, D. Lavery, S. Makovejs, C. Behrens, B. C. Thomsen, P. Bayvel, and S. J. Savory, “Generation and long-haul transmission of polarization-switched QPSK at 42.9 Gb/s,” Opt. Express **19**, 9296–9302 (2011). [CrossRef] [PubMed]

**9. **J. Renaudier, A. Voicila, O. Bertran-Pardo, O. Rival, M. Karlsson, G. Charlet, and S. Bigo, “Comparison of Set-Partitioned Two-Polarization 16QAM Formats with PDM-QPSK and PDM-8QAM for Optical Transmission Systems with Error-Correction Coding,” Proc. ECOCWe.1.C.5, (2010).

**10. **L. D. Coelho and N. Hanik, “Global Optimization of Fiber-Optic Communication Systems using Four-Dimensional Modulation Formats,” Proc. ECOCMo.2.B.4, (2011).

**11. **M. Sjödin, P. Johannisson, J. Li, E. Agrell, P. A. Andrekson, and M. Karlsson, “Comparison of 128-SP-QAM with PM-16-QAM,” Opt. Express20, 8356–8366 (2012). [CrossRef] [PubMed]

**12. **H. Bülow, “Polarization QAM modulation (POL-QAM) for coherent detection schemes,” Proc. OFC/NFOECOWG.2, (2009).

**13. **T. A. Eriksson, P. Johannisson, M. Sjödin, E. Agrell, P. A. Andrekson, and M. Karlsson, “Frequency and Polarization Switched QPSK,” Proc. ECOCTh.2.D.4, (2013).

**14. **S. Ishimura and K. Kikuchi, “Multi-dimensional Permutation Modulation Aiming at Both High Spectral Efficiency and High Power Efficiency,” Proc. OFC/NFOECM3A.2, (2014).

**15. **D. G. Foursa, H. G. Batshon, H. Zhang, M. Mazurczyk, J.-X. Cai, O. Sinkin, A. Pilipetskii, G. Mohs, and N. S. Bergano, “44.1 Tb/s transmission over 9,100 km using coded modulation based on 16QAM signals at 4.9 bits/s/Hz spectral efficiency,” Proc. ECOCPD3.E.1, (2013).

**16. **D. S. Millar, T. Koike-Akino, K. Kojima, and K. Parsons, “A 24-Dimensional Modulation Format Achieving 6 dB Asymptotic Power Efficiency,” Proc. SPPCOMSPM3D.6, (2013).

**17. **T. Koike-Akino, D. S. Millar, K. Kojima, and K. Parsons, “Eight-Dimensional Modulation for Coherent Optical Communications,” Proc. ECOCTu.3.C.3, (2013).

**18. **D. S. Millar, T. Koike-Akino, R. Maher, D. Lavery, M. Paskov, K. Kojima, K. Parsons, B. C. Thomsen, S. J. Savory, and P. Bayvel, “Experimental Demonstration of 24-Dimensional Extended Golay Coded Modulation with LDPC,” Proc. OFC/NFOECM3A.5, (2014).

**19. **D. S. Millar, T. Koike-Akino, S. Ö. Arık, K. Kojima, and K. Parsons, “Comparison of Quaternary Block-Coding and Sphere-Cutting for High-Dimensional Modulation,” Proc. OFC/NFOECM3A.4, (2014).

**20. **S. Ö. Arık, D. S. Millar, T. Koike-Akino, K. Kojima, and K. Parsons, “High-Dimensional Modulation for Mode-Division Multiplexing,” Proc. OFC/NFOECW4J.1, (2014).

**21. **T. A. Eriksson, P. Johannisson, E. Agrell, P. A. Andekson, and M. Karlsson, “Biorthogonal Modulation in 8 Dimensions Experimentally Implemented as 2PPM-PS-QPSK,” Proc. OFC/NFOECW1A.5, (2014).

**22. **J. A. Anguita, J. Herreros, and I. B. Djordjevic, “Coherent Multimode OAM Superpositions for Multidimensional Modulation,” IEEE Photonics J. **6**, 1–11 (2014). [CrossRef]

**23. **I. B. Djordjevic, M. Cvijetic, and C. Lin, “Multidimensional Signaling and Coding Enabling Multi-Tb/s Optical Transport and Networking: Multidimensional aspects of coded modulation,” IEEE Sig. Proc. Mag. **31**22, 104–117 (2014). [CrossRef]

**24. **H. Bülow, T. Rahman, F. Buchali, W. Idler, and W. Kuebart, “Transmission of 4-D Modulation Formats at 28-Gbaud,” Proc. OFC/NFOECJW2A.39, (2013).

**25. **M. Nakazawa, T. Hirooka, M. Yoshida, and K. Kasai, “Extremely Higher-Order Modulation Formats,” in *Optical Fiber Telecommunications VIB*, I. Kaminow, T. Li, and A. E. Willner, eds. (Elsevier, 2013), pp. 297–336. [CrossRef]

**26. **P. J. Winzer, “High-Spectral-Efficiency Optical Modulation Formats,” J. Lightwave Technol. **30**, 3824–3835 (2012). [CrossRef]

**27. **Q. Zhuge, X. Xu, M. Morsy-Osman, M. Chagnon, M. Qiu, and D. Plant, “Time Domain Hybrid QAM Based Rate-Adaptive Optical Transmissions Using High Speed DACs,” Proc. OFC/NFOECOTh4E.6, (2013).

**28. **G.-H. Gho and J. M. Kahn, “Rate-Adaptive Modulation and Coding for Optical Fiber Transmission Systems,” J. Lightwave Technol. **30**, 1812–1818 (2012). [CrossRef]

**29. **P. Poggiolini, “The GN Model of Non-Linear Propagation in Uncompensated Coherent Optical Systems,” J. Lightwave Technol. **30**, 3857–3879 (2012). [CrossRef]

**30. **T. Koike-Akino and V. Tarokh, “Sphere packing optimization and EXIT chart analysis for multi-dimensional QAM signaling,” Proc. IEEE ICC (2009).

**31. **N. J. A. Sloane, R. H. Hardin, T. D. S. Duff, and J. H. Conway, “Minimal-energy clusters of hard spheres,” Discrete & Computational Geometry **14**, 237–259 (1995). [CrossRef]

**32. **J. H. Conway and N. J. A. Sloane, *Sphere Packings, Lattices and Groups*, (Springer, 1998).

**33. **C. D. Murphy, “High-order optimum hexagonal constellations,” IEEE PIMRC **1**, 143–146 (2000).

**34. **G. D. Forney Jr, R. G. Gallager, G. Lang, F. M. Longstaff, and S. U. Qureshi, “Efficient modulation for band-limited channels,” IEEE J. Sel. Areas Commun. **2**, 632–647 (1984). [CrossRef]

**35. **G. Forney Jr, G. David, and L.-F. Wei., “Multidimensional constellations. I. Introduction, figures of merit, and generalized cross constellations,” IEEE J. Sel. Areas Commun. **7**, 877–892 (1989). [CrossRef]

**36. **K. Onohara, T. Sugihara, Y. Konishi, Y. Miyata, T. Inoue, S. Kametani, K. Sugihara, K. Kubo, H. Yoshida, and T. Mizuochi, “Soft-Decision-Based Forward Error Correction for 100 Gb/s Transport Systems,” IEEE J. Sel. Top. Quantum Electron. **16**, 1258–1267 (2010). [CrossRef]

**37. **J. H. van Lint, “A survey of perfect codes,” Rocky Mountain Journal of Mathematics **5**, 199–224 (1975). [CrossRef]