Towards the Limits of Nonlinearity Compensation for Fiber-Optic Channels

Abstract: The performance of long-haul coherent optical systems is fundamentally limited by fiber nonlinearity and its interplay with chromatic dispersion and noise. Due to nonlinearity, the signal propagating through the fiber interacts with itself and with the noise generated from the inline amplifiers. This process results in nonlinear inter-symbol interference (NISI) and nonlinear signal–noise interaction (NSNI). The state-of-the-art algorithm for combating these impairments is digital backpropagation (DBP) and is typically used as a benchmark against other detectors. However, DBP compensates only for NISI, while studies have revealed that NSNI limits the capacity of the coherent optical communications. The goal of the thesis is to use a methodical approach to develop a near-optimal nonlinearity compensation algorithm that also accounts for NSNI. This allows us to identify the fundamental performance limits of the fiber-optic channel. Starting from the maximum a posteriori principle, we develop an algorithm called stochastic digital backpropagation (SDBP) using the framework of factor graphs. In contrast to DBP, SDBP accounts not only for NISI but also for NSNI. To account for the effects of pulse shaping, we propose three variants of SDBP in this thesis. In the first variant, the output of SDBP is processed using a matched filter (MF) followed by sampling, and decisions are taken on a symbol-by-symbol (SBS) basis. In terms of symbol error rate (SER), SBS-SDBP has better performance than DBP. However, residual memory remains after performing the MF as the MF operation need not be the optimal processing for the fiber-optic channel. This is accounted for in the second variant of SDBP, where the Viterbi algorithm is used after the MF to compensate for the residual memory. The SER of this variant is further improved compared to SBS-SDBP. In the third variant of SDBP, we use Gaussian message passing to account for the effect of pulse shaping, instead of using the MF. The SER of this third variant of SDBP is better than SBS-SDBP. For estimating the achievable throughput in a typical transmission system, mutual information is a better metric than error rate for soft-decision coded optical systems. We show that SDBP can be used as a tool to compute lower bounds on the mutual information, which are tighter than those obtained using DBP.