Constellation Shaping in Optical Communication Systems

Abstract: Exploiting the full-dimensional capacity of coherent optical communication systems is needed to overcome the increasing bandwidth demands of the future Internet. To achieve capacity, both coding and shaping gains are required, and they are, in principle, independent. Therefore it makes sense to study shaping and how it can be achieved in various dimensions and how various shaping schemes affect the whole performance in real systems. This thesis investigates the performance of constellation shaping methods including geometric shaping (GS) and probabilistic shaping (PS) in coherent fiber-optic systems. To study GS, instead of considering machine learning approaches or optimization of irregular constellations in two dimensions, we have explored multidimensional lattice-based constellations. These constellations provide a regular structure with a fast and low-complexity encoding and decoding. In simulations, we show the possibility of transmitting and detecting constellation with a size of more than 10^{28} points which can be done without a look-up table to store the constellation points. Moreover, improved performance in terms of bit error rate, symbol error rate, and transmission reach are demonstrated over the linear additive white Gaussian noise as well as the nonlinear fiber channel compared to QAM formats. Furthermore, we investigate the performance of PS in two separate scenarios, i.e., transmitter impairments and transmission over hybrid systems with on-off keying channels. In both cases, we find that while PS-QAM outperforms the uniform QAM in the linear regime, uniform QAM can achieve better performance at the optimum power in the presence of transmitter or channel nonlinearities.