Life of a Security Middlebox Challenges with Emerging Protocols and Technologies

Abstract: The Internet of today has intermediary devices known as middleboxes that perform more functions than the normal packet forwarding function of a router. Security middleboxes are a subset of these middleboxes and face an increasingly difficult task to perform their functions correctly. These middleboxes make many assumptions about the traffic that may not hold true any longer with the advent of new protocols such as MPTCP and technologies like end-to-end encryption.The work in this thesis focuses on security middleboxes and the challenges they face. We develop methods and solutions to help these security middleboxes continue to function correctly. In particular, we investigate the case of using MPTCP over traditional security infrastructure as well as the case of end-to-end encryption. We study how practical it is to evade a security middlebox by fragmenting and sending traffic across multiple paths using MPTCP. We then go on to propose possible solutions to detect such attacks and implement them. The potential MPTCP scenario where security middleboxes only have access to part of the traffic is also investigated and addressed. Moreover, the thesis contributes a machine learning based approach to help security middleboxes detect malware in encrypted traffic without decryption.