Observations on Operating System Security Vulnerabilities

Abstract: This thesis presents research on computer security vulnerabilities in general-purpose operating systems. The objective is to investigate intrusions in such systems in order to find and model the underlying generic weaknesses, i.e., weaknesses that would be applicable to many different systems. An attempt is made to create a conceptual basis for the generic modeling of vulnerabilities, addressing security concepts, definitions, and terminology. The investigation of intrusions is based on empirical data collected from three different systems, UNIX, Novell NetWare, and Windows NT. The UNIX and Novell NetWare data were generated from a number of practical intrusion experiments with Masters students, while the Windows NT data resulted from a security analysis that we performed ourselves. This analysis showed that Windows NT, initially thought to be quite secure, still displayed a significant number of vulnerabilities. A comparison with earlier UNIX analyses indicates that the security differences between the systems are related more to factors such as time on market and security-by-obscurity than to inherent security performance. A different approach was taken with the Novell NetWare system. Here, the initial experiments showed quite poor security behavior and we have investigated the security evolution as later versions of the system have been released and have reached the conclusion that, even though some effort has been made to improve security, no significant difference has been achieved, primarily due to backward compatibility requirements. In summary, the work performed here represents a further step towards a full understanding of the generic weaknesses and vulnerabilities that impair commercially available operating systems. This understanding is essential to our aspiration to make these systems secure, or at least sufficiently secure. It is expected that this work, when fully accomplished, will comprise a powerful basis for improving the security of operating systems, at least to the extent that research results are taken into consideration by software developers and manufacturers