Abstract

Abstract: Monolithic operating system designs undermine the security of computing systems by allowing single exploits anywhere in the kernel to enjoy full supervisor privileges. The nested kernel operating system architecture addresses this problem by “nesting” a small, isolated kernel within a traditional monolithic kernel. The “nested kernel” interposes on all updates to virtual memory translations to assert protections on physical memory, thus significantly reducing the trusted computing base for memory access control enforcement. We incorporated the nested kernel architecture into FreeBSD on x86-64 hardware by write-protecting MMU translations and de-privileging the untrusted part of the kernel, thereby enabling the entire operating system, trusted and untrusted components alike, to operate at the highest hardware privilege level. Our implementation inherently enforces kernel code integrity while still allowing dynamically loaded kernel modules, thus defending against code injection attacks. We also demonstrate, by introducing write-mediation and write-logging services, that the nested kernel architecture allows kernel developers to isolate memory in ways not possible in monolithic kernels. Performance of the nested kernel prototype shows modest overheads: < 1% average for Apache, and 2.7% for kernel compile. Overall, our results and experience show that the nested kernel design can be retrofitted to existing monolithic kernels, providing important security benefits.

Bio: Nathan Dautenhahn is a sixth-year doctoral candidate in the Department of Computer Science at the University of Illinois at Urbana-Champaign. His research investigates trustworthy system design by developing experimental operating systems, compilers, and hardware components. This research has led to publications in key systems and security venues, including IEEE S&P, CCS , NDSS, ASPLOS , and ISCA . His latest work, on the nested kernel architecture, is identifying solutions for defending against insecure and malicious operating systems; this is the topic of his thesis. The nested kernel architecture is also under consideration for inclusion in HardenedBSD, an operating system variant of FreeBSD. Dautenhahn also actively contributes to the CS department graduate program by participating in many activities, such as establishing the Doctoral Education Perspectives seminar, formally mentoring undergraduate and graduate students, and serving on the Computer Science Graduate Academic Council and the Engineering Graduate Student Advisory Committee.