In some ways, operating systems are just like any other user-space program. They are written in a high-level language, they allocate memory, and they have pointers, references, and array indexing. Enforcing memory safety for an operating system is conceptually the same as enforcing memory safety on user-space applications.
And yet, operating systems are not quite the same. They introduce new challenges. They often use custom alloctors, must induce very low overhead, and have time-critical functionality. Operating systems also perform certain operations that are rare in the application world: context switching, direct hardware I/O, and asynchronous event delivery, just to name a few.
It's therefore not surprising that operating systems have gotten special treatment in the research literature. Below are some papers on this specialized topic.
Operating System Memory Safety Papers
Memory Safety for Low-Level Software/Hardware Interactions
John Criswell, Nicolas Geoffray, and Vikram Adve
Proceedings of the Eighteenth USENIX Security Symposium, Montreal, Canada, August 2009.
Secure Virtual Architecture: A Safe Execution Environment for Commodity
John Criswell, Andrew Lenharth, Dinakar Dhurjati, and Vikram Adve
Proceedings of the Twenty First ACM Symposium on Operating Systems Principles (SOSP '07), Stevenson, WA, October 2007.
Received an SOSP 2007 Audience Choice Award.
SafeDrive: Safe and Recoverable Extensions Using Language-based
Feng Zhou, Jeremy Condit, Zachary Anderson, Ilya Bagrak, Rob Ennals, Matthew Harren, George Necula, and Eric Brewer
Proceedings of the Seventh Symposium on Operating Systems Design and Implementation (OSDI '06), Seattle, WA, 2006.