CS 7301--003: Recent Advances in Computing - OPERATING SYSTEMS SECURITY

Course Overview

CS 7301 is a graduate level (PhD in particular), research oriented, systems and security, seminar class.

The goal of this course is to read, understand, and present the recent advances (which has not been systemized into text book yet) in operating systems security. We will select the most recent papers from both operating systems venues including SOSP, OSDI, USENIX ATC, EuroSys, ASPLOS, and security venues including IEEE S&P, ACM CCS, USENIX Security, and NDSS.

In particular, we will cover

Why kernel level attacks. What's the advantage of kernel level attacks? What are the attack vectors? How they are launched? How we can defend against them?
Out-of-the-box techniques. How to design the layer-below systems to secure OS kernel. What are the challenges? What is the state-of-the-art?
In-box techniques. Why we want to design in-box monitoring? What's the advantage and disadvantage? What's the state-of-the-art?
Emerging Systems Security. Android and iOS security.

The learning outcome is students will be able to understand the recent advances in operating systems security, the relevent security problems, and how these approaches/solutions are proposed.

Tentative Schedule

Please login your elearning website and download the lecture notes.

Week	Topic To Be Covered
0	Course Overview
1	Overivew of the landscape of modern computing, and OS Kernel Key Components
Understanding Kernel Level Attacks
2	Kernel level attack vectors: memory exploits, code based attacks
3	Kernel level attack vectors: rootkits, data-based, DKOM
Out-of-VM Techniques
4	Virtual Machine Introspection
5	Bridging the Semantic-Gap: SBCFI, VMwatcher, Virtuoso, VMST
6	Hardware-based monitoring: Copilot, Vigilare, mGuard, KI-Mon
7	Overshadow, CloudVisor
8	VM Management, forking, replication, deduplication
In-VM Techniques l
9	In-VM monitoring, Access Control
10	Control Flow Integrity (for COTS Binaries), CCFIR, STIR
11	Binary Rewriting, In-lined Reference Monitors
12	Vulnerability Analysis: KINT, KGuard
Potpourri
13	Kernel Data Structure Analysis: KOP, MAS, SigGraph, Value Invariant
14	Kernel ASLR, ROP
15	The papers of our interest
16	The papers of our interest

Reading List

We do not have a text book, but we have the following reading list. Students are required to read, and present one or two of the following papers. Note that the order these papers is of no particular meaning but just for you to track which paper you will be presenting.

[1] Just-In-Time Code Reuse: On the Effectiveness of Fine-Grained Address Space Layout Randomization [Oakland'13]
[2] Practical Control Flow Integrity & Randomization for Binary Executables [Oakland'13]
[3] Design, Implementation and Verification of an eXtensible and Modular Hypervisor Framework [Oakland'13]
[4] PrivExec: Private Execution as an Operating System Service [Oakland'13]
[5] Practical Timing Side Channel Attacks Against Kernel Space ASLR [Oakland'13]
[6] Control Flow Integrity for COTS Binaries [USENIX-Security'13]
[7] KI-Mon: A Hardware-assisted Event-triggered Monitoring Platform for Mutable Kernel Object [USENIX-Security'13]
[8] Jekyll on iOS: When Benign Apps Become Evil [USENIX-Security'13]
[9] Steal This Movie: Automatically Bypassing DRM Protection in Streaming Media Services [USENIX-Security'13]
[10] Safe Loading - A Foundation for Secure Execution of Untrusted Programs [Oakland'12]
[11] Space Traveling across VM: Automatically Bridging the Semantic Gap in Virtual Machine Introspection via Online Kernel Data Redirection [Oakland'12]
[12] Virtuoso: Narrowing the Semantic Gap in Virtual Machine Introspection [Oakland'11]
[13] PRISM: Program Replication and Integration for Seamless MILS [Oakland'11]
[14] Improving Integer Security for Systems with KINT [OSDI'12]
[15] FlexSC: Flexible System Call Scheduling with Exception-Less System Calls [OSDI'10]
[16] Accountable Virtual Machine [OSDI'10]
[17] Enabling Configuration-Independent Automation by Non-Expert Users [OSDI'10]
[18] Automating Configuration Troubleshooting with Dynamic Information Flow Analysis [OSDI'10]
[19] The Turtles Project: Design and Implementation of Nested Virtualization [OSDI'10]
[20] MiG: Efficient Migration of Desktop VMs Using Semantic Compression [USENIX-ATC'13]
[21] Practical and Effective Sandboxing for Non-root Users [USENIX-ATC'13]
[22] A Compiler-level Intermediate Representation based Binary Analysis and Rewriting System [EuroSys'13]
[23] Isolating Commodity Hosted Hypervisors with HyperLock [EuroSys'12]
[24] kGuard: Lightweight Kernel Protection against Return-to-User Attacks [USENIX-Security'12]
[25] Tracking Rootkit Footprints with a Practical Memory Analysis System [USENIX-Security'12]
[26] DroidScope: Seamlessly Reconstructing the OS and Dalvik Semantic Views for Dynamic Android Malware Analysis [USENIX-Security'12]
[27] CloudVisor: Retrofitting Protection of Virtual Machines in Multi-tenant Cloud with Nested Virtualization [SOSP'11]
[28] Overshadow: A Virtualization-Based Approach to Retrofitting Protection in Commodity Operating Systems [ASPLOS'08]
[29] Stealthy Malware Detection Through VMM-Based Out-of-the-Box Semantic View Reconstruction [CCS'07]
[30] Copilot --a Coprocessor-based Kernel Runtime Integrity Monitor [USENIX-Security'04]
[31] Vigilare: Toward Snoop-based Kernel Integrity Monitor [CCS'12]
[32] Combining Control-Flow Integrity and Static Analysis for Efficient and Validated Data Sandboxing [CCS'12]
[33] Process Out-Grafting: An Efficient Out-of-VM Approach for Fine-Grained Process Execution Monitoring [CCS'11]
[34] Secure In-VM Monitoring Using Hardware Virtualization [CCS'09]
[35] Taming Hosted Hypervisors with (Mostly) Deprivileged Execution [NDSS'13]
[36] Kruiser: Semi-synchronized Non-blocking Concurrent Kernel Heap Buffer Overflow Monitoring [NDSS'12]
[37] SigGraph: Brute Force Scanning of Kernel Data Structure Instances Using Graph-based Signatures [NDSS'11]
[38] A Virtual Machine Introspection Based Architecture for Intrusion Detection [NDSS'03]

Office Hours

Wednesday, Friday 4PM-6PM

Prerequisites

Solid programming/development skills (Assembly, C, C++, Unix) are required for this class. "Operating System", "Compilers", and "Computer Security", are the least prerequisites for this class. In particular, for UTD student

CS 3340 Computer Architecture
CS 3376 C/C++ Programming in a UNIX Environment
CS 4348 Operating Systems Concepts
CS 4393 Computer and Network Security
CS 4394 Implementation of Modern Operating Systems

Note for undergraduate students who may be interested in taking this class, please be aware that the class is designed for graduate students, you are encouraged to attend the first lecture and then talk to the instructor.

Course Projects

Design and Implement a Virtual Machine Introspection tool
...

Course Policy

Grading Policy

10% Class Participation
40% Class Presentation
50% Course Project
Exceptional work will be rewarded appropriately

Late Policy

No late submission.

Collaboration Policy

Students are encouraged to collaborate, particularly on the course project. But we will limit the team member to at most two students.

Cheating Policy

We will strictly follow the university policy on cheating and plagiarism which is available here. Please avoid. There are also several examples of Scholastic Dishonesty If you have any questions regarding this issue, please contact the instructor.

HOME SOURCE