CS 6332: Systems Security and Binary Code Analysis

Course Overview

CS 6332 is a graduate level, research oriented, systems and software security class. The goal of this course is to understand the low-level details of the real system software implementations such as OS kernels by using techniques such as virtual machine introspection; examine the state of the art software vulnerabilities and attacks, such as memory exploits (e.g., ROP); design practical systems defense (e.g., using the recent advances such as hardware support for trusted computing); design program analysis to reverse engineer the binary code.

The learning outcome is students shall be able to understand and know

Automated program analysis for the reverse engineering of the binary code. Static binary code analysis. Dynamic Binary code instrumentation. Data flow analysis, pointer analysis, and control flow analysis. Program slicing.
Vulnerability Discovery, Memory Exploits, and system defense. Understand the common software vulnerabilities such as buffer overflow, format string, integer overflows. Understand how to develop exploits against each vulnerability, and understand how to bypass the state-of-the-art defense
Virtual Machine Introspection. Understand how to use hypervisor level monitoring to introspect kernel events, to design intrusion detection systems, as well as control the guest OS execution.
Hardware supported for trusted computing. Learn the recent hardware advances in trusted computing, e.g., SGX, and understand how to design security applications by using SGX.

Text Books

Required textbooks

[CSAPP] Randal E. Bryant and David R. O'Hallaron. ``Computer Systems: A Programmer's Perspective, 2/E''
[AOE] Erickson, Jon. ``Hacking: The Art of Exploitation'' 2nd Edition

Reference textbooks

[PPA] Nielson, Flemming, Nielson, Hanne R., Hankin, Chris. ``Principles of Program Analysis''. Springer.
[TSH] Kozoil, Jack. ``The Shellcoder's Handbook: Discovering and Exploiting Security Holes''
[CRH] Ed Skoudis; Tom Liston. ``Counter Hack Reloaded: A Step-by-Step Guide to Computer Attacks and Effective Defenses'', Second Edition
[SRE] Eldad Eilam. ``Reversing: Secrets of Reverse Engineering''

Course Notes and Tentative Schedule

$ Last updated Thu Aug 25 11:39:26 2016 Exp $

Lecture#	TOPICS	Assigned Readings
1	Course Logistics and Overview	[CSAPP] [Ch2-Ch6]
Basic Computer Systems Knowledge
2	Low Level Code (x86 assembly)	[CSAPP] [Ch2-Ch6]
3	Low Level Systems Software: OS Kernel	[CSAPP] [Ch2-Ch6]
4	Low Level Systems Software: Compiler, Linker, Loader	[CSAPP] [Ch2-Ch6]
Binary Code Analysis (Reverse Engineering) Techniques
5	Introduction to Binary Analysis	WYSINWYE
6	Understanding ELF Binary Format	ELF Format
7	Dynamic Binary Analysis	Pin Manual
8	Program Representations for Analysis	Program Slicing
9	Program Slicing	Program Slicing
10	Data Flow Analysis	Principles of Program Analysis Ch2
11	Pointer Analysis	Point-to Analysis
12	Shape Analysis	Shape Analysis and [Paper1,Paper2]
13	Dynamic Taint Analysis	Dynamic Data Flow Tracking
14	Value Set Analysis	WYSINWYE
15	Symbolic Execution	Automated Whitebox Fuzzing
Software Security: Vulnerabilities and Defenses
16	Control Flow Hijacks, Buffer Overflows	Stack Smashing, 2011 Stack Smashing
17	Exploit Development	[AOE] Ch3
18	Integer and Heap Overflow	[AOE] Ch3
19	Format String Vulnerability	[AOE] Ch3
20	Control Flow Defense (Canary, DEP, ASLR)	ASLR
21	Return Oriented Programming	ROP, BROP
Systems Security: Kernel Monitoring, Virtualization, and Hardware Security
22	Virtual Machine Introspection: Introduction	VMI Survey
23	Virtual Machine Introspection: Challenges	VMI Survey
24	Virtual Machine Introspection: Approaches	VMI Survey
25	Virtual Machine Introspection: Applications	VMI Survey
26	Hardware Security: Intel Software Guard Extension-I	SGX Manual ch1-ch4
27	Hardware Security: Intel Software Guard Extension-II	SGX Manual ch1-ch4
Final Exam: Date: December 11, 2015 Time: 8:00pm-10:45pm Location: ECSS 2.306

Office Hours

Instructor: Friday 4:00PM - 6:00PM (or by appointment). Office ECSS 3.226
Teaching Assistant: Mr. Vishal Karande, Tuesday, Thursday 2:00PM - 3:00PM (or by appointment vishal.karande@utdallas.edu). Office ECSS 3.612

Prerequisites

This is a highly technical class. We expect students to have a strong technical background before taking this course. Students who have not taken a security class before or whom are otherwise unfamiliar with computer security will likely not be able to complete this class. Specifically, students should satisfy at least three of the following:

Assembly code (Intel X86 preferred)
Knowledge of Computer Security basics (CS 4393 Computer and Network Security)
Proficiency in programing development (gcc/gdb) (CS 3376 C/C++ Programming in a UNIX Environment)
Proficiency in a scripting language (python preferably)
Familiarity with operating system kernel/internals (windows or linux) (CS 4348 Operating Systems Concepts)
Familiarity with command line operation of Windows AND Linux

UTD course catalog (at least three of the following)

CS 3340 Computer Architecture
CS 3376 C/C++ Programming in a UNIX Environment
CS 4348 Operating Systems Concepts
CS 4393 Computer and Network Security
CS 4394 Implementation of Modern Operating Systems

Course Projects

Please visit e-learning to check out the projects.

Course Policy

Late Policy

All late submissions will automatically lose 1 point per delayed day until the points in that project are gone.

Collaboration Policy

Students are encouraged to collaborate, particularly on the discussion on the course project. However, each individual must finish the project by him/her-self.

Cheating Policy

We will strictly follow the university policy on cheating and plagiarism which is available here. Please avoid. There are also several examples of Scholastic Dishonesty If you have any questions regarding this issue, please contact the instructor.

HOME SOURCE