Course Overview

CS-6V81 is a graduate level, research oriented, system and software security course.

The goal of this course is to explain the low-level system details from compiler, linker, loader, to OS kernel and computer architectures, examine the weakest link in each system component, explore the left bits and bytes after all these transformations, and study the state-of-the-art offenses and defenses.

The learning outcome is students shall be able to understand how an attack is launched (e.g., how an expoit is created), and how to do the defense (e.g., developing OS patches, analyzing the binary code, and detecting intrusions)

In particular, we will cover

The class will also have a heavy-hands on project. Students could choose either to perform research (will work on a semester-long research topic of their choosing), or perform an engineering project.

Course Schedule

Date Topic To Be Covered Presenter Slides
01/18 Course Overview Instructor [pdf] [handout]
System and Software Security Foundations: Understanding Binary Code Analysis
01/23 Binary Code/Data Representation Instructor [pdf] [handout]
01/25 Program Representation Instructor [pdf] [handout]
01/30 Dynamic Binary Instrumentation (PIN, Valgrind, Qemu) Instructor [pdf] [handout]
02/01 Principles of Program Analysis Instructor [pdf] [handout]
02/06 Guest Lecture: Recent Cyber Attacks and Implications Jon Shapiro [pdf]
02/08 Guest Lecture: Web Vulnerability (SQL injection, Cross-site scripting) Analysis Duong Ngo N/A
02/13 Design and Implementation of a Data Flow Analysis (taint analysis) Instructor [pdf] [handout]
System and Software Security Foundations: Understanding the OS Kernel
02/15 Understanding the OS Architecture and Linux History Instructor [pdf] [handout]
02/20 An Overview of Linux and Windows Kernel Instructor [pdf] [handout]
02/22 Process Management Instructor [pdf] [handout]
02/27 Virtual Memory (I) Instructor [pdf] [handout]
02/29 Virtual Memory (II) Instructor [pdf] [handout]
03/05 File System (I) Instructor [pdf] [handout]
03/07 File System (II) Instructor [pdf] [handout]
03/12* No-class (Spring-break)
03/14* No-class (Spring-break)
System and Software Security Foundations: Beyond OS Kernel
03/19 Revealing Internals of Executable File Format Instructor [pdf] [handout]
03/21 Revealing Internals of Compiler (gcc) Instructor [pdf] [handout]
03/26 Revealing Internals of Linker (ld) Instructor [pdf] [handout]
03/28 Revealing Internals of Loader ( Instructor [pdf] [handout]
System and Software Security: Techniques, Tools, and Applications
04/02 Library Interposition Instructor [pdf] [handout]
04/04 Virtual Machine Monitor (QEMU/VirtualBox/Xen/KVM) Instructor [pdf] [handout]
04/09 Symbolic Execution and Whitebox Fuzzing Instructor [pdf] [handout]
04/11 Exploits: Buffer Overflows, Heap Overflow, Integer Overflow Instructor [pdf] [handout]
04/16 Robust Exploits: ROP shellcode, Heap Spray Instructor [pdf] [handout]
04/18 Fighting for Malware: Unpack, Disassemble, Decompile Instructor [pdf] [handout]
04/23 Binary Code Reusing Instructor [pdf] [handout]
Student Presentation (15 minutes)
Vulnerability, Exploit, Malware
01/23 Smashing the stack for fun and profit Mitch Adair [pdf]
01/25 Smashing the stack in 2011 Andrew Folloder [pdf]
01/30 Exploiting Format String Vulnerabilities Sanjay Bysani [pdf]
02/01 English Shellcode Shwetha Gopalan [pdf]
02/13 Return-oriented programming Scott Hand [pdf]
02/15 ASLR Smack and Laugh Reference Mohammed Andaleeb Iftekhar [pdf]
02/20 Automated Exploit Generation Matthew Stephen [pdf]
02/22 How to Shop for Free Online - Security Analysis of Cashier-as-a-Service Based Web Stores Isaac Strohl,Avinash Joshi [pdf]
System Defenses: Architecture, OS, Compilation Extension, Code Transformation, Runtime Verification
02/27 Traps and Pitfalls: Practical Problems in System Call Interposition Based Security Tools Vinay Gangasani [ppt]
02/29 Control Flow Integrity Murugesan, Sureshbabu [pdf]
03/05 On the Effectiveness of Address Space Randomization Brian Ricks,Vasundhara Chimmad [ppt]
03/07 Dynamic Taint Analysis for Automatic Detection, Analysis, and Signature Generation of Exploits on Commodity Software Sheikh Qumruzzaman, Khaled Al-Naami [ppt]
03/19 Efficient and Accurate Detection of Integer-based Attacks Allen Helton, Nishant Chithambaram [ppt]
03/21 Bouncer: Securing Software by Blocking Bad Input Yufei Gu,Sathish Kuppuswamy [pdf]
03/26 Static detection of C++ vtable escape vulnerabilities in binary Huseyin Ulusoy [pdf]
03/28 Kruiser: Semi-synchronized Non-blocking Concurrent Kernel Heap Buffer Overflow Shishir K Prasad [pdf]
04/02 Native Client: A Sandbox for Portable, Untrusted x86 Native Code Gil Lundquist [pdf]
04/04 Software fault isolation with API integrity and multi-principal modules. Junyuan Zeng [pdf]
04/09 A Virtual Machine Introspection Based Architecture for Intrusion Detection Donald Talkington,sundarajan srinivasan [ppt]
04/16 Robust Defenses for Cross-Site Request Forgery Saravana M Subramanian [ppt]
Malicious Code Analysis
04/18 Deobfuscation of virtualization-obfuscated software Selvakumar Gopal Rajendran [pdf]
04/23 Who Wrote This Code? Identifying the Authors of Program Binaries Camron [ppt]
04/25 Measuring Pay-per-Install: The Commoditization of Malware Distribution Kevin Hulin [pdf]
Project Presentation
04/30 Project Presentation
05/02 Project Presentation

Reading List

We do not have a text book, but we have the following reading list. Students are required to read all of these papers.

Office Hours

Monday, Wednesday 3-4PM


Solid programming/development skills (Assembly, C, C++, Unix) are required for this class. "Operating System", "Compilers", and "Computer Security", are the least prerequisites for this class. In particular, for UTD student

Note for undergraduate students who may be interested in taking this class, please be aware that the class is designed for graduate students, you are encouraged to attend the first lecture and then talk to the instructor.

Course Projects

Course Policy

Grading Policy

Late Policy

No late submission.

Collaboration Policy

Students are encouraged to collaborate, particularly on the course project. But we will limit the team member to at most three students.

Cheating Policy

We will strictly follow the university policy on cheating and plagiarism which is available here. Please avoid. There are also several examples of Scholastic Dishonesty If you have any questions regarding this issue, please contact the instructor.