CS 6V81--005: System Security and Malicious Code Analysis Monday and Wednesday: 1:00pm-2:15pm SOM 2.903 $Id: spring2012.t2t, v1.0, last updated %%mtime(%c) Exp $ %! Target: %! Options: --toc --css-sugar --encoding=iso-8859-1 %! Style: tech.css %! PreProc: %! PostProc: --------------------------------------------------------------------------- === Course Overview === CS-6V81 is a graduate level, research oriented, system and software security course. The goal of this course is to explain the low-level system details from compiler, linker, loader, to OS kernel and computer architectures, examine the weakest link in each system component, explore the left bits and bytes after all these transformations, and study the state-of-the-art offenses and defenses. The learning outcome is students shall be able to understand how an attack is launched (e.g., how an expoit is created), and how to do the defense (e.g., developing OS patches, analyzing the binary code, and detecting intrusions) In particular, we will cover - **Memory exploits**. We will investigate the unsafe but widely used system programming language C, cover typical vulnerabilities such as buffer overflows, format strings, integer overflows, etc. How to create robust shell code using such as ROP, HeapSpray. - **OS Kernel Internals**. What's the behavior when a program is running on top of OS. Why we use paging. How virtual to physical address translation is performed. How MMU (e.g., TLB) helps this. How OS manage files, and disks. How can we model the program behavior when sitting at OS layer. We will use both Linux and Windows as working kernel. - **Linker and Loader Internals**. How a program can be dynamically linked, and what an attacker can do to cheat the system and meanwhile what we can do to protect the system. - **Kernel-level Defense**, how can we defend against the common exploits, techniques including such as ASR, and DEP, NX-bits. - **User-level Defense**. Safe library, Compiler extension, Binary Transformation/Rewriting, Runtime Verification. - **Binary code reverse engineering**. Static binary code analysis. Dynamic Binary code instrumentation. Data flow analysis, and control flow analysis. Malware packing and unpacking. The class will also have a heavy-hands on project. Students could choose either to perform research (will work on a semester-long research topic of their choosing), or perform an engineering project. --------------------------------------------------------------------------- === Course Schedule === %==== Instructor Lectures (60 minutes) ==== || Date | Topic To Be Covered | Presenter | Slides | | 01/18 | Course Overview | Instructor | [[pdf spring2012/public/lec1.pdf]] [[handout spring2012/public/lec1-handout.pdf]] | || System and Software Security Foundations: Understanding Binary Code Analysis |||||| | 01/23 | Binary Code/Data Representation | Instructor | [[pdf spring2012/public/lec2.pdf]] [[handout spring2012/public/lec2-handout.pdf]] | | 01/25 | Program Representation | Instructor | [[pdf spring2012/public/lec3.pdf]] [[handout spring2012/public/lec3-handout.pdf]] | | 01/30 | Dynamic Binary Instrumentation (PIN, Valgrind, Qemu) | Instructor | [[pdf spring2012/public/lec4.pdf]] [[handout spring2012/public/lec4-handout.pdf]] | | 02/01 | Principles of Program Analysis | Instructor | [[pdf spring2012/public/lec5.pdf]] [[handout spring2012/public/lec5-handout.pdf]] | | 02/06 | Guest Lecture: Recent Cyber Attacks and Implications | Jon Shapiro | [[pdf spring2012/lec6.pptx]] | | 02/08 | Guest Lecture: Web Vulnerability (SQL injection, Cross-site scripting) Analysis | Duong Ngo | N/A | | 02/13 | Design and Implementation of a Data Flow Analysis (taint analysis) | Instructor | [[pdf spring2012/public/lec6.pdf]] [[handout spring2012/public/lec6-handout.pdf]] | || System and Software Security Foundations: Understanding the OS Kernel |||||| | 02/15 | Understanding the OS Architecture and Linux History | Instructor | [[pdf spring2012/public/lec7.pdf]] [[handout spring2012/public/lec7-handout.pdf]] | | 02/20 | An Overview of Linux and Windows Kernel | Instructor | [[pdf spring2012/public/lec8.pdf]] [[handout spring2012/public/lec8-handout.pdf]] | | 02/22 | Process Management | Instructor | [[pdf spring2012/public/lec9.pdf]] [[handout spring2012/public/lec9-handout.pdf]] | | 02/27 | Virtual Memory (I) | Instructor | [[pdf spring2012/public/lec10.pdf]] [[handout spring2012/public/lec10-handout.pdf]] | | 02/29 | Virtual Memory (II) | Instructor | [[pdf spring2012/public/lec11.pdf]] [[handout spring2012/public/lec11-handout.pdf]] | | 03/05 | File System (I) | Instructor | [[pdf spring2012/public/lec12.pdf]] [[handout spring2012/public/lec12-handout.pdf]] | | 03/07 | File System (II) | Instructor | [[pdf spring2012/public/lec13.pdf]] [[handout spring2012/public/lec13-handout.pdf]] | | 03/12* | No-class (Spring-break) | | | | | | 03/14* | No-class (Spring-break) | | | | | || System and Software Security Foundations: Beyond OS Kernel |||||| | 03/19 | Revealing Internals of Executable File Format | Instructor | [[pdf spring2012/public/lec14.pdf]] [[handout spring2012/public/lec14-handout.pdf]] | | 03/21 | Revealing Internals of Compiler (gcc) | Instructor | [[pdf spring2012/public/lec15.pdf]] [[handout spring2012/public/lec15-handout.pdf]] | | 03/26 | Revealing Internals of Linker (ld) | Instructor | [[pdf spring2012/public/lec16.pdf]] [[handout spring2012/public/lec16-handout.pdf]] | | 03/28 | Revealing Internals of Loader (ld-linux.so) | Instructor | [[pdf spring2012/public/lec17.pdf]] [[handout spring2012/public/lec17-handout.pdf]] | || System and Software Security: Techniques, Tools, and Applications |||||| | 04/02 | Library Interposition | Instructor | [[pdf spring2012/public/lec18.pdf]] [[handout spring2012/public/lec18-handout.pdf]] | | 04/04 | Virtual Machine Monitor (QEMU/VirtualBox/Xen/KVM) | Instructor | [[pdf spring2012/public/lec19.pdf]] [[handout spring2012/public/lec19-handout.pdf]] | | 04/09 | Symbolic Execution and Whitebox Fuzzing | Instructor | [[pdf spring2012/public/lec20.pdf]] [[handout spring2012/public/lec20-handout.pdf]] | | 04/11 | Exploits: Buffer Overflows, Heap Overflow, Integer Overflow | Instructor | [[pdf spring2012/public/lec21.pdf]] [[handout spring2012/public/lec21-handout.pdf]] | | 04/16 | Robust Exploits: ROP shellcode, Heap Spray | Instructor | [[pdf spring2012/public/lec22.pdf]] [[handout spring2012/public/lec22-handout.pdf]] | | 04/18 | Fighting for Malware: Unpack, Disassemble, Decompile | Instructor | [[pdf spring2012/public/lec23.pdf]] [[handout spring2012/public/lec23-handout.pdf]] | | 04/23 | Binary Code Reusing | Instructor | [[pdf spring2012/public/lec24.pdf]] [[handout spring2012/public/lec24-handout.pdf]] | %==== Students Presentation (15 minutes) ==== || Student Presentation (15 minutes) |||| || Vulnerability, Exploit, Malware |||| | 01/23 | [Smashing the stack for fun and profit http://insecure.org/stf/smashstack.html] | Mitch Adair | [[pdf spring2012/lec2.pdf]] | | 01/25 | [Smashing the stack in 2011 http://paulmakowski.wordpress.com/2011/01/25/smashing-the-stack-in-2011/] | Andrew Folloder | [[pdf spring2012/lec3.pdf]] | | 01/30 | [Exploiting Format String Vulnerabilities http://crypto.stanford.edu/cs155old/cs155-spring08/papers/formatstring-1.2.pdf] | Sanjay Bysani | [[pdf spring2012/lec4.pdf]] | | 02/01 | [English Shellcode http://www.cs.jhu.edu/~sam/ccs243-mason.pdf] | Shwetha Gopalan | [[pdf spring2012/lec5.pdf]] %| 02/06 | Guest Lecture: Recent Cyber Attacks and Implications | Jon Shapiro | [[pdf spring2012/lec6.pptx]] | %| 02/08 | Guest Lecture: Web Vulnerability (SQL injection, Cross-site scripting) Analysis | Duong Ngo | | | 02/13 | [Return-oriented programming http://dl.acm.org/citation.cfm?id=1315313&CFID=77962366&CFTOKEN=86878402] | Scott Hand | [[pdf spring2012/lec8.pdf]] | | 02/15 | [ASLR Smack and Laugh Reference http://www.ece.cmu.edu/~dbrumley/courses/18739c-s11/docs/aslr.pdf] | Mohammed Andaleeb Iftekhar | [[pdf spring2012/lec9.pptx]] | | 02/20 | [Automated Exploit Generation http://security.ece.cmu.edu/aeg/aeg-current.pdf] | Matthew Stephen | [[pdf spring2012/lec10.pdf]] | | 02/22 | [How to Shop for Free Online - Security Analysis of Cashier-as-a-Service Based Web Stores http://research.microsoft.com/pubs/145858/caas-oakland-final.pdf] | Isaac Strohl,Avinash Joshi | [[pdf spring2012/lec11.pdf]] | || System Defenses: Architecture, OS, Compilation Extension, Code Transformation, Runtime Verification |||| | 02/27 | [Traps and Pitfalls: Practical Problems in System Call Interposition Based Security Tools http://www.stanford.edu/~talg/papers/traps/traps-ndss03.pdf] | Vinay Gangasani | [[ppt spring2012/lec12.pptx]] | | 02/29 | [Control Flow Integrity http://dl.acm.org/citation.cfm?id=1609960&dl=ACM&coll=DL] | Murugesan, Sureshbabu | [[pdf spring2012/lec13.pdf]] | | 03/05 | [On the Effectiveness of Address Space Randomization http://dl.acm.org/citation.cfm?id=1030124] | Brian Ricks,Vasundhara Chimmad | [[ppt spring2012/lec14.pptx]] | | 03/07 | [Dynamic Taint Analysis for Automatic Detection, Analysis, and Signature Generation of Exploits on Commodity Software http://www.isoc.org/isoc/conferences/ndss/05/proceedings/papers/taintcheck.pdf] | Sheikh Qumruzzaman, Khaled Al-Naami | [[ppt spring2012/taint.pptx]] | | 03/19 | [Efficient and Accurate Detection of Integer-based Attacks http://www.ece.cmu.edu/~dbrumley/pubs/integer-ndss-07.pdf] | Allen Helton, Nishant Chithambaram | [[ppt spring2012/lec15.pptx]] | | 03/21 | [Bouncer: Securing Software by Blocking Bad Input http://www.sosp2007.org/papers/sosp166-costa.pdf] | Yufei Gu,Sathish Kuppuswamy | [[pdf spring2012/lec16.pdf]] | | 03/26 | [Static detection of C++ vtable escape vulnerabilities in binary code] | Huseyin Ulusoy | [[pdf spring2012/vtable.pdf]] | | 03/28 | [Kruiser: Semi-synchronized Non-blocking Concurrent Kernel Heap Buffer Overflow Monitoring] | Shishir K Prasad | [[pdf spring2012/lec17.pdf]] | | 04/02 | [Native Client: A Sandbox for Portable, Untrusted x86 Native Code http://src.chromium.org/viewvc/native_client/data/docs_tarball/nacl/googleclient/native_client/documentation/nacl_paper.pdf] | Gil Lundquist | [[pdf spring2012/lec18.pdf]] | | 04/04 | [Software fault isolation with API integrity and multi-principal modules. http://people.csail.mit.edu/nickolai/papers/mao-lxfi.pdf] | Junyuan Zeng | [[pdf spring2012/lec19.pdf]] | | 04/09 | [A Virtual Machine Introspection Based Architecture for Intrusion Detection http://suif.stanford.edu/papers/vmi-ndss03.pdf] | Donald Talkington,sundarajan srinivasan | [[ppt spring2012/lec20.ppt]] | | 04/16 | [Robust Defenses for Cross-Site Request Forgery http://dl.acm.org/citation.cfm?id=1455782] | Saravana M Subramanian | [[ppt spring2012/csr.pptx]] | || Malicious Code Analysis |||| | 04/18 | [Deobfuscation of virtualization-obfuscated software http://dl.acm.org/citation.cfm?id=2046739] | Selvakumar Gopal Rajendran | [[pdf spring2012/de.pdf]] | | 04/23 | [Who Wrote This Code? Identifying the Authors of Program Binaries http://dl.acm.org/citation.cfm?id=2041239] | Camron | [[ppt spring2012/camron.ppt]] | | 04/25 | [Measuring Pay-per-Install: The Commoditization of Malware Distribution http://www.icir.org/vern/papers/ppi-usesec11.pdf] | Kevin Hulin | [[pdf spring2012/kevin.pdf]] || Project Presentation |||| | 04/30 | Project Presentation | | | | 05/02 | Project Presentation | | | --------------------------------------------------------------------------- === Reading List === We do not have a text book, but we have the following reading list. Students are required to read all of these papers. - [Smashing the Stack for Fun and Profit http://insecure.org/stf/smashstack.html] - [Smashing the Stack in 2011 http://paulmakowski.wordpress.com/2011/01/25/smashing-the-stack-in-2011/] - [Exploiting Format String Vulnerabilities http://crypto.stanford.edu/cs155old/cs155-spring08/papers/formatstring-1.2.pdf] - [English Shellcode http://www.cs.jhu.edu/~sam/ccs243-mason.pdf] - [On the Effectiveness of Address Space Randomization http://dl.acm.org/citation.cfm?id=1030124] - [Return-oriented programming http://dl.acm.org/citation.cfm?id=1315313&CFID=77962366&CFTOKEN=86878402] - [Backwards-Compatible Array Bound Checking for C http://llvm.org/pubs/2006-05-24-SAFECode-BoundsCheck.pdf] - [ASLR Smack and Laugh Reference http://www.ece.cmu.edu/~dbrumley/courses/18739c-s11/docs/aslr.pdf] - [Control Flow Integrity http://dl.acm.org/citation.cfm?id=1609960&dl=ACM&coll=DL] - [Robust Defenses for Cross-Site Request Forgery http://dl.acm.org/citation.cfm?id=1455782] - [All you ever wanted to know about dynamic taint analysis and symbolic execution http://www.ece.cmu.edu/~ejschwar/papers/oakland10.pdf] - [Dynamic Taint Analysis for Automatic Detection, Analysis, and Signature Generation of Exploits on Commodity Software http://www.isoc.org/isoc/conferences/ndss/05/proceedings/papers/taintcheck.pdf] - [Automated Exploit Generation http://security.ece.cmu.edu/aeg/aeg-current.pdf] - [Traps and Pitfalls: Practical Problems in System Call Interposition Based Security Tools http://www.stanford.edu/~talg/papers/traps/traps-ndss03.pdf] - [Native Client: A Sandbox for Portable, Untrusted x86 Native Code http://src.chromium.org/viewvc/native_client/data/docs_tarball/nacl/googleclient/native_client/documentation/nacl_paper.pdf] - [Deobfuscation of virtualization-obfuscated software http://dl.acm.org/citation.cfm?id=2046739] - [BitShred: Fast, Scalable Code Reuse Detection in Binary Code http://www.ece.cmu.edu/~jiyongj/papers/cmu-cylab-10-006.pdf] - [Kruiser: Semi-synchronized Non-blocking Concurrent Kernel Heap Buffer Overflow Monitoring] - [How to Shop for Free Online - Security Analysis of Cashier-as-a-Service Based Web Stores http://research.microsoft.com/pubs/145858/caas-oakland-final.pdf] - [Differential Slicing: Identifying Causal Execution Differences for Security Applications http://bitblaze.cs.berkeley.edu/papers/diffslicing_oakland11.pdf] - [Efficient and Accurate Detection of Integer-based Attacks http://www.ece.cmu.edu/~dbrumley/pubs/integer-ndss-07.pdf] - [Bouncer: Securing Software by Blocking Bad Input http://www.sosp2007.org/papers/sosp166-costa.pdf] - [Semantics-aware Malware Detection http://www.cs.berkeley.edu/~dawnsong/papers/semantic-aware.pdf] - [Static detection of C++ vtable escape vulnerabilities in binary code] - [Identifying and Analyzing Pointer Misuses for Sophisticated Memory-corruption Exploit Diagnosis] - [A Virtual Machine Introspection Based Architecture for Intrusion Detection http://suif.stanford.edu/papers/vmi-ndss03.pdf] - [Software fault isolation with API integrity and multi-principal modules. http://people.csail.mit.edu/nickolai/papers/mao-lxfi.pdf] - [Measuring Pay-per-Install: The Commoditization of Malware Distribution http://www.icir.org/vern/papers/ppi-usesec11.pdf] - [A Study of the Packer Problem and Its Solutions http://www.ecsl.cs.sunysb.edu/tr/TR237.pdf] --------------------------------------------------------------------------- === Office Hours === Monday, Wednesday 3-4PM --------------------------------------------------------------------------- === Prerequisites === Solid programming/development skills (Assembly, C, C++, Unix) are required for this class. "Operating System", "Compilers", and "Computer Security", are the least prerequisites for this class. In particular, for UTD student - CS 3340 Computer Architecture - CS 3376 C/C++ Programming in a UNIX Environment - CS 4348 Operating Systems Concepts - CS 4393 Computer and Network Security - CS 4394 Implementation of Modern Operating Systems Note for undergraduate students who may be interested in taking this class, please be aware that the class is designed for graduate students, you are encouraged to attend the first lecture and then talk to the instructor. --------------------------------------------------------------------------- === Course Projects === - Dynamic Taint Analysis - Vulnerability Discovery - Reverse Engineering - Forensic data caving - ... --------------------------------------------------------------------------- === Course Policy === ==== Grading Policy ==== - 20% In-Class Presentations - 10% Class participation - 20% Scribs - 50% Class Project - Exceptional work will be rewarded appropriately ==== Late Policy ==== No late submission. ==== Collaboration Policy ==== Students are encouraged to collaborate, particularly on the course project. But we will limit the team member to at most three students. ==== Cheating Policy ==== We will strictly follow the university policy on cheating and plagiarism which is available [here http://www.utdallas.edu/judicialaffairs/UTDJudicialAffairs-policies.html]. Please [avoid http://www.utdallas.edu/judicialaffairs/UTDJudicialAffairs-AvoidDishonesty.html]. There are also several examples of [Scholastic Dishonesty http://www.utdallas.edu/judicialaffairs/UTDJudicialAffairs-Basicexamples.html] If you have any questions regarding this issue, please contact the instructor. === Acknowledgememt/References === - [Understanding the Linux Kernel. http://oreilly.com/catalog/9780596005658] - [Penetration Testing and Vulnerability Analysis http://pentest.cryptocity.net/] - [Computer Systems: A Programmer's Perspective http://csapp.cs.cmu.edu/] - [Software Security at CMU http://www.ece.cmu.edu/~dbrumley/courses/18732-f11] - [Computer Security at UC Berkeley http://inst.eecs.berkeley.edu/~cs161/archives.html] - [Computer and Network Security at Stanford https://courseware.stanford.edu/pg/courses/lectures/170183] - [Computer and Network Security at MIT http://courses.csail.mit.edu/6.857/2011/handouts] - [Secure Programming at iSecLab http://www.iseclab.org/secprog/] %!include: ''sp12.js'' --------------------------------------------------------------------------- [HOME index.html] [SOURCE %%infile]