CS 6332: Systems Security and Binary Code Analysis Friday 19:00-21:45 at ECSS 2.201 $Id: fall2016.t2t, v1.0, last updated %%mtime(%c) Exp $ %! Target: %! Options: --toc --css-sugar --encoding=iso-8859-1 %! Style: tech.css %! PreProc: %! PostProc: --------------------------------------------------------------------------- === Course Overview === CS 6332 is a graduate level, research oriented, systems and software security class. The goal of this course is to understand the low-level details of the real system software implementations such as OS kernels by using techniques such as virtual machine introspection; examine the state of the art software vulnerabilities and attacks, such as memory exploits (e.g., ROP); design practical systems defense (e.g., using the recent advances such as hardware support for trusted computing); design program analysis to reverse engineer the binary code. The learning outcome is students shall be able to understand and know - Automated program analysis for the reverse engineering of the binary code. Static binary code analysis. Dynamic Binary code instrumentation. Data flow analysis, pointer analysis, and control flow analysis. Program slicing. - Vulnerability Discovery, Memory Exploits, and system defense. Understand the common software vulnerabilities such as buffer overflow, format string, integer overflows. Understand how to develop exploits against each vulnerability, and understand how to bypass the state-of-the-art defense - Virtual Machine Introspection. Understand how to use hypervisor level monitoring to introspect kernel events, to design intrusion detection systems, as well as control the guest OS execution. - Hardware supported for trusted computing. Learn the recent hardware advances in trusted computing, e.g., SGX, and understand how to design security applications by using SGX. --------------------------------------------------------------------------- === Text Books === ====Required textbooks==== - [CSAPP] Randal E. Bryant and David R. O'Hallaron. [``Computer Systems: A Programmer's Perspective, 2/E'' http://csapp.cs.cmu.edu/] - [AOE] Erickson, Jon. [``Hacking: The Art of Exploitation'' 2nd Edition http://proquest.safaribooksonline.com/book/networking/security/9781593271442] ====Reference textbooks==== - [PPA] Nielson, Flemming, Nielson, Hanne R., Hankin, Chris. ``Principles of Program Analysis''. Springer. - [TSH] Kozoil, Jack. [``The Shellcoder's Handbook: Discovering and Exploiting Security Holes'' http://proquest.safaribooksonline.com/book/networking/security/9780470080238] - [CRH] Ed Skoudis; Tom Liston. [``Counter Hack Reloaded: A Step-by-Step Guide to Computer Attacks and Effective Defenses'', Second Edition http://proquest.safaribooksonline.com/9780131481046] - [SRE] Eldad Eilam. [``Reversing: Secrets of Reverse Engineering'' http://proquest.safaribooksonline.com/book/software-engineering-and-development/9780764574818] --------------------------------------------------------------------------- === Course Notes and Tentative Schedule === $ Last updated %%mtime(%c) Exp $ || Lecture# | TOPICS | Assigned Readings | | 1 | Course Logistics and Overview | [CSAPP] [Ch2-Ch6] | || Basic Computer Systems Knowledge ||||| | 2 | Low Level Code (x86 assembly) | [CSAPP] [Ch2-Ch6] | | 3 | Low Level Systems Software: OS Kernel | [CSAPP] [Ch2-Ch6] | | 4 | Low Level Systems Software: Compiler, Linker, Loader | [CSAPP] [Ch2-Ch6] | || Binary Code Analysis (Reverse Engineering) Techniques ||||| | 5 | Introduction to Binary Analysis | [WYSINWYE http://research.cs.wisc.edu/wpis/papers/wysinwyx05.pdf] | | 6 | Understanding ELF Binary Format | [ELF Format http://www.skyfree.org/linux/references/ELF_Format.pdf] | | 7 | Dynamic Binary Analysis | [Pin Manual https://software.intel.com/sites/landingpage/pintool/docs/71313/Pin/html/] | | 8 | Program Representations for Analysis | [Program Slicing http://www.cs.toronto.edu/~chechik/courses06/csc2125/tip95survey.pdf] | | 9 | Program Slicing | [Program Slicing http://www.cs.toronto.edu/~chechik/courses06/csc2125/tip95survey.pdf] | | 10 | Data Flow Analysis | [Principles of Program Analysis Ch2 http://www2.imm.dtu.dk/~riis/PPA/slides2.pdf] | | 11 | Pointer Analysis | [Point-to Analysis http://www.cs.utexas.edu/~pingali/CS395T/2012sp/lectures/points-to.pdf] | | 12 | Shape Analysis | [Shape Analysis and Applications][[Paper1 http://www.utdallas.edu/~zhiqiang.lin/file/f15/shape-analysis-ch12.pdf],[Paper2 http://www.cs.utexas.edu/users/mckinley/papers/dsa-ismm-2009.pdf]] | | 13 | Dynamic Taint Analysis | [Dynamic Data Flow Tracking http://nsl.cs.columbia.edu/papers/2012/libdft.vee12.pdf] | | 14 | Value Set Analysis | [WYSINWYE http://research.cs.wisc.edu/wpis/papers/wysinwyx05.pdf] | | 15 | Symbolic Execution | [Automated Whitebox Fuzzing http://research.microsoft.com/en-us/um/people/pg/public_psfiles/ndss2008.pdf] | || Software Security: Vulnerabilities and Defenses ||||| | 16 | Control Flow Hijacks, Buffer Overflows | [Stack Smashing http://www-inst.eecs.berkeley.edu/~cs161/fa08/papers/stack_smashing.pdf], [2011 Stack Smashing https://paulmakowski.wordpress.com/2011/01/25/smashing-the-stack-in-2011/] | | 17 | Exploit Development | [AOE] [Ch3 http://proquest.safaribooksonline.com/9781593271442/exploitation] | | 18 | Integer and Heap Overflow | [AOE] [Ch3 http://proquest.safaribooksonline.com/9781593271442/exploitation] | | 19 | Format String Vulnerability | [AOE] [Ch3 http://proquest.safaribooksonline.com/9781593271442/exploitation] | | 20 | Control Flow Defense (Canary, DEP, ASLR) | [ASLR www.utdallas.edu/~zhiqiang.lin/file/aslr.pdf] | | 21 | Return Oriented Programming | [ROP https://cseweb.ucsd.edu/~hovav/dist/rop.pdf], [BROP http://www.scs.stanford.edu/brop/bittau-brop.pdf] | || Systems Security: Kernel Monitoring, Virtualization, and Hardware Security ||||| | 22 | Virtual Machine Introspection: Introduction | [VMI Survey http://www.utdallas.edu/~zhiqiang.lin/file/CSUR15.pdf] | | 23 | Virtual Machine Introspection: Challenges | [VMI Survey http://www.utdallas.edu/~zhiqiang.lin/file/CSUR15.pdf] | | 24 | Virtual Machine Introspection: Approaches | [VMI Survey http://www.utdallas.edu/~zhiqiang.lin/file/CSUR15.pdf] | | 25 | Virtual Machine Introspection: Applications | [VMI Survey http://www.utdallas.edu/~zhiqiang.lin/file/CSUR15.pdf] | | 26 | Hardware Security: Intel Software Guard Extension-I | [SGX Manual ch1-ch4 https://software.intel.com/sites/default/files/managed/48/88/329298-002.pdf] | | 27 | Hardware Security: Intel Software Guard Extension-II | [SGX Manual ch1-ch4 https://software.intel.com/sites/default/files/managed/48/88/329298-002.pdf] | || Final Exam: Date: December 11, 2015 Time: 8:00pm-10:45pm Location: ECSS 2.306 ||||| --------------------------------------------------------------------------- === Office Hours === - Instructor: Friday 4:00PM - 6:00PM (or by appointment). Office ECSS 3.226 - Teaching Assistant: Mr. Vishal Karande, Tuesday, Thursday 2:00PM - 3:00PM (or by appointment vishal.karande@utdallas.edu). Office ECSS 3.612 --------------------------------------------------------------------------- === Prerequisites === This is a highly technical class. We expect students to have a strong technical background before taking this course. Students who have not taken a security class before or whom are otherwise unfamiliar with computer security will likely not be able to complete this class. Specifically, students should satisfy at least **three** of the following: - Assembly code (Intel X86 preferred) - Knowledge of Computer Security basics (CS 4393 Computer and Network Security) - Proficiency in programing development (gcc/gdb) (CS 3376 C/C++ Programming in a UNIX Environment) - Proficiency in a scripting language (python preferably) - Familiarity with operating system kernel/internals (windows or linux) (CS 4348 Operating Systems Concepts) - Familiarity with command line operation of Windows AND Linux UTD course catalog (at least **three** of the following) - CS 3340 Computer Architecture - CS 3376 C/C++ Programming in a UNIX Environment - CS 4348 Operating Systems Concepts - CS 4393 Computer and Network Security - CS 4394 Implementation of Modern Operating Systems --------------------------------------------------------------------------- === Course Projects === Please visit e-learning to check out the projects. --------------------------------------------------------------------------- === Course Policy === ==== Late Policy ==== All late submissions will automatically lose 1 point per delayed day until the points in that project are gone. ==== Collaboration Policy ==== Students are encouraged to collaborate, particularly on the discussion on the course project. However, each individual must finish the project by him/her-self. ==== Cheating Policy ==== We will strictly follow the university policy on cheating and plagiarism which is available [here http://www.utdallas.edu/judicialaffairs/UTDJudicialAffairs-policies.html]. Please [avoid http://www.utdallas.edu/judicialaffairs/UTDJudicialAffairs-AvoidDishonesty.html]. There are also several examples of [Scholastic Dishonesty http://www.utdallas.edu/judicialaffairs/UTDJudicialAffairs-Basicexamples.html] If you have any questions regarding this issue, please contact the instructor. %!include: ''new.js'' --------------------------------------------------------------------------- [HOME index.html] [SOURCE %%infile]