CAP 5137 Software Reverse Engineering and Malware Analysis

Spring 2025

Schedule:

Tu/Th 11:35AM - 12:50PM

Love 103

Rational:         

Computers and communication technologies have been incorporated into many applications and have fundamentally changed many aspects of human activities. Unfortunately, the changes have also created new problems, from spyware that steal data, computer viruses and worms that destroy data, to network enabled weapons, to cyber wars that can disable companies and even countries (such as Stuxnet). All these problems are related to computer security. Due to its paramount importance, computer security is not just one academic research area. Many security products are installed on typical computers; in the United States, there are multiple federal agencies dedicated to computer security; the computer security is a multibillion industry that is estimated to grow steadily. Computer security related issues have been widely recognized in software development companies. As computer security techniques evolve continuously along with product improvements and new service opportunities, computer security is and will remain an important and valuable area in the perceivable future with new career opportunities. As all computers (including communication devices) execute instructions, a fundamental requirement to achieving security is to be able to analyze binary programs as source code is not available in many situations and security often relies on implementation details not present in source code. This course is designed to cover the basic principles and techniques for software reverse engineering so that you can audit binary programs and analyze firmware samples and other stripped binaries.

Course Description:

This course covers fundamental problems, principles, and techniques in software reverse engineering of binaries including static analysis techniques, disassembly algorithms, dynamic analysis techniques, automated static and dynamic analysis techniques, malware analysis techniques, anti-analysis techniques, and malware obfuscation and packing techniques; many of the techniques will be demonstrated and practiced using IDA or Ghidra. It also involves research opportunities to analyze new malware samples and firmware samples, and develop new analysis tools.

Prerequisites:

CDA 3100 - Computer Organization I; having a good understanding of instruction set architectures (registers, instruction encoding and decoding, and memory organization) and basic data types, data structures, function calls (calling conventions), and memory layout of programs; be able to understand x86 and other assembly (assuming that instruction reference manuals are available); having a general understanding of computer security. 

Course Objectives

Upon successful completion of this course of study, the student will be able to: 

• recognize commonly used file formats
• Extract information from files in PE format and ELF format
• Dissemble code segments using the linear sweep and recursive descent disassembly algorithms
• Recognize the commonly used function calling conventions in disassembled files
• Construct basic blocks and calling graphs
• Identify conditional execution constructs in disassembled files
• Identify loop constructs in disassembled files
• Identify switch statements using a jump table in disassembled files
• Perform data flow analysis
• Recognize commonly used anti-disassembly techniques and overcome them in GHIDRA/IDA
• Execute malware samples safely in a virtual machine
• Use a debugger to monitor program execution
• Analyze an executable file in GHIDRA/IDA
• Recognize commonly used anti-debugging techniques and overcome them in GHIDRA/IDA
• Recognize commonly used anti-virtual machine techniques and overcome them in GHIDRA/IDA
• Recognize buffer overflow vulnerabilities in disassembled files
• Recognize and decode commonly used encoding algorithms in disassembled files
• Recognize and overcome commonly used obfuscation techniques in disassembled files
• Recognize commonly used packing methods in disassembled files
• Analyze malware samples packed using common packing techniques in GHIDRA/IDA
• Recognize commonly used malware mechanisms in disassembled files
• Incorporate new methods documented in research papers in reversing binaries
• Incorporate new malware analysis techniques documented in research papers in malware analysis
• Use scripts in GHIDRA/IDA
• Analyze firmware in GHIDRA/IDA

Textbooks and Course Materials

• Required textbook

  • "Practical Malware Analysis: The Hands-on Guide to Dissecting Malicious Software", by Michael Sikorski and Andrew Honig (published by No Starch Press, 2012).

• Recommended optional textbooks
  • "Hacking: The Art of Exploitation, 2nd Edition", by Jon Erickson. This is a book with accurate and detailed descriptions and commands of common vulnerabilities and corresponding exploits.
  • "The IDA PRO Book: The Unofficial Guide to the World's Most Popular Disassembler", 2nd Edition, by Chris Eagle (published by No Starch Press, 2011); this book provides a comprehensive coverage about IDA Pro, one of the most commonly used reverse engineering tools.
  • "The GHIDRA Book: The Definitive Guide" by Chris Eagle and Kara Nance (Penguin Random House Publisher Services, 2020); this book provides a systematic coverage of Ghidra.
  • In addition to the textbooks, papers and notes from the literature will be distributed along with the lectures.

Student Responsibilities

This course will cover certain techniques to exploit and break down known systems in order to demonstrate their vulnerabilities. It is illegal, however, to practice these techniques on others' systems. The students will be liable for their own behaviors and therefore consequences.

Workloads and Grading:

There will be one final exam, one midterm exam, about six homework assignments (most of them will involve using Ghidra/IDA, one term hands-project, and a few quizzes

1. About 5 homework assignments (40%)
2. Two exams (cumulative, 35%)
   • Midterm - 15% 
   • Final Exam - 20%
3. One term hands-on project - 10 %
4. A few quizzes - 10%
5. A research paper review - 5%

Final letter grades (based on weighted totals)

A	[93-100]
A-	[90-93)
B+	[87-90)
B	[83-87)
B-	[80-83)
C+	[77-80)
C	[73-77)
C-	[70-73)
D+	[67-70)
D	[63-67)
D-	[60-63)
F	<60

Course Policies:

Attendance Policy:

The university requires attendance in all classes, and it is also important to your learning. The attendance record may be provided to deans who request it. If your grade is just a little below the cutoff for a higher grade, your attendance will be one of the factors that we consider, in deciding whether to "bump" you up to the higher grade. Missing three or fewer lectures will be considered good attendance. In rare cases, such as medical needs or jury duty, absences may be excused with appropriate documentation. You should let me know in advance, when possible, and submit the documentation I seek. You should make up for any materials missed due to absences.

Missed exam Policy:

A missed exam will be recorded as a grade of zero. We will follow the university rules regarding missed final exams (see http://registrar.fsu.edu/dir%5Fclass/spring/exam_schedule.htm), for all the exams, including the final exam.

Late Assignment Policy:

In order to enable us to provide timely solutions to assignments, we have the following policy regarding submission of late assignments.

• An assignment that is turned in no more than 24 hours late will be scored with a 10% penalty.
• An assignment that is turned in more than 24 and no less than 48 hours late will be scored with a 20% penalty.
• An assignment that is turned in more than 48 hours late will receive the score of zero, though we will review it and comment on it.

Incomplete Grade (Grade of 'I') Policy:

The grade of 'I' will be assigned only under the following exceptional circumstances:

• The final exam is missed with an accepted excuse for the absence. In this case, the final exam must be made up during the first two weeks of the following semester.

ACADEMIC HONOR POLICY:
The Florida State University Academic Honor Policy outlines the University's expectations for the integrity of students' academic work, the procedures for resolving alleged violations of those expectations, and the rights and responsibilities of students and faculty members throughout the process.  Students are responsible for reading the Academic Honor Policy and for living up to their pledge to . . . be honest and truthful and . . . [to] strive for personal and institutional integrity at Florida State University.  (Florida State University Academic Honor Policy, found at http://fda.fsu.edu/Academics/Academic-Honor-Policy.)

AMERICANS WITH DISABILITIES ACT (ADA):

Students with disabilities needing academic accommodation should:
(1) register with and provide documentation to the Student Disability Resource Center; and
(2) bring a letter to the instructor indicating the need for accommodation and what type.  This should be done during the first week of class.

This syllabus and other class materials are available in alternative format upon request.

For more information about services available to FSU students with disabilities, contact the:

Student Disability Resource Center
874 Traditions Way
108 Student Services Building
Florida State University
Tallahassee, FL 32306-4167
(850) 644-9566 (voice)
(850) 644-8504 (TDD)
(850) 644-7164
sdrc@admin.fsu.edu
http://www.disabilitycenter.fsu.edu/

Academic Integrity:

Remember that the goal of programming assignments and homework is to enhance your analysis, reasoning, and programming skills. Indulging in academic dishonesty defeats this purpose apart from being unfair to other students. In case you have any questions about whether an act of collaboration may be construed as academic dishonesty, please clarify the issue with the instructor before you collaborate.

All students should follow FSU Academic Honor Code. You might be assigned a grade of 'F', if you are found to have indulged in academic dishonesty.

• It is understandable that discussing a problem with other people may lead to more insight into the issues involved. Thus discussing a problem in assignments/homeworks with other people is fine. However, discussing the solutions to the problem is NOT acceptable. 

• Every student must write his/her own code and homework. Showing your code or homework to members of other teams, giving it to them, or making it accessible to them (e.g., by making the files world-readable) is academic dishonesty.

• You are responsible for ensuring that your code/documentation/results are adequately protected and not accessible to other teams. Change permissions of your working directory to 0700 ('chmod 0700 <directory>).

• Consulting code/material from a textbook, or from the Internet, in order to understand specific aspects of your assignment is fine. However, copying such code/material will be considered academic dishonesty. If you borrow small parts of code/material from these sources, you must acknowledge this in your submission and additionally you must clearly understand and be able to explain the borrowed code/material.

Syllabus Changes

This syllabus is a guide for the course and is subject to change with advance notice.