Blog

Scenario: Compromised System Investigation

You are a cybersecurity analyst who has been alerted to suspicious activity on one of your Linux servers. A network monitoring system has flagged unusual outbound traffic from this server, and several users have reported performance issues and unauthorized access attempts. Your task is to investigate how the system was compromised and determine the extent of the breach.

Initial Clues:

1. Unusual Outbound Connections: Network logs show a spike in outbound connections to an unfamiliar IP address: 203.0.113.42.
2. New User Accounts: An IT staff member noticed a new user account named backupadmin that they did not create.
3. Process Anomalies: The server load has increased significantly, and there are several unfamiliar processes consuming a lot of CPU and memory.
4. File Changes: Some critical system files, including /etc/passwd and /var/log/auth.log, have recent modification timestamps.
5. Open Ports: A network scan reveals that a new port, 9090, is open and actively listening for connections.

These clues suggest a multi-faceted compromise, possibly involving a combination of unauthorized user account creation, malicious processes, file tampering, and communication with an external server.

---

Please proceed with your investigation by typing in Linux commands to uncover more details about the breach.

1. Unusual Outbound Connections: unfamiliar IP address: 203.0.113.42.

Network logs show a spike in outbound connections to an unfamiliar IP address: 203.0.113.42.

---

2. New User Accounts: backupadmin

An IT staff member noticed a new user account named backupadmin that they did not create.

To further investigate the system compromise, a good next step would be to examine the recent user accounts and any suspicious processes running on the system.

Check for recently added or modified user accounts:

cat /etc/passwd | grep 'backupadmin'

The command cat /etc/passwd | grep 'backupadmin' is used to search for the presence of a specific user account named backupadmin in the /etc/passwd file. Here’s a breakdown of the command:

1. cat /etc/passwd:
   • cat is a command that reads the contents of a file and outputs them to the terminal.
   • /etc/passwd is a system file on Unix and Linux systems that contains user account information. Each line in this file represents a user account and contains fields separated by colons, detailing information such as the username, user ID (UID), group ID (GID), home directory, and the shell used.
2. | (pipe):
   • The pipe symbol | is used to pass the output of one command as input to another command. In this case, it takes the output of cat /etc/passwd and passes it to the grep command.
3. grep 'backupadmin':
   • grep is a command-line utility used for searching plain-text data for lines that match a regular expression or pattern.
   • 'backupadmin' is the pattern that grep is searching for in the input it receives from cat /etc/passwd.

In summary, this command sequence reads the contents of the /etc/passwd file and searches for any lines that contain the string backupadmin. If such a line exists, it will be displayed in the terminal, indicating that a user account with the name backupadmin is present on the system.

Explanation of the result:

backupadmin:x:1001:1001::/home/backupadmin:/bin/bash

The result backupadmin:x:1001:1001::/home/backupadmin:/bin/bash is a line from the /etc/passwd file, which contains information about user accounts on the system. Here’s a breakdown of each field in this line:

1. Username (backupadmin): This is the name of the user account.
2. Password (x): Traditionally, the password field used to contain an encrypted password for the user account. However, the x in modern Unix-like systems indicates that the encrypted password is stored in the /etc/shadow file for security reasons.
3. User ID (1001): This is the unique numerical identifier assigned to the user account. User IDs below 1000 are typically reserved for system accounts.
4. Group ID (1001): This is the numerical identifier of the primary group to which the user belongs. In this case, the user belongs to a group with ID 1001.
5. User Info (empty): This field traditionally contains additional information about the user, such as full name or contact information. However, it is often left empty.
6. Home Directory (/home/backupadmin): This is the directory where the user’s files are stored. In this case, the home directory for backupadmin is /home/backupadmin.
7. Login Shell (/bin/bash): This is the default shell for the user, which determines the command-line interface used when the user logs in. /bin/bash is a common shell on Unix-like systems.

In summary, the result indicates that there is a user account named backupadmin on the system with a user ID and group ID of 1001. The home directory for this user is /home/backupadmin, and the default shell is /bin/bash.

How to remove this account?

To remove the backupadmin user account from the system, you can use the userdel command in Linux. Here’s how you can do it:

sudo userdel backupadmin

Explanation:

• sudo: This command is used to run the following command with superuser (administrator) privileges, which are required to delete a user account.
• userdel: This is the command used to delete a user account from the system.
• backupadmin: This is the username of the account you want to delete.

After running this command, the backupadmin user account will be removed from the system. Additionally, you may want to remove the home directory associated with the user account. By default, the userdel command doesn’t remove the home directory. To remove the home directory as well, you can use the -r option:

sudo userdel -r backupadmin

This command will delete the backupadmin user account along with its home directory and any files it contains. Make sure to double-check before executing this command to avoid unintended data loss.

---

3. Process Anomalies: unfamiliar processes consuming a lot of CPU and memory.

The server load has increased significantly, and there are several unfamiliar processes consuming a lot of CPU and memory.

The command

ps aux --sort=-%cpu

is used to display information about all running processes on a Unix-like operating system, sorted by their CPU usage in descending order. Here’s a breakdown of the command:

• ps: This is the process status command, which is used to display information about active processes.
• aux: These options are combined to provide detailed information about the processes:
  • a: Displays information about all users’ processes.
  • u: Provides a user-oriented format showing more detailed information, such as the user who owns the process, the CPU and memory usage, and the start time.
  • x: Includes processes that are not attached to a terminal, which is useful for finding daemon processes.
• --sort=-%cpu: This option sorts the output by the CPU usage percentage in descending order. The - sign indicates descending order, meaning processes using the most CPU will be listed at the top.

Example Output:

The output of this command typically includes the following columns:

• USER: The user who owns the process.
• PID: The process ID.
• %CPU: The percentage of CPU usage by the process.
• %MEM: The percentage of memory usage by the process.
• VSZ: The virtual memory size of the process.
• RSS: The resident set size (physical memory usage).
• TTY: The terminal associated with the process.
• STAT: The process status code.
• START: The time when the process started.
• TIME: The cumulative CPU time used by the process.
• COMMAND: The command that was used to start the process.

Result of executing: ps aux --sort=-%cpu

By using this command, you can quickly identify processes that are consuming the most CPU resources, which can be helpful in detecting performance issues or identifying potentially malicious processes.

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root      1324  45.6  2.3 228736 47612 ?        Ssl  12:05   5:32 /usr/bin/some-malicious-process
backupadmin 1480  35.3  1.9 198532 39524 ?      S    12:10   4:10 /usr/bin/suspicious-process
root       893  15.4  1.2 178432 25012 ?        S    11:59   3:45 /usr/sbin/sshd -D
nobody     763  12.3  1.0 156732 20512 ?        S    11:55   3:22 /usr/bin/httpd -k start
root       354  10.1  0.9 136732 18512 ?        S    11:50   2:50 /usr/sbin/rsyslogd -n

Analysis:

• /usr/bin/some-malicious-process and /usr/bin/suspicious-process are consuming significant CPU resources and seem suspicious, especially since one is running under the backupadmin account, which was flagged as unauthorized.
• /usr/sbin/sshd -D and /usr/bin/httpd -k start are standard processes for SSH and HTTP servers, respectively, but should be checked to ensure they are legitimate.
• /usr/sbin/rsyslogd -n is a system logging daemon, which is normal, but the presence of any unusual behavior or logs should be monitored.

Next Steps:

3.1. Investigate the suspicious processes further:

• Identify their purpose and origin.
• Check the file integrity and history.

ls -l /usr/bin/some-malicious-process /usr/bin/suspicious-process
file /usr/bin/some-malicious-process /usr/bin/suspicious-process

Result

ls -l /usr/bin/some-malicious-process /usr/bin/suspicious-process
-rwxr-xr-x 1 root root 105688 May 30 22:15 /usr/bin/some-malicious-process
-rwxr-xr-x 1 backupadmin backupadmin 85672 May 31 08:20 /usr/bin/suspicious-process

file /usr/bin/some-malicious-process /usr/bin/suspicious-process
/usr/bin/some-malicious-process: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 3.2.0, stripped
/usr/bin/suspicious-process:       ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 3.2.0, stripped

Analysis:

• File Permissions:
• Both files have executable permissions (-rwxr-xr-x).
• /usr/bin/some-malicious-process is owned by root.
• /usr/bin/suspicious-process is owned by the backupadmin user, which is already a suspicious account.
• File Type:
• Both files are ELF 64-bit executables for the x86-64 architecture.
• The binaries are stripped, meaning they do not contain debugging information, which is often done to make reverse engineering harder.

Next Steps:

1. Check if the files have been modified recently:

• Examine the file timestamps to see if they match the time of the compromise.

   stat /usr/bin/some-malicious-process /usr/bin/suspicious-process

2. Analyze the network connections of these processes:

• Check if they are communicating with the suspicious IP 203.0.113.42.

   netstat -plant | grep -E '1324|1480'

3. Review the command history for the backupadmin user:

• This can provide clues about how the files were introduced.

   sudo -u backupadmin history

4. Check for other instances of unauthorized changes:

• Review the /var/log/auth.log for entries related to these processes and the backupadmin account.

   tail -n 100 /var/log/auth.log

By executing these commands, you can gain further insight into how these malicious processes were introduced and potentially uncover additional signs of compromise.

3.2. Check the network connections:

• See if these processes are establishing outbound connections.

   netstat -plant | grep -E '1324|1480'

3.3.Review system logs for unusual activity:

• Look into /var/log/auth.log and other relevant logs.

   tail -n 100 /var/log/auth.log

3.4. Check the new user account backupadmin:

• Verify its creation date and permissions.

   cat /etc/passwd | grep backupadmin
   cat /etc/shadow | grep backupadmin

By executing these commands, you can gather more detailed information to understand the nature of the suspicious processes and the extent of the system compromise.

---