Performance Monitoring Commands on Linux

• ps
• top
• vmstat
• free
• du
• lsof
• ss
• netstat

Process Snapshot: ps

$ ps aux
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.0  0.0   4056  2816 pts/0    Ss+  Apr18   0:00 /bin/bash /.exegol/entrypoint.sh load_setups desktop endless
root        26  0.0  0.0  16032 11796 ?        Ss   Apr18   0:00 /usr/bin/perl /usr/bin/vncserver -localhost yes -rfbport 54119 -geometry 1920x1080 -SecurityTypes Plain -PAMService tigervnc -PlainUsers root :1

$ ps -elf
F S UID        PID  PPID  C PRI  NI ADDR SZ WCHAN  STIME TTY          TIME CMD
4 S root         1     0  0  80   0 -  1014 wait_w Apr18 pts/0    00:00:00 /bin/bash /.exegol/entrypoint.sh load_setups desktop endless
5 S root        26     1  0  80   0 -  4008 hrtime Apr18 ?        00:00:00 /usr/bin/perl /usr/bin/vncserver -localhost yes -rfbport 54119 -geometry 1920x1080 -SecurityTypes Plain -PAMService tigervnc -PlainUsers root :1

Process Display: top

top is a highly popular and powerful tool in Linux and Unix systems for displaying a dynamic view of processes in the system in real-time. It is primarily used to monitor system resource usage, view running processes, and provide various statistical information about the system such as CPU utilization, memory usage, and system load.

top -u $USERNAME
top -o %CPU
top -d 2

Within the top command interface, you can use several interactive commands to control the display content or modify behavior:

• h: Display the help screen.
• k: Kill a process. Requires entering the process ID and signal.
• f: Select which fields to display.
• r: Renice a process to reset its priority.
• u: Display only processes belonging to a specific user.
• R: Toggle sort.
• P: Sort by CPU usage.
• M: Sort by memory usage.
• T: Sort by time/cumulative time.
• q: Quit top.

Sections of top Interface

The output of the top command is usually divided into several sections:

• Tasks and System Information Area: Displays system uptime, number of users, load averages, total number of processes, and number of running processes.
• CPU State Area: Shows various CPU state percentages, including user mode time, system mode time, and idle time.
• Memory State Area: Displays the usage of physical and swap memory.
• Process List: Shows a list of current active processes, typically including process ID, user, priority, virtual memory usage, physical memory usage, shared memory amount, status, CPU usage, and runtime.

CPU State Area

Memory State Area

Swap State Area

Process List

Virtual Memory Statistics: vmstat

vmstat is a tool to monitor and report information about system processes, memory, paging, block IO, traps, and CPU activity. It helps system administrators to diagnose performance issues and understand the behavior of their systems under different loads.

vmstat 1 # update the system statistics every second
vmstat 2 6 # every 2 seconds for 6 cycles

The vmstat command, short for "virtual memory statistics," is a powerful tool used in Unix-like operating systems to monitor and report information about system processes, memory, paging, block IO, traps, and CPU activity. It helps system administrators to diagnose performance issues and understand the behavior of their systems under different loads.

Basic Usage

To run vmstat, simply type vmstat in the terminal followed by the update interval in seconds. For instance, vmstat 1 will update the system statistics every second.

vmstat 1

Main Output Sections

The output of vmstat is divided into several columns, each providing specific types of information:

1. Procs (Processes)

   • r: The number of processes waiting for run time.
   • b: The number of processes in uninterruptible sleep.

2. Memory

   • swpd: The amount of virtual memory used.
   • free: The amount of idle memory.
   • buff: The amount of memory used as buffers.
   • cache: The amount of memory used as cache.

3. Swap

   • si: Amount of memory swapped in from disk per second.
   • so: Amount of memory swapped out to disk per second.

4. IO

   • bi: Blocks received from a block device (blocks/s).
   • bo: Blocks sent to a block device (blocks/s).

5. System

   • in: The number of interrupts per second, including the clock.
   • cs: The number of context switches per second.

6. CPU (These are percentages of total CPU time)

   • us: Time spent running non-kernel code. (user time, including nice time)
   • sy: Time spent running kernel code. (system time)
   • id: Time spent idle.
   • wa: Time waiting for IO.
   • st: Time stolen from a virtual machine.

Common Usage Scenarios

• Performance Monitoring: vmstat is often used to monitor system performance in real-time, particularly to check for issues like high system load, excessive swapping, and IO bottlenecks.
• Resource Tracking: It helps in tracking how processes are affecting the system's resources, giving insights into memory and CPU usage.
• Trend Analysis: By capturing output over time, system administrators can identify trends and patterns which could indicate potential problems or the need for system tuning.

Free & Used Memory Display: free

$ free -h
               total        used        free      shared  buff/cache   available
Mem:            15Gi       1.0Gi        14Gi        11Mi       697Mi        14Gi
Swap:          1.0Gi          0B       1.0Gi

The -v or --committed option in the free command is used to display information about the amount of committed memory and the commit limit in the system. This option is particularly useful for understanding how much memory is currently guaranteed to processes, which is a critical aspect of memory management in Unix-like operating systems.

Committed Memory

Committed memory refers to the total amount of memory that the kernel has promised to provide to applications or processes. When a process requests a certain amount of memory (via malloc or other memory management calls), the kernel commits to providing this memory, either immediately or eventually, when it is actually accessed.

Commit Limit

The commit limit is the maximum amount of memory that can be committed to all processes on the system without leading to overcommitment. Overcommitment occurs when the total amount of committed memory exceeds the actual available memory resources, including physical RAM and swap space. The kernel manages this limit to ensure that the system remains stable and to prevent it from crashing due to running out of memory.

The commit limit is calculated based on several factors, including the total size of the physical RAM and swap space. The kernel's overcommit policy can also influence this limit. There are usually three settings:

• Mode 0: Heuristic overcommit handling where the kernel guesses if sufficient memory is available.
• Mode 1: Always overcommit, where the kernel allows all memory allocations regardless of the current committed memory.
• Mode 2: Never overcommit, where the kernel does not allow allocations that exceed the commit limit.

Disk Free: df

$ df -h
Filesystem              Size  Used Avail Use% Mounted on
overlay                 917G   58G  813G   7% /
tmpfs                    64M     0   64M   0% /dev
shm                      64M     0   64M   0% /dev/shm
/dev/vda1               917G   58G  813G   7% /etc/hosts
tmpfs                   7.9G     0  7.9G   0% /proc/acpi
tmpfs                   7.9G     0  7.9G   0% /sys/firmware

$ df -hTi --total

Here are some of the most commonly used options with the df command:

• -h, --human-readable: Print sizes in human-readable format (e.g., 1K, 234M, 2G).
• -T, --print-type: Include the type of each filesystem in the output.
• -a, --all: Include dummy file systems in the output.
• --total: Produce a grand total of all disk usage found.
• -i, --inodes: List inode information instead of block usage.
• -k: List sizes in kilobytes (this is the default if no size option is provided).
• -m: List sizes in megabytes.
• --sync: Invoke sync before getting file system usage; this can ensure the data is up-to-date (useful for certain types of file systems).

The typical output of the df command includes several columns such as:

• Filesystem: The name of the filesystem/device.
• 1K-blocks/Size: Total size of the filesystem.
• Used: Space currently used within the filesystem.
• Available: Space still available within the filesystem.
• Use%: Percentage of the filesystem that is used.
• Mounted on: Directory on which the filesystem is mounted.

Disk Usage: du

du (disk usage) is a standard Unix and Linux utility that is used to estimate and report the disk space usage of files and directories.

• -a, --all: List the disk usage for each individual file, not just the totals for each directory.
• -h, --human-readable: Display the disk usage in a human-readable format (e.g., KB, MB, GB).
• -s, --summarize: Display only a total for each argument (file or directory).
• --max-depth=N: Limits the display of disk usage to N levels deep. Useful for not overwhelming the output with too much detail.
• -c, --total: Produce a grand total at the end of the output. This is useful for adding up the total disk usage of multiple directories or files.
• -k: Display sizes in kilobytes, which is useful for scripts that need consistent units.
• -m: Display sizes in megabytes.

du -sh /path/to/directory
du -sm /path/to/directory
du -h --max-depth=3 # List disk usage of all files and directories up to three levels deep
du -ch /dir1 /dir2 /dir3 # Calculate the total disk usage of several directories and display the grand total

List Open Files: lsof

The lsof is a tool to report a list of files that are opened by processes running on the system. Since everything in Unix is either a file or a process, lsof provides crucial insights into the system, helping in troubleshooting and system monitoring.

lsof [options] [names]

Here are some common options that enhance the utility of the lsof command:

• -u <username>: Lists files opened by the processes owned by the specified user.
• +D <directory>: Lists all files in the specified directory and its subdirectories.
• -i: Lists all network connections.
• -p <PID>: Lists files opened by the specified process ID.
• -c <command>: Lists files opened by processes that begin with the specified command name.

The typical output of lsof includes several columns of information:

• COMMAND: The command that has opened the file.
• PID: Process ID of the process that has opened the file.
• USER: User ID of the process owner.
• FD: File descriptor, which can be a number (0, 1, 2, etc.), or text like cwd (current working directory), txt (text files), mem (memory-mapped files).
• TYPE: The type of file (e.g., DIR, REG, CHR).
• DEVICE: Device numbers.
• SIZE/OFF: Size of the file or the file offset in bytes.
• NODE: Inode number.
• NAME: The name of the file or socket.

lsof -u username # Listing all open files by a specific user:
lsof +D /path/to/directory # Listing open files in a specific directory
lsof -p 1234 # Listing files opened by a specific process
lsof -i # Listing all network connections
lsof -i :80 # Finding which process is using a specific port

Network Information

lsof -i :<port>
ss -lnpt
netstat -tunlp # -anop