Kernel Internals

What is the Linux Kernel?

  • The kernel is the core of the OS — it manages hardware, processes, memory, and I/O.
  • Monolithic kernel with loadable kernel modules (LKM) — best of both worlds.
  • Written in C (and some Assembly for architecture-specific code).
graph TD
  HW["🖥️ Hardware\nCPU · RAM · Disk · Network · GPU"]
  KM["Kernel Mode (Ring 0)\nFull hardware access"]
  UM["User Mode (Ring 3)\nRestricted — apps run here"]
  SC["System Call Interface\nsyscall / int 0x80"]
  HW --> KM
  KM --> SC
  SC --> UM

Kernel Subsystems

SubsystemResponsibility
Process SchedulerDecides which process runs on CPU and when
Memory ManagerVirtual memory, paging, swapping, OOM killer
VFS (Virtual File System)Unified interface for all filesystems
Network StackTCP/IP implementation, socket layer
Device DriversHardware abstraction (block, char, network)
IPCInter-process communication (pipes, signals, sockets, shared mem)
Security (LSM)Linux Security Modules — SELinux, AppArmor hooks

Kernel Version & Info

Kernel information commands
uname -r                        # kernel version (e.g. 6.5.0-45-generic)
uname -a                        # full info: kernel + arch + hostname
cat /proc/version               # kernel version + compiler
cat /proc/sys/kernel/osrelease  # release string
ls /boot/vmlinuz*               # installed kernels
dmesg | head -50                # kernel ring buffer (boot messages)
dmesg -T | grep -i error        # kernel errors with timestamps
journalctl -k                   # kernel messages via systemd

Kernel Modules (LKM)

Kernel module management
lsmod                           # list loaded modules
modinfo module_name             # module details (author, params, deps)
sudo modprobe module_name       # load module (with deps)
sudo modprobe -r module_name    # unload module
sudo insmod module.ko           # load specific .ko file
sudo rmmod module_name          # remove module
cat /proc/modules               # loaded modules (raw)
ls /lib/modules/$(uname -r)/    # available modules
depmod -a                       # rebuild module dependency map
# Blacklist a module (prevent loading):
echo "blacklist nouveau" | sudo tee /etc/modprobe.d/blacklist-nouveau.conf

Kernel Parameters (sysctl)

sysctl — kernel runtime parameters
sysctl -a                       # list all parameters
sysctl net.ipv4.ip_forward      # check IP forwarding
sudo sysctl -w net.ipv4.ip_forward=1  # enable IP forwarding (temp)
# Permanent: add to /etc/sysctl.conf or /etc/sysctl.d/99-custom.conf
sudo sysctl -p                  # reload sysctl config
ParameterPurposeRecommended Value
net.ipv4.ip_forwardEnable IP routing1 (router) / 0 (host)
net.ipv4.tcp_syncookiesSYN flood protection1
kernel.randomize_va_spaceASLR2 (full)
net.ipv4.conf.all.rp_filterReverse path filtering1
net.ipv4.conf.all.accept_redirectsICMP redirects0
kernel.dmesg_restrictRestrict dmesg to root1
vm.swappinessSwap aggressiveness10 (SSD)
fs.file-maxMax open file descriptors2097152

System Calls & Process Internals

What are System Calls?

  • The interface between user space and kernel space.
  • When a program needs kernel services (read file, create process, allocate memory), it makes a syscall.
  • Linux has ~350+ system calls defined in <sys/syscall.h>.

sequenceDiagram
  participant App as 🖥️ User App
  participant Lib as 📚 glibc (libc)
  participant SC as 🔀 Syscall Interface
  participant K as ⚙️ Kernel
  App->>Lib: printf() / fopen() / malloc()
  Lib->>SC: syscall(SYS_write, ...) / int 0x80
  SC->>K: Switches to Ring 0
  K->>SC: Executes + returns result
  SC->>Lib: Returns to Ring 3
  Lib->>App: Returns result

Common System Calls

SyscallNumberDescription
read0Read from file descriptor
write1Write to file descriptor
open2Open a file
close3Close a file descriptor
stat4Get file status
mmap9Map memory
brk12Change data segment size
fork57Create child process
execve59Execute a program
exit60Terminate process
socket41Create network socket
connect42Connect socket to address
clone56Create thread/process

Tracing System Calls

strace — trace system calls
strace ls                           # trace all syscalls of ls
strace -e trace=open,read ls        # trace specific syscalls
strace -p 1234                      # attach to running process
strace -o output.txt ls             # save to file
strace -c ls                        # count + summarize syscalls
strace -f ./program                 # follow child processes
ltrace ./program                    # trace library calls (libc)

Process Lifecycle

graph LR
   NEW["New\nProcess created\nfork() / clone()"]
   READY["Ready\nWaiting for CPU\nIn run queue"]
   RUN["Running\nOn CPU\nexecve()"]
   WAIT["Waiting\nBlocked on I/O\nor signal"]
   ZOMBIE["Zombie\nExited, parent\nnot called wait()"]
   DEAD["Dead\nParent called wait()\nPCB freed"]
   NEW --> READY --> RUN
   RUN --> WAIT --> READY
   RUN --> ZOMBIE --> DEAD

Process Information

Process inspection
ps aux                          # all processes
ps -ef --forest                 # process tree with parent info
pstree -p                       # visual tree with PIDs
cat /proc/1234/status           # process status (PID 1234)
cat /proc/1234/maps             # memory map of process
cat /proc/1234/fd/              # open file descriptors
cat /proc/1234/cmdline          # command line arguments
cat /proc/1234/environ          # environment variables
ls -la /proc/1234/exe           # executable path
lsof -p 1234                    # open files by PID
lsof -i :80                     # process using port 80

Signals

SignalNumberDefault ActionDescription
SIGHUP1TerminateHangup (reload config)
SIGINT2TerminateInterrupt (Ctrl+C)
SIGQUIT3Core dumpQuit (Ctrl+\)
SIGKILL9TerminateCannot be caught or ignored
SIGSEGV11Core dumpSegmentation fault
SIGTERM15TerminateGraceful termination (default kill)
SIGSTOP19StopCannot be caught or ignored
SIGCONT18ContinueResume stopped process
SIGUSR110TerminateUser-defined signal 1
SIGUSR212TerminateUser-defined signal 2
Signal commands
kill -l                         # list all signals
kill -15 1234                   # SIGTERM (graceful)
kill -9 1234                    # SIGKILL (force)
kill -HUP 1234                  # SIGHUP (reload)
killall -9 nginx                # kill all by name
pkill -f "python script.py"     # kill by pattern
trap 'echo "Caught SIGINT"' INT # trap signal in bash script

Memory Management

Virtual Memory Architecture

graph TD
   subgraph VM["Virtual Address Space (per process)"]
       K["Kernel Space\n(top, shared across all processes)"]
       S["Stack\n(grows downward)\nLocal vars, return addresses"]
       M["Memory-mapped files\nmmap() region"]
       H["Heap\n(grows upward)\nmalloc() / new"]
       BSS["BSS Segment\nUninitialized global vars"]
       DATA["Data Segment\nInitialized global vars"]
       TEXT["Text Segment\nProgram code (read-only)"]
   end

Memory Concepts

ConceptDescription
Virtual MemoryEach process has its own address space — isolated from others
Physical MemoryActual RAM — managed by kernel page allocator
PageFixed-size block (4KB default) — unit of memory management
Page TableMaps virtual addresses → physical addresses
TLBTranslation Lookaside Buffer — cache for page table lookups
SwapDisk space used when RAM is full — slow
OOM KillerKernel kills processes when RAM is exhausted
ASLRAddress Space Layout Randomization — randomizes memory layout
NX/XD bitNo-Execute bit — prevents code execution in data pages
SMEP/SMAPSupervisor Mode Execution/Access Prevention

Memory Commands

Memory analysis
free -h                         # RAM + swap usage
cat /proc/meminfo               # detailed memory info
vmstat -s                       # memory statistics
vmstat 1 5                      # virtual memory stats every 1s, 5 times
slabtop                         # kernel slab cache usage
cat /proc/buddyinfo             # buddy allocator info
# Per-process memory
cat /proc/1234/status | grep -i vm   # virtual memory stats
pmap 1234                       # memory map of process
pmap -x 1234                    # extended map with RSS
# Find memory hogs
ps aux --sort=-%mem | head -10

Memory Security Features

Check memory security features
# Check ASLR
cat /proc/sys/kernel/randomize_va_space
# 0=disabled  1=partial  2=full (recommended)
 
# Check if binary has security features
checksec --file=/usr/bin/ls     # requires checksec tool
# Or: readelf -l binary | grep GNU_STACK
# Or: hardening-check binary    # Debian/Ubuntu
 
# ASLR + PIE + Stack Canary + NX + RELRO

/proc Filesystem

/proc — virtual filesystem for kernel/process info
cat /proc/cpuinfo               # CPU details
cat /proc/meminfo               # memory details
cat /proc/loadavg               # load average (1/5/15 min)
cat /proc/uptime                # uptime in seconds
cat /proc/net/tcp               # TCP connections (hex)
cat /proc/net/if_inet6          # IPv6 interfaces
cat /proc/sys/                  # sysctl parameters
ls /proc/                       # numbered dirs = running PIDs
cat /proc/self/maps             # current process memory map

Linux Networking Deep Dive

Network Stack Architecture

graph TD
   APP["Application Layer\nsocket() · bind() · connect() · send() · recv()"]
   SOCK["Socket Layer\nAF_INET · AF_UNIX · AF_NETLINK"]
   TCP["Transport Layer\nTCP (reliable) · UDP (fast)"]
   IP["Network Layer\nIP routing · ICMP · Netfilter/iptables"]
   DRV["Device Driver\neth0 · wlan0 · lo"]
   HW["Network Hardware\nNIC · Wi-Fi card"]
   APP --> SOCK --> TCP --> IP --> DRV --> HW

Socket Programming Concepts

Socket TypeProtocolUse Case
SOCK_STREAMTCPReliable, ordered, connection-based
SOCK_DGRAMUDPFast, connectionless, no guarantee
SOCK_RAWRaw IPCustom protocol, packet crafting
AF_UNIXUnix domainIPC on same machine (faster than TCP)
AF_NETLINKNetlinkKernel ↔ userspace communication

Advanced Network Commands

Advanced networking tools
# ip — modern replacement for ifconfig/route
ip addr show                    # all interfaces + IPs
ip addr add 192.168.1.100/24 dev eth0
ip addr del 192.168.1.100/24 dev eth0
ip link set eth0 up/down
ip link set eth0 mtu 9000       # jumbo frames
ip route show                   # routing table
ip route add 10.0.0.0/8 via 192.168.1.1
ip route del 10.0.0.0/8
ip neigh show                   # ARP table
ip neigh flush dev eth0         # flush ARP cache
 
# ss — socket statistics (replaces netstat)
ss -tulnp                       # TCP+UDP listening + process
ss -s                           # socket summary
ss -tp                          # TCP connections + process
ss -o state established         # established connections
ss 'dport = :80'                # filter by destination port
 
# Network performance
iperf3 -s                       # start iperf server
iperf3 -c server_ip             # test bandwidth to server
iperf3 -c server_ip -u -b 100M  # UDP test at 100Mbps
mtr --report google.com         # traceroute + packet loss report

iptables / nftables (Firewall)

iptables — Linux firewall
# View rules
iptables -L -n -v               # list all rules
iptables -L INPUT -n -v         # INPUT chain only
iptables -t nat -L -n -v        # NAT table
 
# Basic rules
iptables -A INPUT -p tcp --dport 22 -j ACCEPT   # allow SSH
iptables -A INPUT -p tcp --dport 80 -j ACCEPT   # allow HTTP
iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
iptables -A INPUT -j DROP       # drop everything else
 
# NAT / Masquerading (router setup)
iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
echo 1 > /proc/sys/net/ipv4/ip_forward
 
# Port forwarding
iptables -t nat -A PREROUTING -p tcp --dport 80 -j REDIRECT --to-port 8080
 
# Save/restore rules
iptables-save > /etc/iptables/rules.v4
iptables-restore < /etc/iptables/rules.v4
 
# nftables (modern replacement)
nft list ruleset
nft add rule inet filter input tcp dport 22 accept

Network Namespaces (Containers/VMs)

Network namespaces
ip netns list                   # list namespaces
ip netns add myns               # create namespace
ip netns exec myns bash         # run shell in namespace
ip netns exec myns ip addr      # run command in namespace
ip link add veth0 type veth peer name veth1  # virtual ethernet pair
ip link set veth1 netns myns    # move veth1 to namespace
# This is how Docker creates isolated networks

TCP Deep Dive

sequenceDiagram
   participant C as Client
   participant S as Server
   Note over C,S: 3-Way Handshake (Connection Setup)
   C->>S: SYN (seq=x)
   S->>C: SYN-ACK (seq=y, ack=x+1)
   C->>S: ACK (ack=y+1)
   Note over C,S: Data Transfer
   C->>S: Data
   S->>C: ACK
   Note over C,S: 4-Way Teardown (Connection Close)
   C->>S: FIN
   S->>C: ACK
   S->>C: FIN
   C->>S: ACK
TCP StateDescription
LISTENServer waiting for connections
SYN_SENTClient sent SYN, waiting for SYN-ACK
SYN_RECEIVEDServer received SYN, sent SYN-ACK
ESTABLISHEDConnection active, data transfer
FIN_WAIT_1Sent FIN, waiting for ACK
TIME_WAITWaiting to ensure remote received ACK (2×MSL)
CLOSE_WAITReceived FIN, waiting for app to close

Filesystem Internals

Inode Architecture

graph LR
   FN["Filename\n(directory entry)"]
   IN["Inode\nMetadata: size, permissions\ntimestamps, owner, block pointers"]
   B1["Data Block 1"]
   B2["Data Block 2"]
   B3["Data Block N"]
   FN --> IN --> B1
   IN --> B2
   IN --> B3

Filesystem Types

FilesystemFeaturesUse Case
ext4Journaling, extents, 1EB maxDefault Linux FS
XFSHigh performance, parallel I/OLarge files, databases
BtrfsCopy-on-write, snapshots, RAIDModern workloads
ZFSEnterprise, checksums, dedupNAS, servers
tmpfsRAM-based, volatile/tmp, /run
overlayfsLayered FSDocker containers
NFSNetwork filesystemShared storage
FUSEUserspace filesystemCustom FS implementations

Filesystem Commands

Filesystem inspection and management
# Filesystem info
df -hT                          # disk usage + filesystem type
du -sh /* 2>/dev/null | sort -rh | head -20
lsblk -f                        # block devices + filesystems
blkid                           # UUIDs + filesystem types
tune2fs -l /dev/sda1            # ext4 filesystem info
xfs_info /dev/sda1              # XFS filesystem info
dumpe2fs /dev/sda1 | head -50   # ext4 superblock info
 
# Inode info
stat file.txt                   # inode, permissions, timestamps
ls -i file.txt                  # show inode number
df -i                           # inode usage per filesystem
find / -inum 12345              # find file by inode number
 
# Filesystem check + repair
sudo fsck -n /dev/sdb1          # check without fixing
sudo fsck -y /dev/sdb1          # auto-fix (unmounted only)
sudo e2fsck -f /dev/sdb1        # force check ext4
 
# Mount options
mount -o remount,ro /           # remount root read-only
mount -o noexec,nosuid /tmp     # security mount options
cat /proc/mounts                # currently mounted filesystems

Advanced File Operations

Advanced file operations
# Hard vs Symbolic links
ln file.txt hardlink            # hard link (same inode)
ln -s /path/to/file symlink     # symbolic link (different inode)
readlink -f symlink             # resolve symlink to real path
 
# File attributes (ext4)
lsattr file.txt                 # list attributes
chattr +i file.txt              # immutable (even root can't delete)
chattr -i file.txt              # remove immutable
chattr +a file.txt              # append-only
 
# Extended attributes
getfattr -d file.txt            # get extended attributes
setfattr -n user.comment -v "my note" file.txt
 
# Sparse files
dd if=/dev/zero of=sparse.img bs=1 count=0 seek=1G  # 1GB sparse file
ls -lh sparse.img               # shows 1GB
du -sh sparse.img               # shows actual disk usage (tiny)
 
# File descriptor limits
ulimit -n                       # current open file limit
ulimit -n 65536                 # increase for current session
cat /proc/sys/fs/file-max       # system-wide max

Performance Analysis & Tuning

Performance Monitoring Stack

graph TD
   APP["Application Metrics\nResponse time · Throughput · Error rate"]
   SYS["System Metrics\nCPU · Memory · Disk I/O · Network"]
   KERN["Kernel Metrics\nScheduler · Interrupts · Context switches"]
   HW["Hardware Metrics\nCPU cycles · Cache misses · Bus bandwidth"]
   APP --> SYS --> KERN --> HW

CPU Analysis

CPU performance analysis
top -d 1                        # real-time, refresh every 1s
htop                            # interactive process viewer
mpstat -P ALL 1                 # per-CPU stats every 1s
sar -u 1 10                     # CPU utilization 10 times
pidstat -u 1                    # per-process CPU usage
perf top                        # real-time CPU profiling
perf stat ./program             # CPU event counts for program
perf record ./program && perf report  # flame graph data
 
# Load average interpretation
# /proc/loadavg: 1min 5min 15min running/total lastpid
# Load = 1.0 on single core = 100% utilized
# Load > num_cores = overloaded
nproc                           # number of CPU cores
cat /proc/cpuinfo | grep "cpu cores" | head -1

Memory Analysis

Memory performance analysis
free -h                         # RAM + swap overview
vmstat 1 5                      # virtual memory stats
# vmstat columns: r=run queue, b=blocked, swpd=swap used
# si/so = swap in/out (bad if non-zero)
# bi/bo = block in/out (disk I/O)
# us/sy/id/wa = user/system/idle/wait
 
sar -r 1 5                      # memory stats
slabtop -s c                    # kernel slab cache (sorted by cache size)
cat /proc/slabinfo              # raw slab info
# Find memory leaks:
valgrind --leak-check=full ./program

Disk I/O Analysis

Disk I/O analysis
iostat -x 1                     # extended disk stats every 1s
# Key columns: %util (saturation), await (latency ms), r/s w/s (IOPS)
iotop -o                        # per-process I/O (only active)
iotop -a                        # accumulated I/O
lsof +D /path                   # processes with files open in dir
fio --name=test --rw=randread --bs=4k --size=1G  # disk benchmark
 
# Block device info
hdparm -I /dev/sda              # ATA device info
hdparm -tT /dev/sda             # disk read speed test
nvme smart-log /dev/nvme0       # NVMe health info
smartctl -a /dev/sda            # SMART disk health

Network Performance

Network performance analysis
sar -n DEV 1 5                  # network interface stats
nethogs                         # per-process bandwidth
iftop -i eth0                   # per-connection bandwidth
nload eth0                      # real-time bandwidth graph
ss -s                           # socket statistics summary
cat /proc/net/dev               # raw interface stats
ethtool eth0                    # NIC info + speed
ethtool -S eth0                 # NIC statistics (drops, errors)

System Profiling with perf

perf — Linux performance analysis
perf list                       # available events
perf stat -e cycles,instructions,cache-misses ./program
perf record -g ./program        # record with call graph
perf report                     # interactive report
perf top -e cache-misses        # real-time cache miss profiling
# Flame graphs (requires FlameGraph tool):
perf record -F 99 -g -p PID -- sleep 30
perf script | stackcollapse-perf.pl | flamegraph.pl > flame.svg

Linux Tuning for Production

Production server tuning
# /etc/sysctl.d/99-production.conf
/etc/sysctl.d/99-production.conf
# Network performance
net.core.somaxconn = 65535
net.core.netdev_max_backlog = 65535
net.ipv4.tcp_max_syn_backlog = 65535
net.ipv4.tcp_fin_timeout = 15
net.ipv4.tcp_keepalive_time = 300
net.ipv4.tcp_tw_reuse = 1
net.core.rmem_max = 134217728
net.core.wmem_max = 134217728
 
# Memory
vm.swappiness = 10
vm.dirty_ratio = 15
vm.dirty_background_ratio = 5
 
# File descriptors
fs.file-max = 2097152
fs.nr_open = 2097152
 
# Security
net.ipv4.tcp_syncookies = 1
kernel.randomize_va_space = 2
net.ipv4.conf.all.rp_filter = 1

Linux Security Hardening

Hardening Checklist

SSH Hardening

/etc/ssh/sshd_config — hardened configuration
Port 2222                           # change default port
Protocol 2                          # SSH2 only
PermitRootLogin no                  # never allow root login
PasswordAuthentication no           # key-based auth only
PubkeyAuthentication yes
AuthorizedKeysFile .ssh/authorized_keys
MaxAuthTries 3                      # limit auth attempts
LoginGraceTime 30                   # 30s to authenticate
ClientAliveInterval 300             # disconnect idle after 5min
ClientAliveCountMax 2
AllowUsers deploy admin             # whitelist users
X11Forwarding no                    # disable X11 forwarding
AllowTcpForwarding no               # disable TCP forwarding (if not needed)
Banner /etc/ssh/banner              # show legal warning banner

SELinux

SELinux — Mandatory Access Control
getenforce                      # Enforcing / Permissive / Disabled
sestatus                        # detailed SELinux status
setenforce 1                    # enable enforcing (temp)
setenforce 0                    # permissive mode (temp, for debugging)
# Permanent: edit /etc/selinux/config → SELINUX=enforcing
 
# Context management
ls -Z file.txt                  # show SELinux context
ps -eZ | grep nginx             # process context
chcon -t httpd_sys_content_t /var/www/html/  # change context
restorecon -Rv /var/www/html/   # restore default context
 
# Troubleshooting
ausearch -m avc -ts recent      # recent SELinux denials
audit2why < /var/log/audit/audit.log  # explain denials
audit2allow -a -M mypolicy      # generate policy from denials
semodule -i mypolicy.pp         # install policy module

AppArmor

AppArmor — Profile-based MAC
aa-status                       # show status + profiles
aa-enforce /etc/apparmor.d/usr.sbin.nginx   # enforce profile
aa-complain /etc/apparmor.d/usr.sbin.nginx  # complain mode
aa-disable /etc/apparmor.d/usr.sbin.nginx   # disable profile
apparmor_parser -r /etc/apparmor.d/usr.sbin.nginx  # reload
 
# Generate profile for new app
aa-genprof /usr/bin/myapp       # interactive profile generation
aa-logprof                      # update profile from logs

auditd — System Auditing

auditd — kernel audit framework
sudo systemctl enable --now auditd
auditctl -l                     # list active rules
auditctl -s                     # audit system status
 
# Add audit rules
auditctl -w /etc/passwd -p wa -k passwd_changes   # watch passwd file
auditctl -w /etc/sudoers -p wa -k sudoers_changes
auditctl -a always,exit -F arch=b64 -S execve -k exec_commands
auditctl -a always,exit -F arch=b64 -S open -F exit=-EACCES -k access_denied
 
# Persistent rules: /etc/audit/rules.d/audit.rules
 
# Search audit logs
ausearch -k passwd_changes      # search by key
ausearch -m USER_LOGIN -ts today  # today's logins
ausearch -ua 1000               # events by UID
aureport --summary              # summary report
aureport --auth                 # authentication report
aureport --failed               # failed events

File Integrity Monitoring (AIDE)

AIDE — Advanced Intrusion Detection Environment
sudo apt install aide -y
sudo aideinit                   # initialize database (takes time)
sudo cp /var/lib/aide/aide.db.new /var/lib/aide/aide.db
sudo aide --check               # check for changes
sudo aide --update              # update database after approved changes
 
# Schedule daily checks
echo "0 3 * * * root /usr/bin/aide --check | mail -s 'AIDE Report' admin@example.com" \
 | sudo tee /etc/cron.d/aide

CIS Benchmark Audit (Lynis)

Lynis — security auditing tool
sudo apt install lynis -y
sudo lynis audit system         # full system audit
sudo lynis audit system --quick # quick scan
# Results: /var/log/lynis.log
# Hardening index score + recommendations

Privilege Escalation Prevention

Check for common misconfigurations
# SUID binaries (should be minimal)
find / -perm -u=s -type f 2>/dev/null
 
# World-writable files (dangerous)
find / -perm -o+w -type f 2>/dev/null | grep -v /proc
 
# Cron jobs writable by non-root
ls -la /etc/cron* /var/spool/cron/
 
# Sudo permissions
sudo -l
 
# Capabilities (can replace SUID)
getcap -r / 2>/dev/null
 
# Writable /etc/passwd (critical)
ls -la /etc/passwd /etc/shadow
 
# PATH hijacking risk
echo $PATH | tr ':' '\n' | xargs ls -ld 2>/dev/null | grep -v "^d..x..x..x"

Advanced Shell & Scripting

Bash Advanced Features

Advanced bash scripting
# Process substitution
diff <(ls dir1) <(ls dir2)
while read line; do echo "$line"; done < <(command)
 
# Here documents
cat << 'EOF' > script.sh
#!/bin/bash
echo "This is a heredoc"
EOF
 
# Here strings
grep "pattern" <<< "string to search"
 
# Brace expansion
echo {a,b,c}_{1,2,3}            # a_1 a_2 a_3 b_1 ...
mkdir -p project/{src,tests,docs,build}
cp file.txt{,.bak}              # backup: file.txt → file.txt.bak
 
# Parameter expansion
${var:-default}                 # use default if var unset
${var:=default}                 # set and use default if unset
${var:?error message}           # error if unset
${#var}                         # length of var
${var#prefix}                   # remove shortest prefix
${var##prefix}                  # remove longest prefix
${var%suffix}                   # remove shortest suffix
${var%%suffix}                  # remove longest suffix
${var/old/new}                  # replace first occurrence
${var//old/new}                 # replace all occurrences
${var^^}                        # uppercase
${var,,}                        # lowercase
 
# Arrays
arr=(one two three)
echo "${arr[@]}"                # all elements
echo "${#arr[@]}"               # length
echo "${arr[@]:1:2}"            # slice (index 1, length 2)
arr+=("four")                   # append
unset arr[1]                    # delete element
 
# Associative arrays (bash 4+)
declare -A map
map[key]="value"
echo "${map[key]}"
echo "${!map[@]}"               # all keys

Advanced Text Processing

awk, sed, grep advanced
# awk — field processing
awk '{print $1, $3}' file                   # print fields 1 and 3
awk -F: '{print $1}' /etc/passwd            # custom delimiter
awk 'NR==5,NR==10' file                     # print lines 5-10
awk '/pattern/{print NR": "$0}' file        # print matching with line numbers
awk '{sum+=$1} END{print sum}' file         # sum column 1
awk '!seen[$0]++'                           # remove duplicate lines
awk '{print NF}' file                       # print field count per line
 
# sed — stream editor
sed 's/old/new/g' file                      # replace all
sed -i.bak 's/old/new/g' file               # in-place with backup
sed -n '5,10p' file                         # print lines 5-10
sed '/pattern/d' file                       # delete matching lines
sed '/pattern/a\new line after' file        # append after match
sed '/pattern/i\new line before' file       # insert before match
sed 's/^/  /' file                          # indent all lines
sed 's/[[:space:]]*$//' file                # trim trailing whitespace
 
# grep advanced
grep -P '\d{3}-\d{4}' file                 # Perl regex
grep -o 'pattern' file                     # print only match
grep -c 'pattern' file                     # count matches
grep -l 'pattern' *.txt                    # files containing match
grep -A 3 -B 3 'pattern' file              # 3 lines context
grep -E '(error|warning|critical)' log     # multiple patterns

Job Control & Background Processing

Job control
command &                       # run in background
jobs                            # list background jobs
fg %1                           # bring job 1 to foreground
bg %1                           # resume job 1 in background
Ctrl+Z                          # suspend current job
disown %1                       # detach from shell (survives logout)
nohup command &                 # immune to hangup signal
wait                            # wait for all background jobs
 
# GNU Screen
screen -S session_name          # new named session
screen -ls                      # list sessions
screen -r session_name          # reattach
# Inside screen: Ctrl+A d=detach  Ctrl+A c=new window  Ctrl+A n=next
 
# tmux
tmux new -s session_name        # new session
tmux ls                         # list sessions
tmux attach -t session_name     # attach
# Inside tmux: Ctrl+B d=detach  Ctrl+B c=new window  Ctrl+B %=split vertical

More Learn

Github & Webs

Master Playlists YouTube