Bash Utils

Essential Bash Utilities

grep "error" logfile.txt

cut -d',' -f1 data.csv

sed 's/old/new/g' file.txt

awk '{print $1}' file.txt

🔹 grep - Search Text

The grep command is a fundamental tool for pattern matching and text searching within files and streams. It scans input line-by-line, printing lines that match a specified regular expression or simple string. Essential for developers and sysadmins, grep is used to filter application logs for errors, search source code for functions or variables, and extract specific entries from large datasets. With options like -i for case-insensitive search, -v to invert matches, and -r for recursive directory searches, it's a versatile tool for data discovery and troubleshooting.

#!/bin/bash

# Basic search
grep "error" system.log

# Case-insensitive search
grep -i "warning" system.log

# Count matches
grep -c "success" system.log

# Show line numbers
grep -n "failed" system.log

# Recursive search in directory
grep -r "TODO" /project/src/

# Invert match (show non-matching lines)
grep -v "debug" system.log

error: connection failed
Warning: low memory
5
12: operation failed
src/main.sh:15:# TODO: implement feature
info: process started
info: process completed

🔹 cut - Extract Columns

The cut command extracts precise sections, or columns, from each line of text or structured data. It operates on delimited fields using the -f flag (e.g., cut -d',' -f1,3 for CSV) or on character positions with -c. This makes it ideal for quickly parsing tabular data from log files, CSV exports, or command output like ps or ls -l. While less flexible than awk for complex parsing, cut offers a lightweight and fast solution for straightforward column extraction tasks in shell scripts and data pipelines.

#!/bin/bash

# Sample data
echo "John,25,Engineer" > data.txt
echo "Jane,30,Designer" >> data.txt

# Extract first field (comma-delimited)
cut -d',' -f1 data.txt

# Extract multiple fields
cut -d',' -f1,3 data.txt

# Extract character range
echo "Hello World" | cut -c1-5

# Extract from specific position
cut -d':' -f1 /etc/passwd | head -3

John
Jane
John,Engineer
Jane,Designer
Hello
root
daemon
bin

🔹 sed - Stream Editor

The sed command performs non-interactive, stream-based text transformations, making it a powerhouse for automation. Its most common use is find-and-replace using the s/pattern/replacement/ syntax, which can edit files in-place with the -i flag. Beyond substitution, sed can delete lines (d), insert/appended text, and perform complex multi-step edits using scripts. It processes input line-by-line, applying rules sequentially, which is perfect for batch-editing configuration files, cleaning data files, or modifying code bases directly from the command line without opening an editor.

#!/bin/bash

# Replace text
echo "Hello World" | sed 's/World/Universe/'

# Replace all occurrences
echo "foo bar foo" | sed 's/foo/baz/g'

# Delete lines containing pattern
sed '/debug/d' logfile.txt

# Print specific lines
sed -n '1,3p' file.txt

# Insert text before line
sed '1i\Header Line' file.txt

# Replace in file (in-place)
sed -i 's/old/new/g' config.txt

Hello Universe
baz bar baz
error: connection failed
warning: timeout
Line 1
Line 2
Line 3
Header Line
Line 1

🔹 awk - Pattern Processing

awk is a complete text processing language designed for field-based data manipulation and reporting. It automatically splits each input line into fields (by default, whitespace-separated), accessible as $1, $2, etc. You can write rules consisting of patterns and actions, such as awk '$3 > 100 {print $1, $2}' to print specific columns based on a condition. It supports variables, arithmetic operations, arrays, and built-in functions, making it ideal for generating summaries, transforming structured text (like logs or CSV), and performing calculations on the fly within shell scripts.

#!/bin/bash

# Print first column
echo -e "John 25\nJane 30" | awk '{print $1}'

# Print with custom separator
echo -e "a:b:c\nd:e:f" | awk -F':' '{print $2}'

# Perform calculations
echo -e "10\n20\n30" | awk '{sum+=$1} END {print sum}'

# Conditional processing
awk '$2 > 25 {print $1}' ages.txt

# Format output
awk '{printf "Name: %s, Age: %d\n", $1, $2}' data.txt

# Multiple fields
ps aux | awk '{print $1, $2, $11}'

John
Jane
b
e
60
Jane
Name: John, Age: 25
Name: Jane, Age: 30
root 1 /sbin/init

🔹 sort and uniq

The sort and uniq commands are a classic duo for ordering data and deduplicating results in Unix pipelines. sort arranges lines in alphabetical or numerical order, with options like -n for numbers, -r for reverse, and -k for key-based sorting. The uniq command removes adjacent duplicate lines, often used after sorting. Together, they are essential for data cleanup: counting unique occurrences with sort file.txt | uniq -c, finding distinct values in logs, preparing data for further analysis, and organizing command output for better readability in reports and scripts.

#!/bin/bash

# Sort lines
echo -e "banana\napple\ncherry" | sort

# Numeric sort
echo -e "10\n2\n30" | sort -n

# Reverse sort
echo -e "a\nb\nc" | sort -r

# Remove duplicates
echo -e "apple\napple\nbanana" | sort | uniq

# Count occurrences
echo -e "red\nblue\nred\nred" | sort | uniq -c

# Show only duplicates
echo -e "a\na\nb\nc\nc" | sort | uniq -d

apple
banana
cherry
2
10
30
c
b
a
apple
banana
3 red
1 blue
a
c

🔹 wc - Word Count

The wc (word count) command provides quick statistics about text files or input streams. By default, it outputs three numbers: line count, word count, and byte count. Using flags like -l (lines only), -w (words only), or -m (characters) allows you to get specific metrics. It's invaluable for scripting: checking if a log file has new entries, validating the size of generated reports, monitoring data volume in pipelines, or ensuring document length meets requirements. Its simplicity and speed make it a staple for basic data validation and quick audits.

#!/bin/bash

# Count lines
echo -e "line1\nline2\nline3" | wc -l

# Count words
echo "Hello World from Bash" | wc -w

# Count characters
echo "Hello" | wc -c

# All counts
echo "Hello World" | wc

# Count files in directory
ls | wc -l

# Count specific pattern
grep "error" logfile.txt | wc -l

3
4
6
1 2 12
15
42

🔹 find - Search Files

The find command is a powerful utility for locating files and directories based on a wide array of criteria. You can search by name (-name), type (-type f for files, -d for directories), size (-size +10M), modification time (-mtime -7 for files modified in the last 7 days), and permissions. Its real power comes with the -exec option, which allows you to run commands (like chmod, rm, or mv) on each found item, enabling complex batch operations and automated file system management directly from the terminal.

#!/bin/bash

# Find by name
find /home -name "*.txt"

# Find by type
find . -type f  # files only
find . -type d  # directories only

# Find by size
find . -size +10M  # larger than 10MB

# Find and execute command
find . -name "*.log" -exec rm {} \;

# Find modified in last 7 days
find . -mtime -7

# Find with multiple conditions
find . -name "*.sh" -type f -executable

/home/user/notes.txt
/home/user/readme.txt
./file1.txt
./file2.txt
./docs
./src
./large_file.zip
./backup.log (deleted)
./recent_file.txt
./script.sh