Bash URL Transfer (curl)

Common curl Commands

curl https://example.com

curl -O https://site.com/file.zip

curl -X POST -d "data" url.com

curl -H "Key: Value" url.com

🔹 Basic curl Usage

The most fundamental curl command fetches and displays the content of a URL directly in your terminal. Simply type curl https://example.com, and the raw HTML or data response is printed to stdout. This default behavior is perfect for quick checks: verifying a website is up, inspecting the raw JSON response from a REST API endpoint, or downloading small snippets of data for testing. It’s the go-to tool for HTTP interactions from the command line, forming the basis for more complex web requests and data retrieval automation in scripts.

# Get webpage content
curl https://example.com

# Get JSON from API
curl https://api.github.com/users/github

🔹 Download Files with curl

To save content to a file, curl offers two primary options: -O and -o. The uppercase -O (remote) saves the file using its original name from the remote server, ideal for straightforward downloads like curl -O https://example.com/file.zip. The lowercase -o (output) lets you specify a custom local filename and path, providing full control over the download destination, such as curl -o archive.zip https://example.com/data. Understanding this distinction is key for organizing downloads and integrating curl into automated backup or deployment scripts.

# Download with original filename
curl -O https://example.com/file.pdf

# Download with custom name
curl -o myfile.pdf https://example.com/document.pdf

# Download multiple files
curl -O https://site.com/file1.zip -O https://site.com/file2.zip

🔹 Follow Redirects

Web servers often use HTTP redirects (status codes 301, 302), and curl needs explicit instruction to follow them. The -L (location) flag tells curl to automatically follow any redirects until it reaches the final destination resource. Without this flag, curl will simply output the initial redirect response (often just headers or a short HTML message) and stop, failing to retrieve the actual content you intended to fetch. This option is crucial for reliably accessing many modern websites, APIs, and shortened URLs that rely on redirection chains.

# Follow redirects automatically
curl -L https://bit.ly/shortlink

# Download following redirects
curl -L -O https://shortened-url.com/file

🔹 POST Requests with Data

Sending data to a server via a POST request is essential for interacting with web APIs and submitting forms. Use the combination -X POST to specify the method and -d 'your data' to include the request body. For example, curl -X POST -d 'key1=value1&key2=value2' https://api.example.com/submit sends form-encoded data. To send JSON, add a content-type header: curl -X POST -H "Content-Type: application/json" -d '{"name":"John"}' https://api.example.com/users. This capability makes curl a full-featured client for testing RESTful endpoints and automating data submissions.

# POST form data
curl -X POST -d "username=john&password=secret" https://api.example.com/login

# POST JSON data
curl -X POST -H "Content-Type: application/json" \
  -d '{"name":"John","age":30}' \
  https://api.example.com/users

🔹 Custom Headers

Custom HTTP headers are vital for API authentication, content negotiation, and sending metadata. Use the -H flag to add any header to your curl request. You can specify multiple headers by using -H repeatedly in a single command. Common use cases include setting the Authorization header with a Bearer token (-H "Authorization: Bearer YOUR_TOKEN"), defining the accepted content type (-H "Accept: application/json"), or specifying custom API keys. This flexibility allows curl to mimic browser requests and interact with virtually any modern web service that requires specific headers.

# Add authorization header
curl -H "Authorization: Bearer token123" https://api.example.com/data

# Multiple headers
curl -H "Content-Type: application/json" \
  -H "API-Key: abc123" \
  https://api.example.com/endpoint

🔹 Show Response Headers

Inspecting HTTP response headers is critical for debugging API calls and understanding server behavior. Curl provides two main options for this: -i includes both the response headers and the body in the output, giving you the full HTTP response. The -I flag (capital i) performs a HEAD request, fetching only the headers and no body at all, which is faster and useful for checking status codes, content types, caching directives, or cookies without downloading the entire resource. These options are indispensable for web development and API integration troubleshooting.

# Show headers with content
curl -i https://example.com

# Show only headers (HEAD request)
curl -I https://example.com

# Verbose output with detailed info
curl -v https://example.com

🔹 Authentication with curl

curl supports a wide range of authentication mechanisms required by secure web services and APIs. For Basic Authentication, use the -u username:password flag, which base64-encodes the credentials into the Authorization header. For token-based auth (like OAuth 2.0 Bearer tokens), you manually add the token as a custom header: -H "Authorization: Bearer YOUR_ACCESS_TOKEN". It also supports digest auth, NTLM, and can handle cookies for session-based authentication. This makes curl a versatile tool for accessing protected resources, testing login endpoints, and automating authenticated workflows.

# Basic authentication
curl -u username:password https://api.example.com/data

# Basic auth with prompt for password
curl -u username https://api.example.com/data

# Bearer token authentication
curl -H "Authorization: Bearer your_token_here" https://api.example.com/secure