REST APIs power modern application communication, yet many developers and operators struggle to move beyond basic GET requests when debugging, testing, or automating workflows. This guide provides practical, production-ready command patterns for the most common REST API operations — authentication, resource manipulation, pagination, error handling, and rate limit management — using curl and httpie as primary tools. Every example uses explicit placeholders, includes verification steps, and documents failure signals with recovery paths so you can operate safely in staging and production environments.
Authentication and Authorization Patterns
Before making any state-changing requests, you must establish authenticated sessions. Most production APIs use one of three patterns: API keys, Bearer tokens (JWT/OAuth2), or mutual TLS. The examples below assume you have already obtained credentials through your provider's documented flow.
API Key in Header (Common for AWS, Stripe, SendGrid)
curl -X GET "https://api.example.com/v1/resources" \
-H "Authorization: Bearer ${API_KEY}" \
-H "Accept: application/json"
Verification: HTTP 200 with JSON array response. Failure signal: HTTP 401 with {"error": "invalid_api_key"} — rotate key in provider dashboard and update secret store.
Bearer Token with Automatic Refresh (OAuth2/OIDC)
# Initial token request
TOKEN_RESPONSE=$(curl -s -X POST "https://auth.example.com/oauth/token" \
-H "Content-Type: application/x-www-form-urlencoded" \
-d "grant_type=client_credentials&client_id=${CLIENT_ID}&client_secret=${CLIENT_SECRET}&scope=api.read api.write")
ACCESS_TOKEN=$(echo "${TOKEN_RESPONSE}" | jq -r '.access_token')
EXPIRES_IN=$(echo "${TOKEN_RESPONSE}" | jq -r '.expires_in')
# Use token with expiry tracking
curl -X GET "https://api.example.com/v1/resources" \
-H "Authorization: Bearer ${ACCESS_TOKEN}" \
-H "Accept: application/json"
Verification: Token decodes via jq -r '.access_token' | cut -d. -f2 | base64 -d | jq . showing exp claim > current time. Failure signal: HTTP 401 with WWW-Authenticate: Bearer error="invalid_token" — re-run token request, check clock skew, verify client secret not rotated.
Mutual TLS (mTLS) for Zero-Trust Environments
curl -X GET "https://api.internal.example.com/v1/sensitive-data" \
--cert /etc/certs/client.pem \
--key /etc/certs/client-key.pem \
--cacert /etc/certs/ca-chain.pem \
-H "Accept: application/json"
Verification: HTTP 200. Failure signal: SSL handshake failure (curl exit code 35/58) — verify certificate expiry with openssl x509 -in /etc/certs/client.pem -text -noout | grep "Not After", confirm CA chain matches server trust store.
Resource CRUD Operations with Idempotency
REST APIs map HTTP methods to CRUD semantics, but production safety requires understanding idempotency guarantees and conditional requests.
Create Resource (POST) — Non-Idempotent, Use Idempotency Keys
IDEMPOTENCY_KEY=$(uuidgen)
curl -X POST "https://api.example.com/v1/orders" \
-H "Authorization: Bearer ${ACCESS_TOKEN}" \
-H "Content-Type: application/json" \
-H "Idempotency-Key: ${IDEMPOTENCY_KEY}" \
-d '{
"customer_id": "cust_abc123",
"items": [{"sku": "SKU-001", "quantity": 2}],
"shipping_address": {"line1": "123 Main St", "city": "Boston", "state": "MA", "postal_code": "02101", "country": "US"}
}'
Verification: HTTP 201 with Location: /v1/orders/ord_xyz789 header and response body containing {"id": "ord_xyz789", "status": "pending"}. Failure signal: HTTP 409 with {"error": "idempotency_key_conflict"} — same key used with different payload; generate new key. HTTP 422 — validate required fields against API spec.
Read Resource (GET) — Safe, Cacheable, Use ETags
# First request captures ETag
RESPONSE=$(curl -s -D - -X GET "https://api.example.com/v1/orders/ord_xyz789" \
-H "Authorization: Bearer ${ACCESS_TOKEN}" \
-H "Accept: application/json")
ETAG=$(echo "${RESPONSE}" | grep -i "^etag:" | cut -d' ' -f2 | tr -d '\r')
# Subsequent request with conditional header
curl -X GET "https://api.example.com/v1/orders/ord_xyz789" \
-H "Authorization: Bearer ${ACCESS_TOKEN}" \
-H "Accept: application/json" \
-H "If-None-Match: ${ETAG}"
Verification: HTTP 200 on first request, HTTP 304 (Not Modified) on conditional request with empty body. Failure signal: HTTP 412 Precondition Failed — resource modified, fetch fresh copy without If-None-Match.
Update Resource (PATCH) — Prefer Over PUT for Partial Updates
curl -X PATCH "https://api.example.com/v1/orders/ord_xyz789" \
-H "Authorization: Bearer ${ACCESS_TOKEN}" \
-H "Content-Type: application/json" \
-H "If-Match: ${ETAG}" \
-d '{"status": "confirmed", "metadata": {"confirmed_by": "ops_user", "confirmed_at": "2024-01-15T10:30:00Z"}}'
Verification: HTTP 200 with updated resource representation. Failure signal: HTTP 412 — ETag mismatch, resource changed since read; re-fetch and retry. HTTP 409 — business logic conflict (e.g., cannot confirm cancelled order).
Delete Resource (DELETE) — Idempotent After First Success
curl -X DELETE "https://api.example.com/v1/orders/ord_xyz789" \
-H "Authorization: Bearer ${ACCESS_TOKEN}" \
-H "If-Match: ${ETAG}"
Verification: HTTP 204 No Content. Failure signal: HTTP 404 — already deleted (idempotent success). HTTP 409 — deletion blocked by dependent resources; check API docs for cascade/force parameters.
Pagination, Filtering, and Large Dataset Handling
Production APIs paginate results. Blindly fetching page 1 misses data; naive loops hit rate limits. Use cursor-based pagination where available (more stable than offset), and respect Retry-After headers.
Cursor-Based Pagination (Recommended for Large/Changing Datasets)
NEXT_CURSOR=""
while true; do
URL="https://api.example.com/v1/events?limit=100"
[ -n "${NEXT_CURSOR}" ] && URL="${URL}&cursor=${NEXT_CURSOR}"
RESPONSE=$(curl -s -X GET "${URL}" \
-H "Authorization: Bearer ${ACCESS_TOKEN}" \
-H "Accept: application/json")
# Extract and process items
echo "${RESPONSE}" | jq -c '.data[]' | while read -r item; do
echo "${item}" | jq -r '.id' # Replace with actual processing
done
# Check for next page
NEXT_CURSOR=$(echo "${RESPONSE}" | jq -r '.pagination.next_cursor // empty')
[ -z "${NEXT_CURSOR}" ] && break
# Respect rate limit — minimal delay
sleep 0.2
done
Verification: Loop terminates when next_cursor is null/empty. Failure signal: HTTP 429 with Retry-After: 45 — parse header, sleep $(echo "${RETRY_AFTER}" | tr -d '\r'), then retry same cursor.
Offset/Limit Pagination with Total Count (For Static Reports)
TOTAL=$(curl -s -X GET "https://api.example.com/v1/reports?limit=1" \
-H "Authorization: Bearer ${ACCESS_TOKEN}" | jq -r '.pagination.total')
PAGES=$(( (TOTAL + 99) / 100 ))
for ((page=1; page<=PAGES; page++)); do
curl -s -X GET "https://api.example.com/v1/reports?limit=100&offset=$(( (page-1)*100 ))" \
-H "Authorization: Bearer ${ACCESS_TOKEN}" \
-H "Accept: application/json" | jq -c '.data[]'
sleep 0.1
done
Verification: Aggregated item count matches total from first request. Failure signal: HTTP 5xx mid-loop — log failed offset, resume from that page after backoff.
Filtering and Search (Server-Side, Not Client-Side)
# Date range + status filter + field projection
curl -X GET "https://api.example.com/v1/orders?created_after=2024-01-01T00:00:00Z&created_before=2024-01-31T23:59:59Z&status=confirmed,shipped&fields=id,customer_id,total,status,created_at" \
-H "Authorization: Bearer ${ACCESS_TOKEN}" \
-H "Accept: application/json"
Verification: Response contains only requested fields, all items within date range and status set. Failure signal: HTTP 400 with {"error": "invalid_filter", "field": "created_after", "reason": "must_be_iso8601"} — correct format.
Error Handling, Rate Limits, and Observability
Robust API clients classify errors, implement exponential backoff with jitter, and emit structured logs for debugging.
Structured Error Response Handling
RESPONSE=$(curl -s -w "\n%{http_code}" -X POST "https://api.example.com/v1/webhooks" \
-H "Authorization: Bearer ${ACCESS_TOKEN}" \
-H "Content-Type: application/json" \
-d '{"url": "https://myapp.example.com/webhook", "events": ["order.created", "order.shipped"]}')
HTTP_CODE=$(echo "${RESPONSE}" | tail -n1)
BODY=$(echo "${RESPONSE}" | head -n -1)
case ${HTTP_CODE} in
201) echo "Webhook created: $(echo "${BODY}" | jq -r '.id')" ;;
400) echo "Validation error: $(echo "${BODY}" | jq -r '.errors | map(.field + ": " + .message) | join(", ")')" ;;
409) echo "Conflict: $(echo "${BODY}" | jq -r '.error.message')" ;;
422) echo "Unprocessable: $(echo "${BODY}" | jq -r '.error.details')" ;;
429) RETRY_AFTER=$(curl -s -I "https://api.example.com/v1/webhooks" -H "Authorization: Bearer ${ACCESS_TOKEN}" | grep -i "^retry-after:" | cut -d' ' -f2 | tr -d '\r')
echo "Rate limited. Retry after ${RETRY_AFTER:-60}s" ;;
5*) echo "Server error ${HTTP_CODE}. Body: ${BODY}" ;;
*) echo "Unexpected HTTP ${HTTP_CODE}: ${BODY}" ;;
esac
Verification: Case statement covers all documented error codes for the endpoint. Failure signal: Unhandled HTTP code — add to case statement, alert on-call.
Exponential Backoff with Jitter (Retry Logic for 429/5xx)
retry_with_backoff() {
local max_attempts=5
local base_delay=2
local attempt=1
while [ ${attempt} -le ${max_attempts} ]; do
RESPONSE=$(curl -s -w "\n%{http_code}" "$@")
HTTP_CODE=$(echo "${RESPONSE}" | tail -n1)
BODY=$(echo "${RESPONSE}" | head -n -1)
if [ ${HTTP_CODE} -eq 200 ] || [ ${HTTP_CODE} -eq 201 ] || [ ${HTTP_CODE} -eq 204 ]; then
echo "${BODY}"
return 0
elif [ ${HTTP_CODE} -eq 429 ] || [ ${HTTP_CODE} -ge 500 ]; then
# Extract Retry-After or calculate backoff
RETRY_AFTER=$(echo "${RESPONSE}" | grep -i "^retry-after:" | cut -d' ' -f2 | tr -d '\r')
if [ -n "${RETRY_AFTER}" ] && [ "${RETRY_AFTER}" -gt 0 ]; then
DELAY=${RETRY_AFTER}
else
DELAY=$(( base_delay * (2 ** (attempt - 1)) + RANDOM % 5 ))
fi
echo "Attempt ${attempt}/${max_attempts} failed (HTTP ${HTTP_CODE}). Retrying in ${DELAY}s..." >&2
sleep ${DELAY}
((attempt++))
else
echo "${BODY}" >&2
return 1
fi
done
echo "Max retries exceeded" >&2
return 1
}
# Usage
retry_with_backoff -X GET "https://api.example.com/v1/reports/large" \
-H "Authorization: Bearer ${ACCESS_TOKEN}" \
-H "Accept: application/json"
Verification: Function returns 0 on success, non-zero after max attempts. Failure signal: Max retries exceeded — alert with last HTTP code and body for manual investigation.
Request/Response Logging for Audit Trails
LOG_DIR="/var/log/api-client"
mkdir -p "${LOG_DIR}"
TIMESTAMP=$(date -u +"%Y-%m-%dT%H:%M:%SZ")
REQUEST_ID=$(uuidgen)
# Log request (sanitize secrets)
curl -X POST "https://api.example.com/v1/payments" \
-H "Authorization: Bearer ${ACCESS_TOKEN}" \
-H "Content-Type: application/json" \
-H "X-Request-ID: ${REQUEST_ID}" \
-d '{"amount": 9999, "currency": "USD", "payment_method_id": "pm_abc"}' \
-w "\nHTTP %{http_code} %{time_total}s" \
-o "${LOG_DIR}/${REQUEST_ID}.response.json" \
-D "${LOG_DIR}/${REQUEST_ID}.response.headers" \
2>"${LOG_DIR}/${REQUEST_ID}.curl.stderr" | tee "${LOG_DIR}/${REQUEST_ID}.request.log"
# Sanitize logs before retention
sed -i 's/"authorization": "Bearer [^"]*"/"authorization": "Bearer ***REDACTED***"/g' "${LOG_DIR}/${REQUEST_ID}.request.log"
Verification: Log files created with request ID correlation. Failure signal: Missing log files — check disk space, permissions, curl exit code.
Conclusion
Operating REST APIs safely in production requires more than knowing HTTP verbs — it demands disciplined authentication handling, idempotency awareness, pagination strategies that survive dataset changes, and error handling that distinguishes retryable from fatal conditions. The patterns in this guide — cursor-based pagination with rate-limit respect, ETag-protected mutations, exponential backoff with jitter, and structured request logging — form a foundation you can adapt to any REST API. Before automating against a new endpoint, spend fifteen minutes with the provider's OpenAPI spec: note authentication method, pagination style, rate limit headers, idempotency key support, and error code taxonomy. Then implement one read-only verification, confirm the response shape matches expectations, and only then layer in write operations with full observability. A reliable integration makes failure visible, protects credentials at every layer, limits each change to its intended resource, and defines recovery verification before an incident forces the decision.