FTP File Operations with curl

There are many tools for managing FTP files via shell command line, but they all require additional installation. In fact, curl can satisfy most scenarios. This article organizes some common FTP operations (upload/download/delete) and includes examples for deleting old files.

List FTP Directory Files with curl

# Regular file list
curl -s -u username:password ftp://ftp.example.com/test/
# List with file timestamps
curl -s -u username:password ftp://ftp.example.com/test/ -X MLSD

Upload Files to FTP Directory with curl

# -T specifies the file to upload
# --ftp-create-dirs automatically creates non-existent directories
curl -u username:password ftp://ftp.example.com/test/ --ftp-create-dirs -T ./1.txt
# You can also specify the uploaded filename
curl -u username:password ftp://ftp.example.com/test/2.txt --ftp-create-dirs -T ./1.txt

Download Files from FTP with curl

# -o specifies the saved filename
curl -u username:password ftp://ftp.example.com/test/1.txt -o ./my.file
# PS: If the FTP directory is bound to a domain, you can also download via HTTP

Delete FTP Files with curl

# DELE followed by file path
curl -u username:password ftp://ftp.example.com -Q "DELE /test/1.txt"

Move or Rename FTP Files with curl

# RNFR represents the source file, RNTO represents the destination file
curl -u username:password ftp://ftp.example.com -Q "RNFR /test/1.txt" -Q "RNTO /1.txt.done"

Create or Delete Directories with curl

# Create directory (parent directory must exist)
curl -u username:password ftp://ftp.example.com/test/ -Q "MKD /test/aa"
# Delete directory (must be empty)
curl -u username:password ftp://ftp.example.com/test/ -Q "RMD /test/aa"

Advanced: Delete Old Files, Keep the Latest n Files

Example: Sort by file time, keep only the 5 newest files, delete all others. Can be saved as delete_old_ftp_files.sh for testing.

#!/bin/bash

# FTP Server Information
FTP_HOST=""
FTP_USER=""
FTP_PASS=""
# folder path must end with "/"
FTP_DIR="/tmp/"

# Number of files to keep
NUM_FILES_TO_KEEP=5

# Get a list of files in the FTP server
files=$(curl -s -u $FTP_USER:$FTP_PASS ftp://$FTP_HOST${FTP_DIR} -X MLSD | grep 'type=file')
# Sort by modify time
files=$(echo "$files" | cut -d';' -f3,8 | sort -k1)
# echo -e "${files}"
# get file name
files=$(echo "$files" | cut -d';' -f2)

# Find the number of files
num_files=$(echo "$files" | wc -l)

# Delete all files except the newest NUM_FILES_TO_KEEP
if [ $num_files -gt $NUM_FILES_TO_KEEP ]; then
  files_to_delete=$(echo "$files" | head -n $((num_files - NUM_FILES_TO_KEEP)))
  for file in $files_to_delete; do
    # echo "DELETE $file"
    curl -u $FTP_USER:$FTP_PASS -Q "DELE ${FTP_DIR}/$file" ftp://$FTP_HOST${FTP_DIR} > /dev/null 2>&1
  done
fi

Advanced: Delete Files Before a Certain Time

Example: Delete old files from 5 days ago or 2 hours ago.

#!/bin/bash

# FTP Server Information
FTP_HOST=""
FTP_USER=""
FTP_PASS=""
# folder path must end with "/"
FTP_DIR="/tmp/"

# compare_date=$(date -d '-5 days' +'%s') # 5 days ago
compare_date=$(date -d '-2 hours' +'%s') # 2 hours ago

# Get a list of files in the FTP server
files=$(curl -s -u $FTP_USER:$FTP_PASS ftp://$FTP_HOST${FTP_DIR} -X MLSD | grep 'type=file' | cut -d';' -f3,8)
# echo -e "${files}"

# echo $(date +'%Y-%m-%d %H:%M:%S') $(date +'%s')
while IFS= read -r line
do
  # echo "$line"
  file=$(echo "$line" | cut -d';' -f2)
  dt=$(echo "$line" | cut -d';' -f1 | cut -d'=' -f2)
  # -00:00 specifies the timezone of the FTP returned time
  datetime="${dt:0:8} ${dt:8:2}:${dt:10:2}:${dt:12:2} -00:00"
  timestamp=$(date -d "$datetime" +'%s')
  # echo $file, $dt, $(TZ=Asia/Shanghai date -d "@$timestamp" +'%FT%T %z'), $(TZ=Asia/Shanghai date -d "@$compare_date" +'%FT%T %z')
  if [[ "$timestamp" < "$compare_date" ]]; then
    # echo "DELETE $file"
    curl -u $FTP_USER:$FTP_PASS -Q "DELE ${FTP_DIR}/$file" ftp://$FTP_HOST${FTP_DIR} > /dev/null 2>&1
  fi
done < <(printf '%s\n' "$files")

Summary

For SFTP protocol, use sftp://ftp.example.com as the FTP URL.