Sunday, April 12, 2009

Automating FTP on the Mac

There is no shortage of GUI FTP programs, but kicking it old school on
the command line allows you to easily automate uploads and downloads.
The best part is, there is nothing to install. Everything you need
waits patiently behind the warm glow of a Terminal session.

The Mac command line FTP program
The default command line FTP program in OS X 10.5 resides at:
/usr/bin/ftp

By all outward appearances and behavior, the Mac FTP program is
the standard BSD version. The man page is the standard BSD
page and contains a wealth of useful information.
A typical command line FTP session is interactive and goes something
like this:

  • login to an FTP server

  • issue commands (ls (list), get (download), put (upload))

  • quit


If you have a repetitive FTP task, the fun quickly fades into a
mind numbing exercise. This is where FTP automation
shines.

The magical .netrc file
What makes FTP automation possible is a magical, little known file
called .netrc. The .netrc file is a plain text file that is
hidden (the file name starts with a period) and lives in the root of
your home directory. The .netrc file allows FTP to perform automatic
logins to FTP servers based on the name.

The .netrc is not created by default. You have to create it manually.
To create an empty .netrc file, open a Terminal and use the following
commands:

touch .netrc
chmod 700 .netrc


It is critical that you issue the chmod command to
set the permissions so that only the owner of the file can view it. If
the permissions are not set correctly, the FTP client will assume it has
been compromised and will refuse to use it.

Inside the .netrc, you define a block of settings for each FTP
server you use, including the machine name, the login ID and the
password. Here is what a typical block for a mythical FTP server:

machine myftpserver.com
login myuser
password mypassword


There are additional settings that can be included. Check the FTP man
page for more. You can test your settings by typing "ftp
myftpserver.com" at a Terminal prompt and it should automatically login.
Note that you can store multiple FTP server logins in the .netrc file.

Sending FTP commands from a BASH shell script
Once logins are automated, the final piece of the puzzle
is to script a set of FTP commands. The following example uses an
advanced BASH shell scripting technique called a "here" document
to group the FTP commands to be sent to the server.

#!/bin/bash
/usr/bin/ftp -d myftpserver.com << ftpEOF
prompt
put "*.html"
quit
ftpEOF


The FTP command is issued with the -d flag (debug mode) to make it more
verbose. That makes any kind of error more obvious. The connection is
made to myftpserver.com using the ID and password from the .netrc file.
Once the connection is made, the rest of the commands are issued one at
a time until the end of the "here" document at the second "ftpEOF". Note
that any valid FTP commands can be sent. In the example, the
prompt command tells FTP not to prompt for multiple file
operations, then the put uploads all files with an html
extension. If you want to go the extra mile, you can extend the shell
script and do things like reconnect to the FTP server to verify the file
sizes of your uploads.

While there are several ways you can automate FTP, the nice thing about
this method is that it is portable to Linux or any other Unix system.