Cosmic Consulting  www.cosmicconsulting.com

Downloads

SyncUpSite: Documentation from the POD

This documentation webpage was created by using pod2html on the file SyncUpSite.pl. To get this as a manpage when you have SyncUpSite.pl available, do perldoc SyncUpSite.pl.

NAME

 SyncUpSite - web site uploading utility

SYNOPSIS

SyncUpSite [options] sitename

SyncUpSite --usage | --help | --man | --version

DESCRIPTION

SyncUpSite is used to keep a remote FTP (web) site syncronized with a site contained on the local filesystem. When invoked it will examine the local site and compare it with its memory of the remote site. It will make changes on the remote site using FTP to make it identical to the local site. This is primarily intended to efficiently maintain a web site using FTP.

OPTIONS

--help
 Display a brief help message and then terminate.
--version
 Display a brief version message and then terminate.
--verbose n
Set the verbosity to an integer level (0..3). The default is 1. More information is printed to STDOUT when a higher number is used. A value of 0 quiets all but the error messages.

--catchup
When this option is used, no FTP will be done. The local site is examined and all local files and directories are recorded as having been uploaded. This causes the remote site to be considered as being up-to-date.

--catchupfile filepath
This is similar to the ``--catchup'' option except it is for a single local file and not the entire site. This is usually used to temporarily disable the uploading of a specific file. This feature can only be used if the file already exists in SyncUpSite's record of files on the remote site. If the file is new, an error will result.

--exclude filepath
The standard practice is to place exclude statements in the configuration file. This command line option allows you to specify an additional exclusion to the files that will be synchronized. This will have the same effect as adding an extra exclude statement in the configuration file with all of the side-effects. The excluded files and directories will be removed from the sync record. If the exclude operation is subsequently omitted, the now-included files and directories will be considered as brand new and needing updating.

--showchanges
When this option is used, no FTP will be done (except when using --getremote, see below). The actions needed to update the remote site will be described but the actions will not be taken nor recorded. The remote site will still be out-of-date if updates were described. This is used to preview the changes to the site and identify any unintentional changes before an upload is actually done.

--password password
You can specify the FTP password on the command line. This is useful when you do not want to store the password in the configuration file. The password provided here will be used instead of any value in the configuration file. Note that if a password is not provided through this option and is not in the configuration file, a password prompt will be presented as the console if FTP is required.

--configfile filename
The default location for the configuration file is in the user's home directory with the filename .SyncUpSiterc. With this option, the file can be specified. The only caveat to this option is that you have to remember where you put the configuration file. It is also not a good idea to have the same site appear in more than one configuration file.

This option is useful if multiple authors are collaborating on maintaining the same site. This avoids the issue of a configuration file in each user's home directory.

The --configfile option is usually used along with the optional keyword archivefile in the configuration file so that you can specify an appropriate location for the site archive file.

--ignoreperms
Since the configuration file can contain passwords, it is expected to have read/write permission only by the owner. SyncUpSite.pl will examine the configuration file before using it to determine if other users (besides root) can read or write the file and refuse to run if the file permissions are not restrictive enough. If you want to bypass this test, use the --ignoreperms flag. It is recommended that if you have loose permissions on the configuration file, do not store passwords in it and instead let SyncUpSite.pl prompt you for a password.

--getremote localdirectory
This option causes the entire remote site to be downloaded to the local directory specified instead of the normal behavior of uploading the modified files or directories. This operation will refuse to overwrite any files or folders already in this local directory. This feature is present here as a convenience when you are using SyncUpSite for the first time with an existing website. SyncUpSite will pay attention to the exclude statements in your configuration file and only download the correct files and directories. This activity is an all-or-nothing affair. It will not resume partial downloads. You can use this option in combination with --showchanges to find out what would happen if you used this command. An itemized list of files and directories will be provided including the sizes of each file and a total size of the download.

EXAMPLES

In the following examples, there is a website named www.fictionalsite.com with an FTP server ftp.fictionalsite.com. It has an entry in the SyncUpSite configuration file with the site name of fictionalsite. The local copy of the site is contained in /home/joeuser/public_html/fictionalsite as described in the configuration file.

The first time you use SyncUpSite with a new site, SyncUpSite will only know about the local files and assume that there are no remote files. If you have already synchronized the two sites prior to using update site, use the --catchup option to let update site know that the remote site is up-to-date. SyncUpSite will then take a snapshot of the files and directories in ~/public_html/fictionalsite for comparison to the next time you run SyncUpSite.

SyncUpSite.pl --catchup fictionalsite

You can now do your updates on your local files and when you are ready to upload the changed files to the remote site issue the following command.

SyncUpSite.pl fictionalsite

Only those files that have changed since the last time you ran SyncUpSite will be uploaded. Remote directories will be created as needed to match the local site. Any files or directories that have been deleted from the local site will be deleted from the remote site.

Unfortunately SyncUpSite cannot track name changes to local files and simply rename them on the remote site. SyncUpSite will upload the newly named file and delete the old file.

If no changes were found to the local site, no files will be uploaded and furthermore, no FTP connection will be made.

Note that this synchronization is one way. SyncUpSite does not look at files that might have changed on the remote server. It always assumes that the local copy is the master copy. If you edit your remote site from many different locations and it frequently has files more recent than the local copy then maybe SyncUpSite is not the tool you should be using.

If there are any errors during communication with the remote site, SyncUpSite will quit with an appropriate error message. All FTP transactions prior to the error were properly recorded as having been performed successfully. The next time you run SyncUpSite it will freshly analyze the remaining work that needs to be done and do it. This essentially means that it will resume working normally where it had stopped before and it will process any files/directories that had changed since the interuption.

If you would like to see what are the changes to the local site that need to be made on the remote site, use the following command.

SyncUpSite.pl --showchanges fictionalsite

SyncUpSite will then run through its normal activity except that the FTP connection will never be used and the remote site will not be marked as updated. This is a ``dry run'' mode to show you what would happen if you ran SyncUpSite normally.

This is a very useful mode since you can preview the changes and spot any spurious files that should not be uploaded or changes to files you did not realize occurred. You can forecast how much time it will take for the changes you see and schedule your network traffic accordingly.

SyncUpSite.pl --getremote fictionalsite

This will reverse the direction of the transfer. SyncUpSite will inventory the FTP site looking for those files and directories specified in the configuration file for the specified site. Then the files will be downloaded to your local computer. Directories will be created as needed. At present, SyncUpSite will fail with an error if any of the files or directories already exist. SyncUpSite assumes that you do not have a local copy of the website already. That's what this feature is supposed to do for you. Warning: This feature is not frequently used by the author and it might have bugs. Use with caution.

CONFIGURATION FILE

OVERVIEW

The configuration file is a necessary component to the function of SyncUpSite. All of the particulars of the FTP transaction are contained therein.

FORMAT

The configuration file contains lines of ASCII text as keyword/value pairs. The information for managing many different sites is contained in the file so each site's particular information is grouped together under the site keyword. On each line the keyword comes first followed by whitespace and then the value. Presently the value cannot contain whitespace. Blank lines are ignored. The hash symbol (#) is used to denote comments.

For items that specify a file or directory, you may specify a space in the filename or directory name by ``escaping'' it using a backslash. Thus the file ``my site folder'' would be written as ``my\ site\ folder'' (without the quotes). Of course, this would always be part of an absolute path as in ``/home/joeuser/public_html/my\ site\ folder''.

The following keywords (not case sensitive) are allowed:

site sitename
The site keyword is used to start a block of keywords specific to a particular FTP site's transaction (frequently a website). All the keywords following the site keyword and before any other site keyword will apply to that FTP site. Use the sitename on the command line to specify the site in the configuration file to use. The sitename can not contain spaces. (Required.)

There is a reserved sitename called default. It is recommended that this site be specified first in the configuration file. All specifications for the site named ``default'' will be the initial settings used for every other site. These settings may be over written by keywords in the site specific section. Lines in the configuration file that precede a site keyword are assumed to be default settings.

Not every configuration file keyword can appear in the default section. server, remote, local, user, password, and archivefile cannot be used in the default section.

server servername
This specifies a machine name or an IP address for the FTP server. (Required.)

remote remote directory
This specifies the directory on the remote server where the site will be uploaded. This may be a relative or an absolute path but it is usually a relative path. It is relative to the login directory of your FTP account on the server. If the site will be uploaded in the login directory of the server, use the value ``.'' (without the quotes). The directory name does not end in a slash. (Required.)

local local directory
This specifies the directory on the local filesystem where the local copy of the site resides. This must be an absolute path. The directory name does not end in a slash. (Required.)

user FTP username
This specifies the username to be used for the FTP transaction. (Required.)

password FTP password
This specifies the password to be used for the FTP transaction. If omitted, the password may be provided on the command line using the --password option or SyncUpSite will prompt for it on the command line. (Optional.)

exclude RegExp
To exclude a file or directory from any FTP transfer, provide a regular expression to match it using the exclude keyword. Any number of exclude statements can be provided, each on separate a separate line. Do not include the / / marks in the value of RegExp, just type what would go between the / /. This regular expression will match against the full path for the filename or directory. The regular expression syntax is that of the Perl that you are using for SyncUpSite itself. Note that since the value for the keyword exclude cannot have spaces, you should use \s in your regular expression to match whitespace. (Optional.)

port port number
If you are connecting to an FTP server that does not use the standard port number (21) then you can provide the port number with this keyword. Otherwise the connection will be made to port 21 as usual. (Optional.)

timeout timeout in seconds
The default value for the timeout of the FTP session is 120 seconds. If you desire a different value, you can specify that with this keyword. (Optional.)

blocksize size in bytes
The default value for the blocksize of the FTP session is 10240 bytes. If you desire a different value, you can specify that with this keyword. (Optional.)

hashmarks 1 or 0
When this keyword is set to 1, hash marks (#) will be printed to STDERR during the FTP transfer. One ``#'' will be printed for each 1024 bytes transferred. The default value is 0 for this keyword to produce no hash marks. (Optional.)

passive 1 or 0
When this keyword is set to 1, the FTP transaction will be made in Passive mode. With a value of 0, the FTP will be done Active. The default value is 1 (Passive) for this keyword. (Optional.)

archivefile filename
This keyword allows you to specify the archive file used to store the list of files and directories that comprise the remote site. The default behavior is to store this file in the directory '~/.SyncUpSite' and use the sitename as the filename. With this keyword you can store this archive file anywhere you desire. Be sure to provide a full, absolute path when giving the filename.

xferspeed connection speed in bytes/sec
In order for SyncUpSite to provide you with some predictions about how long it will take to transfer files, you can provide the expected transfer speed. Whenever a fie size is displayed, an estimate of the transfer time will be displayed as well if this keyword is used. (Optional.)

CONFIGURATION EXAMPLES

ENVIRONMENT

HOME
Used to determine the user's home directory where the .SyncUpSiterc file and the .SyncUpSite/ directory are found.

FILES

.SyncUpSiterc in the HOME directory is used for configuration details. .SyncUpSite/ in this directory is used to store the site archives which are lists of files and directories (and their modification times) that composed the local sites. The .SyncUpSiterc file can be redefined with the --configfile option. The .SyncUpSite/ directory can be redefined with the archivefile keyword in the configuration file.

CAVEATS

Take care when you are editing files on the remote site from multiple locations. SyncUpSite does not examine what is on the remote site. It only examines its recollection of what it has done with the remote site.

BUGS

None known so far.

AUTHOR

Tom Hays <trhays@cosmicconsulting.com> - main author

HISTORY

SyncUpSite was created primarily because sitecopy did not compile out-of-the-box for Mac OSX (10.2). FTPsync (Christoph Lechleitner <ftpsync@ibcl.at>) was available but solved a different problem. FTPsync specialized in two-way site synchronization. I was looking for the one-way approach with the local site always being the master and optimizing for low network traffic. Instead of getting sitecopy to work (which coincidentally was done a week later by the Fink team), Tom Hays decided to write his own utility using FTPsync as the basis for the code.

LICENSE

    SyncUpSite.pl is free software; you can redistribute it and/or modify
    it under the terms of the GNU General Public License as published by
    the Free Software Foundation; either version 2 of the License, or
    (at your option) any later version.
    This program is distributed in the hope that it will be useful,
    but WITHOUT ANY WARRANTY; without even the implied warranty of
    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
    GNU General Public License for more details.
    You should have received a copy of the GNU General Public License
    along with this program; if not, write to the Free Software
    Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA