There are many solutions to copy files between 2 unix servers, such as :
- Copying the files from unix server to your local machine, and then upload the files to the destination server.
- Using NFS to share the files from your source server to your destination server.
- Using FTP to copy the files.
First solution is recommended if your files are not much and small in sizes. But if your files are big, then you must provide big storage in your local machine. Using NFS can be an alternate solution, it’s easy to set up, easy to mount the directory and easy to copy the files. But sometimes there are some issues with the NFS, so you must be careful in using NFS.
The last solution is using FTP. Maybe you are wondering “How can I copy between 2 unix servers using FTP ? Do I have to copy it to local machine first ? “. Actually you don’t have to, because there is a technology called ‘XFTP’. They said that you can open 2 Unix Server in different window and copy between that two machines. But I haven’t tested that yet.
Another way is using “self made script” in which I don’t create it myself, I copied it from here. The script allows you to connect to a ftp server, get all the the content of that ftp server and it’s all in terminal ( no GUI, no drag and drop, just bash command ). The script also provides you with several options ( you can get the content of specific folder in the ftp server ).
Here is the complete script in case the website is down :
#! /usr/bin/ksh
#
# HardFeed -- Harvest A Remote Directory via
# Ftp Enhanced Exhaustive Duplication
#
# Perderabo 11-23-02
VERSION="1.1" # 03-16-04
USAGE="\
HardFeed [ -v | -s | -d | -r | -f | -m | -p password-file
-l list-command | -x ftp-command ... ] system user [directory]
use \"HardFeed -h\" use for more documentation
-v (verbose) Print stuff during run
-s (symlink) Attempt to duplicate any remote symlinks
-d (directory)Attempt to duplicate any remote directories
-r (recurse) Attempt to descend into any directories and process them
-f (freshen) If remote file is newer than local file, overwrite the local
-m (mode) Attempt a chmod on files and directories that we create
-p (password) Specify a file that contains the password in plaintext
-x (extra) Specify a command to be sent to the ftp client
-l (listcmd) Override the choice of \"ls .\" to get a remote directory"
DOCUMENTATION="HardFeed Version $VERSION
$USAGE
HardFeed copies all of the contents of a remote directory to the current
directory using ftp. It establishes an ftp connection to the remote
site and it uses the ftp command \"ls\" to get a listing of the remote
directory. The two required parameters are the remote sytstem and the user
name. The optional third parameter is the remote directory to copy. The
default is just the home directory of the ftp account.
HardFeed will prompt you for the password. This is very secure but it isn't
any good if you want to run HardFeed automatically. You can set the password in
the environment variable HARDFEED_P as an alternate. HardFeed will set an
internal variable to the password and then clobber the variable HARDFEED_P,
since on some systems, the environment of another process can be displayed.
With most shells, you can also set an environment variable for one command
only, like this: \"HARDFEED_P=xyzzy HardFeed -dR ftpxy joeblow sourcedir\".
A second alternative is to specify a \"password file\" with the -p option.
Such a file contains, in plaintext, the password. HardFeed will read the file
to get the password. You must decide which option makes more sense in your
environment.
Only files are examined. If we don't have a copy of the remote file, we
will get it. HardFeed will never overwrite an existing file system object
with one exception. If you specify -f and we have both a remote file and a
local file, the timestamps are compared. If the remote file is newer, a
retrieval attempt will be made. The local file must be writable for this
to succeed. For the timestamp compare to work, you and the remote system
must be in the same timezone. (You can vary your environment to make this
true.)
Normally symbolic links are ignored. But with -s, we will attempt to create
a symlink with the same link data. Even with -s, we will never overwrite
any existing object with a new symbolic link. You will need to review any
symlinks created and probably correct them.
Normally, directories are ignored. If you specify -d, HardFeed will attempt
to create the directory locally. But again, it will never overwrite an
existing object to create a directory. If you specify -r, HardFeed will
attempt to recurse into a directory and process all of the files there. If
you use both -d and -r, it will copy an entire directory hierarchy. But you
can leave off -d and only pre-create a few directories if you want.
HardFeed will attempt a chmod of any file or directory that it creates if you
specify -m. It will try to match the mode of the remote object.
HardFeed operates by establishing a co-process to the ftp command. Normally,
the output from the co-process is sent to an un-named file in /tmp and
discarded. If you want to capture this output, connect a file to fd 3 and
HardFeed will use it for this purpose. From ksh the syntax is 3>file. You
can also do 3>&1 to see it real time during the run if you really want.
You can make HardFeed send the ftp co-process some extra commands after the
connection is established with -x.
HardFeed gets a directory listing by sending a \"ls .\" command to the server.
Some servers will list dot files with this while others won't. You can use the
-l option to change the command if your server needs a different one to do want
you want. -l \"ls -al\" is one example that I got to work with unix.
For a microsoft ftp server, I had some luck with:
-l \"ls -la\" -x \"quote site dirstyle\"
Note that everything is transferred in binary mode. -x ascii will switch
everything to ascii mode. HardFeed supports embedded spaces in filenames. User
names may be long and contain slashes. All of this may make it somewhat usable
with microsoft ftp servers."
IFS=""
#
# If the password is coming in via the environment, save it in
# a local variable and then clobber the environment variable
unset PASSWORD
if [[ -n $HARDFEED_P ]] ; then
PASSWORD="$HARDFEED-P"
HARDFEED_P='********'
fi
#
# Parse Command Line
#
set -A OPT_CMDS_LIST
OPT_DIRCMD="ls ."
OPT_VERBOSE=0
OPT_SYMLINKS=0
OPT_DIRECTORIES=0
OPT_RECURS=0
OPT_FRESHEN=0
OPT_MODE=0
OPT_PASSWORDFILE=""
OPT_CMDS=0
error=0
while getopts :vsdrfmhp:x:l: o ; do
case $o in
v) OPT_VERBOSE=1
;;
s) OPT_SYMLINKS=1
;;
d) OPT_DIRECTORIES=1
;;
r) OPT_RECURS=1
;;
f) OPT_FRESHEN=1
;;
m) OPT_MODE=1
;;
h) echo "$DOCUMENTATION"
exit 0
;;
p) OPT_PASSWORDFILE=$OPTARG
if [[ ! -f $OPT_PASSWORDFILE ]] ; then
echo error $OPT_PASSWORDFILE is not a file
error=1
fi
;;
x) OPT_CMDS_LIST[OPT_CMDS]="$OPTARG"
((OPT_CMDS=OPT_CMDS+1))
;;
l) OPT_DIRCMD="$OPTARG"
;;
?) print error argument $OPTARG is illegal
error=1
;;
esac
done
shift OPTIND-1
if ((error)) ; then
echo "$USAGE"
exit 1
fi
if [[ $# -ne 2 && $# -ne 3 ]] ; then
echo "$USAGE"
exit 1
fi
SYSTEM=$1
USER=$2
DIRECTORY=$3
[[ -z $DIRECTORY ]] && DIRECTORY=.
#
# Read password file if one is supplied
if [[ -n $OPT_PASSWORDFILE ]] ; then
read PASSWORD < $OPT_PASSWORDFILE
fi
#
# Request password if it didn't come in via env or file
if [[ -z $PASSWORD ]] ; then
print -n password -
stty -echo
read PASSWORD
echo
stty echo
fi
#
# FD 3 will be the transcript of the ftp co-process. If the user
# supplied a file for this, we will use that. Otherwise it will go
# to a nameless file in /tmp
if print -u3 " Transcript of the ftp co-process for HardFeed" 2>/dev/null ; then
LOGFILE=""
else
LOGFILE=/tmp/HardFeed.log.$$
exec 3>$LOGFILE
rm $LOGFILE
fi
#
# Max time to wait for arrivial of file. This is a long time. During
# an interactive run, the user can use SIGINT if it seems to be taking
# too long. This max is intended to assure that a cron job will not
# hang forever.
OPT_MAXWAIT=15
TIMEOUT=/tmp/HardFeed.timeout.$$
#
# Various other initializations
LEV=0
date "+%Y %m" | IFS=" " read THISYEAR THISMONTH
((LASTYEAR=THISYEAR-1))
STARTPATH=$(pwd)
set -A DIR_FILE_NAME
set -A DIR_LINE_NUM
#
# Function to convert month to numeric
conv_month() {
typeset -l month
month=$1
case $month in
jan) nmonth=1 ;;
feb) nmonth=2 ;;
mar) nmonth=3 ;;
apr) nmonth=4 ;;
may) nmonth=5 ;;
jun) nmonth=6 ;;
jul) nmonth=7 ;;
aug) nmonth=8 ;;
sep) nmonth=9 ;;
oct) nmonth=10 ;;
nov) nmonth=11 ;;
dec) nmonth=12 ;;
*) nmonth=0 ;;
esac
echo $nmonth
return $((!nmonth))
}
#
# Function to determine if a file system object exists
#
# neither -a nor -e is really portable 8(
exists() {
[[ -f $1 || -d $1 || -L $1 || -p $1 || -S $1 || -b $1 || -c $1 ]]
return $?
}
#
# Function to wait for a file to arrive
waitfor() {
wanted=$1
if ((OPT_MAXWAIT)) ; then
((GIVEUP=SECONDS+OPT_MAXWAIT))
else
GIVEUP="-1"
fi
while [[ ! -f $wanted && $SECONDS -lt $GIVEUP ]] ; do
sleep 1
done
if [[ ! -f $wanted ]] ; then
echo "FATAL ERROR:" timed out waiting for: 2>&1
echo " " "$wanted" 2>&1
echo
print -p bye 2>/dev/null
exit 2
fi
return 0
}
#
# Function to decode an "ls -l" line.
lsdcode() {
typeset -Z2 nmonth day
typeset -i8 octal
#
# get the line, get the first character, split line into words
line="$1"
char1=${line%%${line#?}}
IFS=" "
set -A things -- $line
IFS=""
#
# We may have a "total" line which needs to be ignored
if [[ ${things[0]} = total ]] ; then
set -A lsdc -- skip 000 000000000000 x x
return 0
fi
#
#
parser=1
month=${things[5]}
xmonth=$(conv_month $month)
if conv_month $month > /dev/null ; then
parser=1
else
parser=0
fi
if ((parser)); then
#
# Strict Left to Right Parse Routine
#
# Break out the fields that we want. This technique requires
# that the user, group, and size fields never run together and
# so they must have at least one space between them. But it
# allows some limited support of filenames with embedded spaces.
echo "$line" | IFS=" " read permstring junk junk junk junk \
month day swing rawname
if [[ $char1 = l ]] ; then
link=${rawname#*-\> }
name=${rawname% -\>*}
else
name="$rawname"
link=""
fi
else
#
# Outside to Inside Parse Routine
#
# Break out the fields that we want. This technique requires
# that no white space exist in the filename. But the user,
# group, and size fields may sometimes run together without
# causing a problem.
echo "WARNING:" badly formatted line in directory listing for: >&2
echo " " "${line}" >&2
echo " " attempting outside-to-inside scan >&2
echo >&2
((pname=${#things[*]}-1))
if [[ $char1 = l ]] ; then
link=${things[pname]}
((pname=pname-2))
else
link=
fi
permstring=${things[0]}
name=${things[pname]}
month=${things[pname-3]}
day=${things[pname-2]}
swing=${things[pname-1]}
if conv_month $month > /dev/null ; then
:
else
echo "ERROR: " outside-to-inside scan has also failed >&2
echo " " giving up on: >&2
echo " " "$line" >&2
echo >&2
set -A lsdc -- skip 000 000000000000 x x
return 0
fi
fi
#
# Ignore . and ..
if [[ $name = . || $name = .. ]] ; then
set -A lsdc -- skip 000 000000000000 x x
return 0
fi
#
# decode permissions (the permission string is first word
set -A perms -- $(print -- ${permstring#?} | sed 's/./& /g')
extras=0
[[ ${perms[2]} = S ]] && { ((extras=extras+4000)); perms[2]=- ; }
[[ ${perms[2]} = s ]] && { ((extras=extras+4000)); perms[2]=x ; }
[[ ${perms[5]} = S ]] && { ((extras=extras+2000)); perms[5]=- ; }
[[ ${perms[5]} = s ]] && { ((extras=extras+2000)); perms[5]=x ; }
[[ ${perms[8]} = T ]] && { ((extras=extras+1000)); perms[8]=- ; }
[[ ${perms[8]} = t ]] && { ((extras=extras+1000)); perms[8]=x ; }
binary=2#$(print -- ${perms[@]} | sed 's/ //g;s/-/0/g;s/[^0]/1/g')
((octal=binary))
result=$(echo $octal)
result=${result#??}
((result=result+extras))
#
# Decode date and time and convert it to yyyymmddhhmm
nmonth=$(conv_month $month)
if [[ $swing = *:* ]] ; then
if [[ $nmonth > $THISMONTH ]] ; then
((year=LASTYEAR))
else
((year=THISYEAR))
time1=${swing%???}
time2=${swing#???}
time="${time1}${time2}"
fi
else
year=$swing
time="0000"
fi
#
# Output the final record
set -A lsdc -- ${char1} ${result} ${year}${nmonth}${day}${time} ${name} ${link}
return
}
#
# Function to process a remote file
# We will not overwrite and existing file unless we in "freshen" mode.
# And unless we are in "freshen" mode, it is an error for a file to
# pre-exist.
process_remote_file() {
VMESS="${VMESS} is a remote file that"
do_get=0
if [[ -f $name ]] ; then
VMESS="${VMESS} already exists"
if ((OPT_FRESHEN)) ; then
line2=$(ls -ld "$name")
lsdcode "$line2"
char12=${lsdc[0]}
mode2=${lsdc[1]}
datestamp2=${lsdc[2]}
name2=${lsdc[3]}
link2=${lsdc[4]}
if [[ $datestamp > $datestamp2 ]] ; then
VMESS="${VMESS} but is out-of-date and"
do_get=1
else
VMESS="${VMESS} and is current"
fi
else
VMESS="${VMESS} and cannot be retrieved"
echo WARNING: no get since $name exists in ${localpath} >&2
fi
else
do_get=1
fi
if ((do_get)) ; then
print -p get \""$name"\"
waitfor $name
VMESS="${VMESS} has been retrieved"
if ((OPT_MODE)) ; then
chmod $mode "$name"
fi
fi
if (($OPT_VERBOSE)) ; then
echo "$VMESS"
fi
return 0
}
# Function to process a remote directory
# To this function, a remote directory is just an object that
# may need to be duplicated in the current directory
process_remote_directory() {
VMESS="${VMESS} is a remote directory that"
if ((OPT_DIRECTORIES)) ; then
if exists $name ; then
if [[ ! -d $name ]] ; then
VMESS="${VMESS} cannot be created due to pre-existing object"
echo WARNING: no mkdir since $name exists in ${localpath} >&2
else
VMESS="${VMESS} already exists"
fi
else
mkdir "$name"
VMESS="${VMESS} has been created locally"
if ((OPT_MODE)) ; then
chmod $mode "$name"
fi
fi
else
VMESS="${VMESS} has been ignored"
fi
if (($OPT_VERBOSE)) ; then
echo "$VMESS"
fi
if ((OPT_RECURS)) ; then
if [[ -d "$name" ]] ; then
cd "$name"
print -p lcd \""$name"\"
exec 4<&-
obtain_and_process_remote_ls "$name"
print -p cd ..
print -p lcd ..
cd ..
exec 4< ${DIR_FILE_NAME[LEV]}
lineno=0
while (( lineno != ${DIR_LINE_NUM[LEV]})) ; do
read -u4 junk
((lineno=lineno+1))
done
fi
fi
return 0
}
#
# Function to process a remote symlink
# Note that we deal with th symlink only -- not
# the object (if any) that the link points to.
process_remote_symlink() {
VMESS="${VMESS} is a remote symlink that"
if ((OPT_SYMLINKS)) ; then
if exists "$name" ; then
if [[ ! -L $name ]] ; then
VMESS="${VMESS} cannot be created due to pre-existing object"
echo WARNING: no symlink since $name exists in ${localpath} >&2
else
VMESS="${VMESS} already exists"
fi
else
ln -s "$link" "$name"
VMESS="${VMESS} has been duplicated locally"
fi
else
VMESS="${VMESS} has been ignored"
fi
if (($OPT_VERBOSE)) ; then
echo "$VMESS"
fi
}
#
# If a remote object is not a file, directory, or
# symlink, we come here.
process_remote_weirdo() {
VMESS="${VMESS} is a remote unknown object that has been ignored"
return 0
}
#
# This function obtains an "ls" listing from the remote ftp system. Then it
# scans the listing line by line to figure out what to do. It will completely
# process the current directory.
obtain_and_process_remote_ls() {
typeset rdir tmpfile okfile ## local scope variables ##
rdir=$1
#
# Set up variables or modify them if we have recursed
((LEV=LEV+1))
tmpfile=/tmp/HardFeed.tp.$$.${LEV}
okfile=/tmp/HardFeed.ok.$$.${LEV}
if ((LEV == 1)) ; then
localpath=$STARTPATH
remotepath=$rdir
else
localpath=${localpath}/$rdir
remotepath=${remotepath}/$rdir
fi
#
# Get a copy of the remote dir output in a local file
# called $tmpfile
print -p cd \""$rdir"\"
print -p $OPT_DIRCMD $tmpfile
print -p $OPT_DIRCMD $okfile
waitfor $okfile
DIR_FILE_NAME[LEV]=$tmpfile
DIR_LINE_NUM[LEV]=0
exec 4< $tmpfile
#
# process each line
#
while read -u4 line ; do
((DIR_LINE_NUM[LEV]=${DIR_LINE_NUM[LEV]}+1))
lsdcode "$line"
char1=${lsdc[0]}
mode=${lsdc[1]}
datestamp=${lsdc[2]}
name=${lsdc[3]}
link=${lsdc[4]}
VMESS="${remotepath}/${name}"
case $char1 in
skip) ;;
-) process_remote_file
;;
d) process_remote_directory
;;
l) process_remote_symlink
;;
*) process_remote_weirdo
;;
esac
done
#
# We may have recursed...so we must put everything back the way
# we found it
localpath=${localpath%$rdir}
localpath=${localpath%/}
remotepath=${remotepath%$rdir}
remotepath=${remotepath%/}
rm $tmpfile
rm $okfile
((LEV=LEV-1))
return 0
}
#
# Main Program
#
ftp -inv >&3 2>&1 |&
print -p open $SYSTEM
print -p user $USER $PASSWORD
print -p binary
i=0
while ((OPT_CMDS>i)) ; do
print -p ${OPT_CMDS_LIST[i]}
((i=i+1))
done
obtain_and_process_remote_ls $DIRECTORY
print -p bye
wait
exit 0
You just need to copy all the script, save it to a file, upload it to your server and run it. But first you must install the ksh, in ubuntu you can just type :
sudo apt-get install ksh
Example of using it :
./hardfeed
- ./hardfeed : the name of the script
- ftp server : the ip or the domain of the ftp server
- username : the username of the ftp
- directory : directory in the ftp server where you want to get the files from
Here the another options which I copied from the website :
HardFeed ftpserver joeblow somedir
This will connect to “ftpserver” and the user “joeblow”. It will cd to “somedir”. It will look at all of the files (and only the files) there. Each remote file that does not exist in the current directory will be copied to the current directory.
HardFeed -ds ftpserver joeblow somedir
This will work as the above example did. Except now we try to create local copies of any symbolic links or directories that we found in “somedir”. Again, though, we will not overwrite any pre-existing object.
HardFeed -rds ftpserver joeblow somedir
Now we will create copies of any remote directories and desend into them. This will copy an entire directory tree. (except that it continues to ignore special files, pipes, etc.)
HardFeed -rs ftpserver joeblow somedir
This is similiar, except we only desend into pre-existing directories. You can use this to copy part of a directory structure. Just pre-create the few directories that you want to copy.
HardFeed -rdsm ftpserver joeblow somedir
This will copy a directory tree, but this time it will try to duplicate the mode (permissions) on each remote object that is duplicated.
HardFeed -rdsf ftpserver joeblow somdir
The -f is “freshen”. Again it copies a directory tree. But if we have a local file and a remote file, the timestamps are compared. The remote file will overwrite the the local file if the remote file was newer.