Proteios FTP Server Usage

Description

This note describes the Proteios FTP Server usage. Installation information can be found at Proteios FTP Server.

Requirements

1. Proteios FTP Server installation (see Proteios FTP Server).
2. An FTP client program to access the Proteios FTP Server.

Usage

Terms used

FTP

File Transfer Protocol

Core File

A File item in the Proteios database

Core Directory

A Directory item in the Proteios database

General Usage Information

1. The Proteios FTP Server must be started before a connection to it is attempted.
2. If desired, the control port setting in configuration file ftp.properties (see Quick Install) may be over-ridden by giving the control port as argument to the start script.
3. The FTP client program should connect to the Proteios FTP Server on its specific control port.
4. When logging in, the username and password of the Proteios account should be used.
5. After login, the user is placed in his/her Proteios home core directory.
6. The Proteios FTP server will display paths using web syntax, with a slash '/' as separator.
7. The user's Proteios Projects will show up as Proteios core directories.
8. The user can request information on core files and directories in the current core directory.
9. The user can navigate to accessible core directories.
10. The user can create a new core sub-directory in the current core directory.
11. The user can remove core files and directories.
12. The user can upload a local file to the current core directory, where it will show up as a core file.
13. The user can download a core file from the current core directory to the local system, where it will show up as a normal file.
14. The FTP session should be ended by logging out from Proteios before stopping the Proteios FTP Server.

Shared Projects and Directories in Proteios SE

Introduction

Proteios SE allows a whole project to be shared with other users, and selected directories and/or files might be shared to another project than they were created in. In the Proteios SE graphical user interface (GUI), shared projects are included in the project list, and directories shared to a selected project are included in the directory list for the project directory of the current project (shown in green color, to indicate that the directory is shared). For descriptions on how to share data in Proteios SE, see Sharing Information.

Notes on Support for Shared Items in Proteios FTP Server

The FTP protocol limits the information that can be sent from and to the FTP server. Lists and tables in the GUI internally use database id values to identify the various items, while selection of directory and file items in an FTP session may result in only the name of an item being sent to the server, which must construct the full path of the item. Also, FTP works with directory trees only, not project views, and listed items cannot be marked in a way that is guaranteed to be supported by all FTP clients. The support for shared items in the Proteios FTP server is therefore a compromise, and some functionality might not work on all FTP clients, or in all situations. The added functionality should be regarded as an option to perform some operations involving shared directories via FTP. However, if problems should arise, you are recommended to perform the action in a two step process, if possible, e.g. by transferring files via FTP to a directory owned by you, and then move the files to the shared directories they were destined for using the GUI.

Shared Projects in Proteios FTP Server

When the contents of your Proteios SE home directory is listed via FTP, a virtual directory named "_Shared_Projects" is displayed in the directory list (it is called "virtual", since it does not correspond to an actual directory on the Proteios SE file system). The name was chosen to make it appear at the top of the directory list in most FTP clients, and be unlikely to be identical to an existing directory name. The shared projects folder will be displayed with the current date and time to indicate its virtual status. When navigating into the shared projects folder and listing its contents, a list of project directories for directories owned by other users and shared to the current user is displayed. The shared directories are reported using their absolute path in order to indicate their shared status, but this might only be visible in text-based FTP clients. To use a shared project directory, select it in the usual way in the FTP client. You may then continue to navigate to a desired sub-directory.

Selected Shared Directories in Proteios FTP Server

Proteios FTP server attempts to include directories shared to a specific project, when the contents of the project directory of the latter is listed. The shared directories are reported using their absolute path in order to indicate their shared status, but this might only be visible in text-based FTP clients. To use a shared directory, select it in the usual way in the FTP client. Note that your ability to navigate further from a shared directory might be limited, if the directory is owned by another user.

Notes on FTP Client Programs

A command given to the FTP client program may generate several raw FTP commands sent to the FTP server. FTP client programs with a command line interface, like the Unix ftp program used in the examples, normally support commands close to raw FTP commands and displays output from the FTP server without much post-processing. The commands of most use when working against the Proteios FTP Server are connection commands (open, close), navigation commands (pwd, cd, dir), directory creation commands (mkdir), directory and file removal commands (rmdir, delete), file upload commands (put, mput), and file download commands (get, mget).

GUI FTP clients, on the other hand, normally perform extra processing:

• Several raw FTP commands in order might be sent to obtain information for displaying the remote system as a directory tree.
• Returned absolute paths for directories or files might be trimmed to only show the item without path information, e.g. "/home/user2/Project20090630b/testdir1" is displayed as simply "testdir1".
• Order of output lines might be modified, e.g. to display lines in alphabetical order.

For a description of the commands available for your FTP client program, please consult its documentation.

FTP Clients Tested

The following FTP clients have been tested and shown to be able to download and upload files from/to Proteios, however, it is not guaranteed that all functionality works:

• Unix ftp (command line client)
• mozilla
• gftp
• FireFTP (plugin to Firefox web browser, restricted to 2GB file size for file upload)
• WinSCP
• Internet Explorer 6 (MS Windows)

Recommendations

For best chance of success in downloading and uploading Proteios files via ftp:

• Move to the remote core directory where you want to upload or download files, so the former is the current directory.
• Don't try to move to a core directory above the home directory for the user logged in.
• Only use a single connection for transferring files (some FTP clients have the option of using multiple connections).
• If possible, end the FTP session by closing the connection before terminating the FTP client program. This is done by the close command for command line FTP clients, while GUI clients may use a "disconnect" button or menu command. Closing the FTP session makes the Proteios FTP Server log out from the used Proteios account.

Example FTP Sessions

Example Setup

In this section the different steps in the setup and use are described, and an example of each step is provided. The example is based on the following assumptions:

IP address of Proteios system	130.235.189.210
Operating system on Proteios system	Unix/Linux
Proteios FTP Server Control Port	8021
Proteios user username	luna
Proteios user password	selena3
Proteios user home core directory	/home/luna/
Proteios user target core directory	/home/luna/Project_Data/
IP address of user system	130.235.189.217
Operating system on user system	Unix/Linux (only concerns available FTP client programs)
User's username on local system	crescent
FTP client program	ftp
File to transfer to Proteios	temp_out.txt (in user's current local directory)

Example setup.

Many of the example data items are only provided in order to explain output from the program, e.g. operating system, IP address of user system, and user's username on local system. There is nothing to prevent some of the variables to be equal, e.g. the usernames on the local system and in Proteios may be the same, and the user system may be the same as the system on which Proteios is installed. For the example it was chosen to use unique values in these cases, in order to avoid confusion regarding what information is referenced in the input and output. For a user with a Proteios account, the additional information needed to use the Proteios FTP Server, are the name or IP address of the Proteios system, and the port number used for the Proteios FTP Server.

The operations are given as commands on a command line. Input is shown in bold monospace font, and output in plain monospace font.

Note on List Information on File and Directory Items

In order to allow parsing by GUI FTP client programs, Proteios FTP Server returns output from raw FTP command LIST in a format close to that of command ls -l in the Unix shell. One line is listed for each item, starting with a d for a directory and - for a plain file:

drwxr-xr-x 2 100 100 4096 Nov 10 14:58 A_Directory
-rw-r--r-- 1 100 100 31127 Nov 10 14:59 A_Plain_File.xml

The relevant information here is the first character, indicating if the item is a directory or a plain file, the creation time and name of the item, as well as the size of plain files. The creation time is given as date plus time of day for items that are less than a year old, otherwise it is given as date plus year:

-rw-r--r-- 1 100 100 31127 Nov 10 14:59 New_File.txt
-rw-r--r-- 1 100 100 31127 Jan 14 2006 Old_File.txt

Starting the Proteios FTP Server

On the Proteios system, use the shell script proteios_ftp_server.sh (on Windows, use batch file proteios_ftp_server.bat) from the command line to start the Proteios FTP Server using Port 8021 as control channel (if no control port number is given, the setting in configuration file ftp.properties is used):

proteios_ftp_server.sh -start -p8021
nohup: appending output to 'nohup.out'

If everything was successful, the end of file nohup.out contains the lines:

FTPServer is now running.
Host: 130.235.189.210
Port: 8021

Note that the nohup.out file is specific for Unix/Linux systems.

Connecting to the Proteios FTP Server using the FTP Client Program

On the local user system, use the FTP client program (ftp in the example) to connect to the Proteios system on the specified port (Port 8021 in the example):

ftp -P8021
ftp>

Here, "ftp>" is the command prompt of the FTP client program, waiting for input.

Uploading a File to Proteios using the FTP Client Program

Below is shown example input and output for the FTP client program, when uploading a file temp_out.txt from the current directory on the local system to core directory Project_Data for the user in the Proteios database:

ftp> open 130.235.189.210
Connected to 130.235.189.210.
220-Welcome to Proteios FTP Server, based on Xerver Free FTP Server 1.05.
220-
220-You can login below now.
220 Features: .
Name (130.235.189.217:crescent): luna
331 User name okay, need password.
Password:
230 User logged in, proceed.
Remote system type is UNIX.
Using binary mode to transfer files.
ftp> pwd
257 "/home/luna/"
ftp> dir
227 Entering passive mode (130,235,189,210,204,129).
150 Starting ASCII transfer for file listing.
drwxr-xr-x 2 100 100 4096 Nov 13 17:09 _Shared_Projects
drwxr-xr-x 2 100 100 4096 Nov 10 14:58 Project_Data
226 Closing data connection; requested file action successful.
ftp> cd Project_Data
250 Present working directory is now /Project_Data/
ftp> dir
227 Entering passive mode (130,235,189,210,202,17).
150 Starting ASCII transfer for file listing.
-rw-r--r-- 1 100 100 31127 Nov 10 14:59 mzDataExtraBlocks.xml
-rw-r--r-- 1 100 100 32929 Nov 10 14:59 mzDataExtraBlocks2_20061107_1136_dev.xml
-rw-r--r-- 1 100 100 715233 Nov 10 14:59 181150420000TEST_A2.xml
-rw-r--r-- 1 100 100 754143 Nov 10 14:59 181150420000TEST_A2_modified.xml
226 Closing data connection; requested file action successful.
ftp> put temp_out.txt
local: temp_out.txt remote: temp_out.txt
227 Entering passive mode (130,235,189,210,202,18).
150 Starting binary file transfer.
100% |***************************| 67493 2.25 MB/s 00:00 ETA
226 Closing data connection; requested file action successful.
67493 bytes sent in 00:00 (277.50 KB/s)
ftp> dir
227 Entering passive mode (130,235,189,210,202,20).
150 Starting ASCII transfer for file listing.
-rw-r--r-- 1 100 100 31127 Nov 10 14:59 mzDataExtraBlocks.xml
-rw-r--r-- 1 100 100 32929 Nov 10 14:59 mzDataExtraBlocks2_20061107_1136_dev.xml
-rw-r--r-- 1 100 100 715233 Nov 10 14:59 181150420000TEST_A2.xml
-rw-r--r-- 1 100 100 754143 Nov 10 14:59 181150420000TEST_A2_modified.xml
-rw-r--r-- 1 100 100 67493 Nov 13 17:10 temp_out.txt
226 Closing data connection; requested file action successful.
ftp> close
221 Bye.
ftp> exit

Stopping the Proteios FTP Server

On the Proteios system, use the shell script proteios_ftp_server.sh (on Windows, use batch file proteios_ftp_server.bat) from the command line to stop the Proteios FTP Server using Port 8021 as control channel (if no control port number is given, the setting in configuration file ftp.properties is used):

proteios_ftp_server.sh -stop -p8021
nohup: appending output to 'nohup.out'

Comments on the Example FTP Session

FTP dates from the early years of computer networks, and as seen in the example, the FTP server sends much connection information back to the FTP client program for each request. The FTP client program may filter the output, but many programs like the Unix ftp program in the example display most of it to the user. The numbers shown in the server's response after entering passive mode, e.g. (130,235,189,210,202,20) is the data port that it has opened, given as the server's IP address followed by the port number. For (130,235,189,210,202,20), the first 4 numbers 130,235,189,210 is the IP address (with a comma separating the numbers instead of the usual dot), and the last 2 numbers show the port number divided by 256 and the remainder, thus 202,20 corresponds to port no. 202 x 256 + 20 = 51732.

Uploading Multiple Files using the FTP Client Program

The commands for transferring multiple files in a session (mput, mget) actually generate several raw FTP commands, one for each single file transfer. The Proteios FTP Server only handles these single file transfer commands, so most problems related to multiple file transfer are solved by consulting the FTP client program's documentation. An exception is when the FTP server refuses some of the file transfers, e.g. when a file to upload has a name identical to a file already on the remote system, or when space limitations prohibit the upload.

A common question when trying to upload multiple files is how to avoid having the FTP client program ask for confirmation for each individual file transfer, e.g. "mput temp_out_a1.txt [anpqy?]?" or "ftp mput temp_out_a1.txt? y/n". This feature can be disabled in most FTP client programs. The Unix ftp program used in the example can be given the command prompt to toggle prompting (the default is 'on'). By starting the FTP client program with option "-i" to ignore prompting, it starts with prompting in 'off' mode (with specification of the control port number, the command would be "ftp -i -P8021"). Below is an extract from an FTP session uploading three files temp_out_a1.txt, temp_out_a2.txt, and temp_out_a3.txt using the mput command with prompting turned off. The extract clearly shows that the multiple file transfer is divided into single file transfers.

ftp -P8021
ftp> open 130.235.189.210
Connected to 130.235.189.210.
...
ftp> prompt
Interactive mode off.
ftp> mput temp_out_a*
local: temp_out_a1.txt remote: temp_out_a1.txt
227 Entering passive mode (130,235,189,210,205,105).
150 Starting binary file transfer.
100% |***************************| 67493 2.38 MB/s 00:00 ETA
226 Closing data connection; requested file action successful.
67493 bytes sent in 00:00 (1.69 MB/s)
local: temp_out_a2.txt remote: temp_out_a2.txt
227 Entering passive mode (130,235,189,210,205,106).
150 Starting binary file transfer.
100% |***************************| 67493 1.90 MB/s 00:00 ETA
226 Closing data connection; requested file action successful.
67493 bytes sent in 00:00 (1.48 MB/s)
local: temp_out_a3.txt remote: temp_out_a3.txt
227 Entering passive mode (130,235,189,210,205,107).
150 Starting binary file transfer.
100% |***************************| 67493 1.60 MB/s 00:00 ETA
226 Closing data connection; requested file action successful.
67493 bytes sent in 00:00 (855.81 KB/s)
ftp> dir
227 Entering passive mode (130,235,189,210,205,108).
150 Starting ASCII transfer for file listing.
-rw-r--r-- 1 100 100 31127 Nov 10 14:59 mzDataExtraBlocks.xml
-rw-r--r-- 1 100 100 32929 Nov 10 14:59 mzDataExtraBlocks2_20061107_1136_dev.xml
-rw-r--r-- 1 100 100 715233 Nov 10 14:59 181150420000TEST_A2.xml
-rw-r--r-- 1 100 100 754143 Nov 10 14:59 181150420000TEST_A2_modified.xml
-rw-r--r-- 1 100 100 67493 Nov 13 17:10 temp_out.txt
-rw-r--r-- 1 100 100 67493 Nov 14 11:54 temp_out_a1.txt
-rw-r--r-- 1 100 100 67493 Nov 14 11:54 temp_out_a2.txt
-rw-r--r-- 1 100 100 67493 Nov 14 11:54 temp_out_a3.txt
226 Closing data connection; requested file action successful.
ftp> close
221 Bye.
ftp> exit

Creating a New Directory using the FTP Client Program

When uploading files in an FTP session, one often finds that it would be preferrable to put the uploaded files in a new directory, that not already exists on the remote system. The Proteios FTP Server supports creating a new core sub-directory in the current core directory. The command for creating a new directory in the example ftp program is mkdir. Below is an extract from an FTP session creating a new core sub-directory New_Data in core directory Project_Data for the user in the Proteios database.

ftp -P8021
ftp> open 130.235.189.210
Connected to 130.235.189.210.
...
ftp> dir
227 Entering passive mode (130,235,189,210,205,200).
150 Starting ASCII transfer for file listing.
drwxr-xr-x 2 100 100 4096 Nov 14 14:00 _Shared_Projects
drwxr-xr-x 2 100 100 4096 Nov 10 14:58 Project_Data
226 Closing data connection; requested file action successful.
ftp> cd Project_Data
250 Present working directory is now /Project_Data/
ftp> mkdir New_Data
257 "New_Data"
ftp> dir
227 Entering passive mode (130,235,189,210,205,205).
150 Starting ASCII transfer for file listing.
drwxr-xr-x 2 100 100 4096 Nov 14 14:01 New_Data
-rw-r--r-- 1 100 100 31127 Nov 10 14:59 mzDataExtraBlocks.xml
-rw-r--r-- 1 100 100 32929 Nov 10 14:59 mzDataExtraBlocks2_20061107_1136_dev.xml
-rw-r--r-- 1 100 100 715233 Nov 10 14:59 181150420000TEST_A2.xml
-rw-r--r-- 1 100 100 754143 Nov 10 14:59 181150420000TEST_A2_modified.xml
-rw-r--r-- 1 100 100 67493 Nov 13 17:10 temp_out.txt
-rw-r--r-- 1 100 100 67493 Nov 14 11:54 temp_out_a1.txt
-rw-r--r-- 1 100 100 67493 Nov 14 11:54 temp_out_a2.txt
-rw-r--r-- 1 100 100 67493 Nov 14 11:54 temp_out_a3.txt
226 Closing data connection; requested file action successful.
ftp> close
221 Bye.
ftp> exit

Note on Proteios Project and Directory Items

A Proteios Project item shows up as a core directory in a Proteios FTP session, but a Proteios Project is more than a directory (you may think of a Proteios Project as being connected to a core directory with the same name as the project). When the Proteios FTP Server is used to create a new core directory, it does precisely this; it never creates a Proteios Project item (giving the new directory a name with "Project" in it doesn't help, it will still be just a core directory).

Removing a Directory or File using the FTP Client Program

Proteios allows removal of core directory and file items, and this is supported by the Proteios FTP Server. The command for removing a directory in the example ftp program is rmdir, and delete for removing a file. Below is an extract from an FTP session where a core directory New_Data2 and file temp_out_a2.txt are removed. After the removal a listing of the parent core directory does no longer show the removed items, and giving the removal command for one of the items a second time results in an error message that the item cannot be found.

...
ftp> cd Project_Data
250 Present working directory is now /Project_Data/
ftp> dir
227 Entering passive mode (130,235,189,210,216,129).
150 Starting ASCII transfer for file listing.
drwxr-xr-x 2 100 100 4096 Nov 14 14:01 New_Data
drwxr-xr-x 2 100 100 4096 Nov 15 14:46 New_Data2
drwxr-xr-x 2 100 100 4096 Nov 15 15:32 New_Data3
-rw-r--r-- 1 100 100 31127 Nov 10 14:59 mzDataExtraBlocks.xml
-rw-r--r-- 1 100 100 32929 Nov 10 14:59 mzDataExtraBlocks2_20061107_1136_dev.xml
-rw-r--r-- 1 100 100 715233 Nov 10 14:59 181150420000TEST_A2.xml
-rw-r--r-- 1 100 100 754143 Nov 10 14:59 181150420000TEST_A2_modified.xml
-rw-r--r-- 1 100 100 67493 Nov 13 17:10 temp_out.txt
-rw-r--r-- 1 100 100 67493 Nov 14 11:54 temp_out_a1.txt
-rw-r--r-- 1 100 100 67493 Nov 14 11:54 temp_out_a2.txt
-rw-r--r-- 1 100 100 67493 Nov 14 11:54 temp_out_a3.txt
226 Closing data connection; requested file action successful.
ftp> rmdir New_Data2
250 New_Data2 was successfully deleted.
ftp> delete temp_out_a2.txt
250 temp_out_a2.txt was successfully deleted.
ftp> dir
227 Entering passive mode (130,235,189,210,216,134).
150 Starting ASCII transfer for file listing.
drwxr-xr-x 2 100 100 4096 Nov 14 14:01 New_Data
drwxr-xr-x 2 100 100 4096 Nov 15 15:32 New_Data3
-rw-r--r-- 1 100 100 31127 Nov 10 14:59 mzDataExtraBlocks.xml
-rw-r--r-- 1 100 100 32929 Nov 10 14:59 mzDataExtraBlocks2_20061107_1136_dev.xml
-rw-r--r-- 1 100 100 715233 Nov 10 14:59 181150420000TEST_A2.xml
-rw-r--r-- 1 100 100 754143 Nov 10 14:59 181150420000TEST_A2_modified.xml
-rw-r--r-- 1 100 100 67493 Nov 13 17:10 temp_out.txt
-rw-r--r-- 1 100 100 67493 Nov 14 11:54 temp_out_a1.txt
-rw-r--r-- 1 100 100 67493 Nov 14 11:54 temp_out_a3.txt
226 Closing data connection; requested file action successful.
ftp> rmdir New_Data2
550 /home/luna/Project_Data/New_Data2: No such directory
ftp> delete temp_out_a2.txt
550 Requested action not taken. File or folder does not exists.

Note on Removed Items in Proteios

When an item in Proteios is removed, it is marked (tagged) as 'removed' in the database, indicating that the corresponding data is no longer needed, but the latter is still present in the database directly after the removal operation. However, there is no guarantee that the data for a removed item will remain intact in the database for a longer period of time. A Proteios item should therefore be considered for removal only when it is definitely of no more use. The default behaviour of Proteios is to ignore items tagged as 'removed', i.e. treating them as not existing, and the Proteios FTP Server is consistent with this.

CAUTION!

The removal operations should be performed with care under FTP, whether connecting to a standard FTP server or the Proteios FTP Server. The FTP connection does not allow the FTP server to prompt the user for confirmation before an item is removed. In addition, as FTP is often used to connect to a physically remote site, data recovery from e.g. a backup may be much harder to achieve from a remote location.

Note on Controlling the Proteios FTP Server under MS-Windows

Under Windows, the commands described above for proteios_ftp_server.sh can be used in a console window for batch file proteios_ftp_server.bat instead. The prompt will reappear after the command has been given, but closing the console window will terminate the programs started in it. (To run the FTP server as a true background process under Windows, it has to be run as a Windows service.)

To simplify management of Proteios FTP Server under Windows, two additional batch files are supplied, proteios_ftp_server_start.bat and proteios_ftp_server_stop.bat, which do not require any arguments, and therefore can be used to start and stop the FTP server by double-clicking on the icon for the corresponding batch file in Windows Explorer. The settings used will be reported in a log file created in the same directory as the batch files. The log files are named "proteios_ftp_server_start_debug_output.txt" and "proteios_ftp_server_stop_debug_output.txt". By default, the FTP control port set in configuration file ftp.properties is used. However, the control port to use and the name of the log file can be set inside the batch files, if desired. Setting the log file name to an empty string will lead to no log file being created.

When starting Proteios FTP Server by double-clicking on the icon for file proteios_ftp_server_start.bat, please note that it might take in the range of 30 seconds before the FTP server is up and a connection can be established. Stopping the FTP server is faster, but here also a delay exists.

Acknowledgement

Proteios FTP Server is based on the Xerver Free Web/FTP Server Package. For licensing information etc., see Third Party Licenses.

References

1. The Xerver home page can be found at ​http://www.javascript.nu/xerver/.