source: trunk/doc/src/docbook/userdoc/import_export_data.xml @ 3314

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE chapter PUBLIC 
  "-//Dawid Weiss//DTD DocBook V3.1-Based Extension for XML and graphics inclusion//EN" 
  "../../../../lib/docbook/preprocess/dweiss-docbook-extensions.dtd" 
[
<!ENTITY runplugin.configure.common 
  "The top of the window displays the names of the selected plug-in and
  configuration, a list with parameters to the left, an area for input fields to the
  right and buttons to proceed with at the bottom. 
  Click on a parameter in the parameter list to show the form fields
  for entering values for the parameter to the right. Parameters
  with an <guilabel>X</guilabel> in front of their names already have a
  value. Parameters marked with a blue rectangle are required and must
  be given a value before it is possible to proceed."
>
]>
<!--
  $Id: import_export_data.xml 3314 2007-05-09 11:52:06Z nicklas $
  
  Copyright (C) Authors contributing to this file.
  
  This file is part of BASE - BioArray Software Environment.
  Available at http://base.thep.lu.se/
  
  BASE is free software; you can redistribute it and/or
  modify it under the terms of the GNU General Public License
  as published by the Free Software Foundation; either version 2
  of the License, or (at your option) any later version.
  
  BASE is distributed in the hope that it will be useful,
  but WITHOUT ANY WARRANTY; without even the implied warranty of
  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
  GNU General Public License for more details.
  
  You should have received a copy of the GNU General Public License
  along with this program; if not, write to the Free Software
  Foundation, Inc., 59 Temple Place - Suite 330,
  Boston, MA  02111-1307, USA.
-->
<chapter id="import_export_data">
  <?dbhtml dir="import_export"?>
  <title>Import and export of data</title>
  <para>
    In some places the only way to get data into BASE is to import it
    from a file. This typically includes raw data, array design features,
    reporters and other things, which would be inconvenient to enter 
    by hand due to the large number of data items.
  </para>
  <para>
    On the other hand, if data should be used outside of BASE, you need
    to export it. Exporting data is possible for almost all kind of data.
    These two actions, importing and exporting, are not built into BASE but
    are always handled by plug-ins.
  </para>
  <para>
    Normally, a plug-in handles one type of items, but some plug-ins,
    for example the table exporter plug-in, can work
    with several types of items. 
    Some plug-ins requires a configuration, for example,
    the import plug-ins need some information about how to find headers and
    data lines in the files. BASE ships with a number of plug-ins, the core plug-ins. 
    A <ulink
    url="http://base.thep.lu.se/chrome/site/doc/admin/plugin_configuration/coreplugins.html"
      >list with the core plug-ins</ulink>
    can be found on BASE's homepage. On this page are also configuration 
    examples that can be downloaded for some of the plugins.
    Go to 
    <menuchoice>
      <guimenu>Administrate</guimenu>
      <guimenuitem>Plugins</guimenuitem>
      <guisubmenu>Definitions</guisubmenu>
    </menuchoice>
    to check which plug-ins are installed on your BASE server.
  </para>
  
  <para>
    When a plug-in that supports import or export of a certain type of item
    an <guibutton>Import</guibutton> or <guibutton>Export</guibutton>
    button is displayed in the toolbar on either the list view or the single-item
    view.
  </para>
  <note>
    <title>Missing/unavailable button</title>
    <para>
      If the import and/or export button is missing from a page were you would expect
      to find them this usually means that:
    </para>
    <itemizedlist>
      <listitem>
        <simpara>
          The logged in user doesn't have permission to use the plug-in.
        </simpara>
      </listitem>
      <listitem>
        <simpara>
          The plug-in requires a configuration, but no one has been
          created or the logged in user doesn't have permission to 
          use any of the existing configurations.
        </simpara>
      </listitem>
    </itemizedlist>
    <para>
      Contact the server administrator or a similar user that has permission to 
      administrate the plug-ins.
    </para>
  </note>

  <sect1 id="import_export_data.import">
    <title>Import</title>

    <para>
      Starting a data import is done by a wizard-like interface. There
      are a number of step you have to go through:
    </para>
    
    <orderedlist>
      <listitem>
        <simpara>
          Select a plug-in and file format to use, or select the
          auto detect option.
        </simpara>
      </listitem>
      <listitem>
        <simpara>
          If you selected the auto detection funtion, you must select
          a file to use.
        </simpara>
      </listitem>
      <listitem>
        <simpara>
          Specify plug-in parameters.
        </simpara>
      </listitem>
      <listitem>
        <simpara>
          Add the import job to the job queue.
        </simpara>
      </listitem>
      <listitem>
        <simpara>
          Wait for the job to finish.
        </simpara>
      </listitem>
    </orderedlist>

    <sect2 id="import_export_data.import.plugin_fileformat">
      <title>Select plug-in and file format</title>
      <para>
        Click on the <guibutton>Import</guibutton> button
        in the toolbar to start the import wizard. The first step is to
        select which plug-in and, if supported, which 
        file format to use. There is also an <guilabel>auto detect</guilabel>
        option that lets you select a file and have BASE try to find a suitable
        plug-in/file format to use. 
      </para>
  
      <figure id="import_export_data.figures.select_import_plugin">
        <title>Select plug-in and file format</title>
        <screenshot>
          <mediaobject>
            <imageobject><imagedata fileref="figures/select_import_plugin.png" format="PNG" /></imageobject>
          </mediaobject>
        </screenshot>
      </figure>
    

      <helptext external_id="import.selectplugin"
        title="Select plug-in and file format for data import">

        <variablelist>
          <varlistentry>
            <term><guilabel>Plugin</guilabel></term>
            <listitem>
              <para>
                A list of all plug-ins that are avilable in the
                current context. The list only includes plug-ins that
                the logged in user has permission to use. If you select
                a plug-in a short description of about it is displayed
                below the lists. More information about the plug-ins can
                be found under the menu choice
                <menuchoice>
                  <guimenu>Administrate</guimenu>
                  <guimenuitem>Plugins</guimenuitem>
                  <guisubmenu>Definitions</guisubmenu>
                </menuchoice>
              </para>
            </listitem>
          </varlistentry>
          
          <varlistentry>
            <term><guilabel>File format</guilabel></term>
            <listitem>
              <para>
                A list of different file formats configurations
                supported by the selected plug-in.
                <menuchoice>
                  <guimenu>Administrate</guimenu>
                  <guimenuitem>Plugins</guimenuitem>
                  <guisubmenu>Configurations</guisubmenu>
                </menuchoice>.
              </para>
              
              <note>
                <title>File format vs. Configuration</title>
                <simpara>
                A file format is the same thing as a plug-in configuration.
                It may be confusing that the interface sometimes use
                <emphasis>file format</emphasis> and sometimes use 
                <emphasis>configuration</emphasis>, but for now, we'll have
                to live with it.
                </simpara>
              </note>
              
            </listitem>
          </varlistentry>
        
        </variablelist>

        <para>
          Proceed to the next step by clicking on the
          <guibutton>Next</guibutton> button.
        </para>

        <seeother>
          <other external_id="import.autodetect">The auto detect function</other>
        </seeother>
      </helptext>

      <sect3 id="import_export_data.import.plugin_fileformat.autodetect">
        <title>The auto detect function</title>
        
        <helptext
          external_id="import.autodetect"
          title="The auto detect function">
        
        <para>
          The auto detect function lets you select a file and have
          BASE try to find a suitable plug-in and file format. This option is
          selected by default in both the plug-in and file format lists when there is
          at least one plug-in that supports auto detection.
        </para>
        <note>
          <title>Support of auto detect</title>
          <para>
            Not all plug-ins support auto detection. The ones that do are marked in
            the list with <guilabel>×</guilabel>.
          </para>
        </note>
        
        <para>
          Select the auto detect option either for both plug-ins and 
          file formats or only for file formats to use this feature. 
          Continue to the next step by clicking on the <guibutton>Next</guibutton>
          button.
        </para>
        
        <seeother>
          <other external_id="import.selectplugin">Select plug-in and file format for data import</other>
          <other external_id="import.autodetect.selectfile">Select file for auto detection</other>
        </seeother>
      
        </helptext>
        
        <para>
          You must now select a file to import from.
        </para>
        
        <figure id="import_export_data.figures.select_autodetect_file">
          <title>Select file for auto detection</title>
          <screenshot>
            <mediaobject>
              <imageobject><imagedata fileref="figures/select_autodetect_file.png" format="PNG" /></imageobject>
            </mediaobject>
          </screenshot>
        </figure>
        
        <helptext external_id="import.autodetect.selectfile" 
          title="Select file for auto detection">
  
          <variablelist>
            <varlistentry>
              <term><guilabel>File</guilabel></term>
              <listitem>
                <para>
                  Enter the path and filename for the
                  file you want to use. Use the <guibutton>Browse&hellip;</guibutton>
                  button to browse after the file in BASE's file system. 
                  If the file doesn't exist in the file system you have the option
                  to upload it. 
                  <nohelp>Read more about this in <xref linkend="file_system" />.</nohelp>
                </para>
              </listitem>
            </varlistentry>
            
            <varlistentry>
              <term><guilabel>Recently used</guilabel></term>
              <listitem>
                <para>
                  A list of files you have recently used
                  for auto detection.
                </para>
              </listitem>
            </varlistentry>
          </variablelist>
          
          <para>
            Click on the <guibutton>Next</guibutton> button
            to start the auto detection.
          </para>

          <para>
            If the auto detection finds a exactly one plug-in and file format
            the next step is to configure any additional parameters needed
            by the plug-in. This is the same step as if you had selected
            the same plug-in and file format in the first step.
            If no plug-in can be found an error message is displayed.
          </para>

          <note>
            <title>More then one compatible plug-in/file format</title>
            <para>
              If more than one matching plug-in or file format is used
              you will be taken back to the first step. This time
              the lists will only include the matching plug-ins/file formats
              and the auto detect option is not present.
            </para>
          </note>

          <seeother>
            <other external_id="import.selectplugin">Select plug-in and file format for data import</other>
            <other external_id="import.autodetect">The auto detect function</other>
          </seeother>
          
        </helptext>
        
      </sect3>

    </sect2>

    <sect2 id="import_export_data.import.pluginparameters">
      <title>Specify plug-in parameters</title>
      <para>
        When you have selected a plug-in and file format or used
        the auto detect function to find one, a form where you
        you can enter additional parameters for the plug-in is displayed.
      </para>
      
      <figure id="import_export_data.figures.confiure_plugin">
        <title>Specify plug-in parameters</title>
        <screenshot>
          <mediaobject>
            <imageobject><imagedata fileref="figures/plugin_parameters.png" format="PNG" /></imageobject>
          </mediaobject>
        </screenshot>
      </figure>
      
      <helptext external_id="runplugin.configure.import" 
        title="Specify plug-in parameters">
      <para>
        &runplugin.configure.common;
      </para>
      
      <para>
        The parameter list is very different from plug-in to plug-in.
        Common parameters for import plug-ins are:
      </para>
      
      <variablelist>
        <varlistentry>
          <term><guilabel>File</guilabel></term>
          <listitem>
            <para>
            The file to import data from. A value is already set if
            you used the auto detect function.
            </para>
          </listitem>
        </varlistentry>
        
        <varlistentry>
          <term><guilabel>Error handling</guilabel></term>
          <listitem>
            <para>
              A section which contains different options how to
              handle errors when parsing the file. Normally you can
              select if the import should fail as a while or if
              the line with the error should be skipped.
            </para>
          </listitem>
        </varlistentry>
      </variablelist>
      
      <para>
        Continue to the next step by clicking the 
        <guibutton>Next</guibutton> button.
      </para>
      </helptext>

    </sect2>
    
    <sect2 id="import_export_data.import.jobqueue">
      <title>Add the import job to the job queue</title>

      <para>
        In this window should information about the job be filled in, like name and
        description. Where name is required and need to have valid string as a value. There
        is also an option if the user want to get a message or not after the job is done.
        The check-box should be ticked if the owner of the job wants to get a message with a
        summary of the job's execution.
      </para>
      <para>
        Clicking on
        <guibutton>Finish</guibutton>
        when everything is set will end the job configuration and place the job in the job queue. 
        A self-refreshing window appears with information about the
        job's status and execution time. How long time it takes before the job starts to run
        depends on which priority it and the other jobs in the queue have. The job doesn't
        depend on the status window to be able to run and the window can be
        closed without interupting the execution.
      </para>
      <tip>
        <title>View job status</title>
        <para>
          A job's status can be viewed at any time by opening it from the job list page,
          <menuchoice>
            <guimenuitem>View</guimenuitem>
            <guimenuitem>Jobs</guimenuitem>
          </menuchoice>.
        </para>
      </tip>
    </sect2>
  </sect1>

  <sect1 id="import_export_data.export">
    <title>Export</title>
    <para>
      Before data from BASE can be used outside the program, it first had to be
      exported to a file which then can be downloaded to a local path from the BASE file
      system.
    </para>
    <para>
      A data export is done by a job that runs an export plug-in for the current context. An
      export job is started by clicking on
      <guibutton>Export&hellip;</guibutton>
      in the toolbar, which will open a pop-up window allowing you to
      select plug-in and specify parameters for it.
    </para>
    
    <sect2 id="import_export_data.export.selectplugin">
      <title>Select plug-in and configuration</title>

      <para>
        This dialog is very similar to the dialog for selecting an 
        import plug-in. See <xref linkend="import_export_data.figures.select_import_plugin" />
        for a screenshot example.
      </para>
    
      <helptext external_id="runplugin.selectplugin" title="Select plug-in">
        <para>
          The first thing in the configuration process is to choose which plug-in 
          to use and a configuration for those plug-ins that require it. More
          information about the plug-ins can be found in each plug-in's documentation.
        </para>
        <note>
          If there is only one plug-in and configuration available, this
          step is skipped and you are taken directly to next step.
        </note>
        
        <variablelist>
          <varlistentry>
            <term>
              <guilabel>Plugin</guilabel>
            </term>
            <listitem>
              <para>
                Select the plug-in to use. Only plug-ins that supports the current
                context and that the logged in user has permission to use are
                available in the list.
              </para>
            </listitem>
          </varlistentry>
          <varlistentry>
            <term>
              <guilabel>Configuration</guilabel>
            </term>
            <listitem>
              <para>
                Select the configuration that should be used together with the plug-in.
                Not all plug-ins supports configuration and this option is only visible
                if the selected plug-in has this support.
              </para>
            </listitem>
          </varlistentry>
        </variablelist>
        <para>
          Click on
          <guibutton>Next</guibutton>
          to show the configuration of parameters for the job.
        </para>
        
        <seeother>
          <other external_id="runplugin.configure.export">Specify plug-in parameters</other>
        </seeother>
      </helptext>
    </sect2>
    
    <sect2 id="import_export_data.export.pluginparameters">
      <title>Specify plug-in parameters</title>
    
      <helptext external_id="runplugin.configure.export" 
        title="Specify plug-in parameters">
        
      <para>
        &runplugin.configure.common;
      </para>
      
      <para>
        The parameters list is very different from plug-in to plug-in.
        Common parameters for export plug-ins are:
      </para>
      
      <variablelist>
        <varlistentry>
          <term><guilabel>Save as</guilabel></term>
          <listitem>
            <para>
              The path and filename in the BASE file system where
              the exported data should be saved. Some plug-ins support immediate 
              download to the local file system if you leave the file parameter 
              empty. For saving the exported data within the BASE file system, 
              it's recommended to use the <guibutton>Browse&hellip;</guibutton>
              button to get the right path and then complement it with the file's name.
            </para>
          </listitem>
        </varlistentry>
      </variablelist>
      <para>
        Click on
        <guibutton>Next</guibutton>
        to proceed to next configuration window.
      </para>
    
    </helptext>
    
    <sect3 id="import_export_data.export.immediatedownload">
      <title>Immediate download of the exported data</title>
      
      <para>
        If the selected plug-in supports immediate download and the file parameter 
        were left empty a new windown with a <guibutton>Download</guibutton>
        button is displayed. Click on this button to start the plug-in execution.
        Do not close the window until a message saying that the export
        was successful (or failed) is displayed. Your browser should open a dialog
        asking you were to save the file on your local computer.
      </para>
      
      <figure id="import_export_data.figures.download_immediately">
        <title>Download immediately</title>
        <screenshot>
          <mediaobject>
            <imageobject><imagedata fileref="figures/download_immediately.png" format="PNG" /></imageobject>
          </mediaobject>
        </screenshot>
      </figure>
      
    </sect3>
    
    <sect3 id="import_export_data.export.savetofilesystem">
      <title>Saving the exported data in the BASE file system</title>

      <para>
        If you choose to save the file within the BASE file system, there will be a
        window where the job should get a name and optionally a description. By then clicking on
        <guibutton>Finish</guibutton>
        the configuration process will end and the job will be put in the job queue. A
        self-refreshing window appears with information about the job's status and execution
        time. The job isn't dependent on the status window to run and it therefore be closed
        without interrupting the execution of the job.
      </para>
      <tip>
        <title>View job status</title>
        <para>
          A job's status can be viewed at any time by opening it from the job list page,
          <menuchoice>
            <guimenuitem>View</guimenuitem>
            <guimenuitem>Jobs</guimenuitem>
          </menuchoice>.
        </para>
      </tip>
    </sect3>
  </sect2>
  
  <sect2 id="import_export_data.export.table_export">
    <title>The table exporter plug-in</title>
    <para>
      The table exporter is a generic export plug-in that works with almost
      all list views in BASE. It can export the lists as an XML-file or a
      tab-separated text file. The table exporter is started,
      like the other export plug-ins, by clicking on
      <guibutton>Export&hellip;</guibutton> in the toolbar.
    </para>
    
    <para>
      Then select the <guilabel>Table exporter</guilabel> and click on
      the <guibutton>Next</guibutton> button. The plug-in selection step is
      only displayed if there is more than one export plug-in that can
      be used in the current context. Usually, the table exporter is the
      only plug-in and you will be take directly to the configuration
      dialog.
    </para>
      
    <para>
      Unlike other plug-ins, the table exporter doesn't use the
      generic parameter input dialog. It has a customized dialog
      that should be easier to use.
    </para>
      
    <figure id="import_export_data.figures.table_exporter">
      <title>The table exporter configuration dialog</title>
      <screenshot>
        <mediaobject>
          <imageobject><imagedata fileref="figures/table_exporter.png" format="PNG" /></imageobject>
        </mediaobject>
      </screenshot>
    </figure>
    
      
    <helptext external_id="simpleexport.options" title="Export options">
      <variablelist>
        <varlistentry>
          <term>
            <guilabel>Format</guilabel>
          </term>
          <listitem>
            <para>
              The generated file can be either a 
              <guilabel>tab-separated text file</guilabel>
              or an <guilabel>XML</guilabel> file. The
              XML-file option will generate a tag for each item and these
              will contain a child tag with property value for each
              selected column.
            </para>
          </listitem>
        </varlistentry>
        <varlistentry>
          <term>
            <guilabel>Which items?</guilabel>
          </term>
          <listitem>
            <para>
              This option decide which items that should be included in
              the exported data.
              <itemizedlist>
                <listitem>
                  <para>
                    <guilabel>Selected items</guilabel>: 
                    Only selected items will be exported. This options
                    is not available if no items have been
                    selected on the list page. 
                  </para>
                </listitem>
                <listitem>
                  <para>
                    <guilabel>Current page</guilabel>: Exports all 
                    items viewed on the current list page.
                  </para>
                </listitem>
                <listitem>
                  <para>
                    <guilabel>All pages</guilabel>: Items on all pages 
                    will be exported.
                  </para>
                </listitem>
              </itemizedlist>
            </para>
          </listitem>
        </varlistentry>
        <varlistentry>
          <term>
            <guilabel>Which columns?</guilabel>
          </term>
          <listitem>
            <para>
              Names of those columns that should be included in the export
              should be listed in the <guilabel>Exported columns</guilabel>
              to the left. A column name is
              moved to the other list-box by first marking it and then
              clicking on one of the buttons located between the
              list-boxes.
            </para>
            <para>
              The order in which the columns should be exported in can be
              changed with the buttons to the left of the list. Simply
              mark a name of a column and click on the buttons to move the
              name either up or down in the list.
            </para>
            <tip>
              <title>Using presets</title>
              <para>
                Use the drop down list of presets, located under the
                option name, to easily get predefined or own presets of
                column settings.
              </para>
            </tip>
          </listitem>
        </varlistentry>
        <varlistentry>
          <term>
            <guilabel>Save as</guilabel>
          </term>
          <listitem>
            <para>
              The path and filename where the exported data
              should be saved. Leave the text field empty if the file is
              to be downloaded immediately or enter a path within the BASE
              file system to store the file on the server. Check
              <guilabel>Overwrite existing file</guilabel>
              if an already existing file with the same name should be
              overwritten.
            </para>
          </listitem>
        </varlistentry>
      </variablelist>
      
      <para>
        Click on <guibutton>Ok</guibutton>
        to proceed when all options have been set for the export.
      </para>
    </helptext>

    </sect2>
  </sect1>
</chapter>