source: trunk/doc/src/docbook/user/import_data.xml @ 5798

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE chapter PUBLIC 
  "-//Dawid Weiss//DTD DocBook V3.1-Based Extension for XML and graphics inclusion//EN" 
  "../../../../lib/docbook/preprocess/dweiss-docbook-extensions.dtd" 
[
<!ENTITY runplugin.configure.common 
  "The top of the window displays the names of the selected plug-in and
  configuration, a list with parameters to the left, an area for input fields to the
  right and buttons to proceed with at the bottom. 
  Click on a parameter in the parameter list to show the form fields
  for entering values for the parameter to the right. Parameters
  with an <guilabel>X</guilabel> in front of their names already have a
  value. Parameters marked with a blue rectangle are required and must
  be given a value before it is possible to proceed."
>
]>
<!--
  $Id: import_data.xml 5798 2011-10-11 16:32:48Z nicklas $
  
  Copyright (C) 2007 Peter Johansson, Nicklas Nordborg, Martin Svensson
  Copyright (C) 2008 Jari Häkkinen
  
  This file is part of BASE - BioArray Software Environment.
  Available at http://base.thep.lu.se/
  
  BASE is free software; you can redistribute it and/or
  modify it under the terms of the GNU General Public License
  as published by the Free Software Foundation; either version 3
  of the License, or (at your option) any later version.
  
  BASE is distributed in the hope that it will be useful,
  but WITHOUT ANY WARRANTY; without even the implied warranty of
  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
  GNU General Public License for more details.
  
  You should have received a copy of the GNU General Public License
  along with BASE. If not, see <http://www.gnu.org/licenses/>.
-->
<chapter id="import_data" chunked="0">
  <title>Import of data</title>
  <para>
    In some places the only way to get data into BASE is to import it
    from a file. This typically includes <emphasis>raw data</emphasis>, 
    <emphasis>array design features</emphasis>, <emphasis>reporters</emphasis>
    and other things, which would be inconvenient
    to enter by hand due to the large number of data items. There is
    also convenience batch importers for importing other items such as
    <emphasis>biosources</emphasis>, <emphasis>samples</emphasis>, and 
    <emphasis>annotations</emphasis>. The batch importers are
    described later in this chapter after the general import
    description.
  </para>
  <para>
    Normally, a plug-in handles one type of items and may require a
    configuration. For example, most import plug-ins need some
    information about how to find headers and data lines in
    files. BASE ships with a number of import plug-ins as a part of
    the core plug-ins package, cf. <xref linkend="coreplugins.import"
    />. The core plug-in section links to configuration examples for
    some of the plugins. Go to
    <menuchoice>
      <guimenu>Administrate</guimenu>
      <guimenuitem>Plug-ins &amp; extensions</guimenuitem>
      <guisubmenu>Plug-in definitions</guisubmenu>
    </menuchoice>
    to check which plug-ins are installed on your BASE server. When
    BASE finds a plug-in that supports import of a certain type of
    item an &gbImport; button is displayed in the toolbar on either
    the list view or the single-item view.
  </para>
    <note>
    <title>No "Import" button?</title>
    <para>
      If the import button is missing from a page were you would expect
      to find them this usually means that:
    </para>
    <itemizedlist>
      <listitem>
        <simpara>
          The logged in user does not have permission to use the plug-in.
        </simpara>
      </listitem>
      <listitem>
        <simpara>
          The plug-in requires a configuration, but no one has been
          created or the logged in user does not have permission to 
          use any of the existing configurations.
        </simpara>
      </listitem>
    </itemizedlist>
    <para>
      Contact the server administrator or a similar user that has permission to 
      administrate the plug-ins.
    </para>
  </note>

    <sect1 id="import_data.import">
      <title>General import procedure</title>

    <para>
      Starting a data import is done by a wizard-like interface. There
      are a number of step you have to go through:
    </para>
    
    <orderedlist>
      <listitem>
        <simpara>
          Select a plug-in and file format to use, or use the
          <emphasis>auto detect</emphasis> option.
        </simpara>
      </listitem>
      <listitem>
        <simpara>
          If you selected the auto detection function, you must select
          a file to use.
        </simpara>
      </listitem>
      <listitem>
        <simpara>
          Specify plug-in parameters.
        </simpara>
      </listitem>
      <listitem>
        <simpara>
          Add the import job to the job queue.
        </simpara>
      </listitem>
      <listitem>
        <simpara>
          Wait for the job to finish.
        </simpara>
      </listitem>
    </orderedlist>

    <sect2 id="import_export_data.import.plugin_fileformat">
      <title>Select plug-in and file format</title>
      <para>
        Click on the &gbImport; button
        in the toolbar to start the import wizard. The first step is to
        select which plug-in and, if supported, which 
        file format to use. There is also an <guilabel>auto detect</guilabel>
        option that lets you select a file and have BASE try to find a suitable
        plug-in/file format to use. 
      </para>
  
      <figure id="import_export_data.figures.select_import_plugin">
        <title>Select plug-in and file format</title>
        <screenshot>
          <mediaobject>
            <imageobject><imagedata fileref="figures/select_import_plugin.png" format="PNG" /></imageobject>
          </mediaobject>
        </screenshot>
      </figure>
    

      <helptext external_id="import.selectplugin"
        title="Select plug-in and file format for data import">

        <variablelist>
          <varlistentry>
            <term><guilabel>Plugin + file format</guilabel></term>
            <listitem>
              <para>
                This is a combined list of plug-ins and their 
                respective file format configurations. The list only 
                includes combinations that
                the logged in user has permission to use. If you select
                an entry a short description about the plug-in and configuration
                is displayed
                below the lists. More information about the plug-ins can
                be found under the menu choices
                <menuchoice>
                  <guimenu>Administrate</guimenu>
                  <guimenuitem>Plug-ins &amp; extensions</guimenuitem>
                  <guisubmenu>Plug-in definitions</guisubmenu>
                </menuchoice>
                and 
                <menuchoice>
                  <guimenu>Administrate</guimenu>
                  <guimenuitem>Plug-ins &amp; extensions</guimenuitem>
                  <guisubmenu>Plug-in configuration</guisubmenu>
                </menuchoice>
              </para>
              <note>
                <title>File format vs. Configuration</title>
                <simpara>
                A file format is the same thing as a plug-in configuration.
                It may be confusing that the interface sometimes use
                <emphasis>file format</emphasis> and sometimes use 
                <emphasis>configuration</emphasis>, but for now, we'll have
                to live with it.
                </simpara>
              </note>
            </listitem>
          </varlistentry>
        </variablelist>

        <para>
          Proceed to the next step by clicking on the
          &gbNext; button.
        </para>

        <seeother>
          <other external_id="import.autodetect">The auto detect function</other>
        </seeother>
      </helptext>

      <sect3 id="import_export_data.import.plugin_fileformat.autodetect">
        <title>The auto detect function</title>
        
        <helptext
          external_id="import.autodetect"
          title="The auto detect function">
        
        <para>
          The auto detect function lets you select a file and have
          BASE try to find a suitable plug-in and file format. This option is
          selected by default in the combined plug-in and file format list when there is
          at least one plug-in that supports auto detection.
        </para>
        <note>
          <title>Support of auto detect</title>
          <para>
            Not all plug-ins support auto detection. The ones that do are marked in
            the list with <guilabel>×</guilabel>.
          </para>
        </note>
        
        <para>
          Select the <guilabel>auto detect (all)</guilabel> option to search for a file format
          in all plug-ins that supports the feature, or select the <guilabel>auto detect (plugin)</guilabel>
          option to only search the file formats for a specific plug-in.
          Continue to the next step by clicking on the &gbNext; button.
        </para>
        
        <seeother>
          <other external_id="import.selectplugin">Select plug-in and file format for data import</other>
          <other external_id="import.autodetect.selectfile">Select file for auto detection</other>
        </seeother>
      
        </helptext>
        
        <para>
          You must now select a file to import from.
        </para>
        
        <figure id="import_export_data.figures.select_autodetect_file">
          <title>Select file for auto detection</title>
          <screenshot>
            <mediaobject>
              <imageobject><imagedata fileref="figures/select_autodetect_file.png" format="PNG" /></imageobject>
            </mediaobject>
          </screenshot>
        </figure>
        
        <helptext external_id="import.autodetect.selectfile" 
          title="Select file for auto detection">
  
          <variablelist>
            <varlistentry>
              <term><guilabel>Plugin</guilabel></term>
              <listitem>
                <para>
                  Displays the selected plug-in or <guilabel>all</guilabel> if the 
                  auto-detection is used on all supporting plug-ins.
                </para>
              </listitem>
            </varlistentry>
            <varlistentry>
              <term><guilabel>File</guilabel></term>
              <listitem>
                <para>
                  Enter the path and file name for the
                  file you want to use. Use the <guibutton>Browse&hellip;</guibutton>
                  button to browse after the file in BASE's file system. 
                  If the file does not exist in the file system you have the option
                  to upload it. 
                  <nohelp>Read more about this in <xref linkend="file_system" />.</nohelp>
                </para>
              </listitem>
            </varlistentry>
            <varlistentry>
              <term><guilabel>Character set</guilabel></term>
              <listitem>
                <para>
                  The character set used in text files. If the selected file has been configured
                  with a character set the correct option is automatically selected. In all 
                  cases, you have the option to override the default selection. Most files,
                  typically use either the UTF-8 or ISO-8859-1 character set.
                </para>
              </listitem>
            </varlistentry>
            <varlistentry>
              <term><guilabel>Recently used</guilabel></term>
              <listitem>
                <para>
                  A list of files you have recently used
                  for auto detection.
                </para>
              </listitem>
            </varlistentry>
          </variablelist>
          
          <para>
            Click on the &gbNext; button
            to start the auto detection. There are three possible outcomes:
          </para>

          <itemizedlist>
            <listitem>
              <para>
              Exactly one matching plug-in and file format is found. The next step is
              to configure any additional parameters needed
              by the plug-in. This is the same step as if you had selected
              the same plug-in and file format in the first step.
              </para>
            </listitem>
            <listitem>
              <para>
              If no matching plug-in and file format is found an error message
              is displayed. If logged in with enough permissions to do so there
              is an option to create a new file format/configuration.
              </para>
            </listitem>
            <listitem>
              <para>
              If multiple matching plug-ins and file formats are found 
              you will be taken back to the first step. This time
              the lists will only include the matching plug-ins/file formats
              and the auto detect option is not present.
              </para>
            </listitem>
          </itemizedlist>

          <seeother>
            <other external_id="import.selectplugin">Select plug-in and file format for data import</other>
            <other external_id="import.autodetect">The auto detect function</other>
          </seeother>
          
        </helptext>
        
      </sect3>

    </sect2>

    <sect2 id="import_export_data.import.pluginparameters">
      <title>Specify plug-in parameters</title>
      <para>
        When you have selected a plug-in and file format or used
        the auto detect function to find one, a form where you
        you can enter additional parameters for the plug-in is displayed.
      </para>
      
      <figure id="import_export_data.figures.configure_plugin">
        <title>Specify plug-in parameters</title>
        <screenshot>
          <mediaobject>
            <imageobject>
              <imagedata 
                scalefit="1" width="100%"
                fileref="figures/plugin_parameters.png" format="PNG" />
            </imageobject>
          </mediaobject>
        </screenshot>
      </figure>
      
      <helptext external_id="runplugin.configure.import" 
        title="Specify plug-in parameters">
      <para>
        &runplugin.configure.common;
      </para>
      
      <para>
        The parameter list is very different from plug-in to plug-in.
        Common parameters for import plug-ins are:
      </para>
      
      <variablelist>
        <varlistentry>
          <term><guilabel>File</guilabel></term>
          <listitem>
            <para>
            The file to import data from. A value is already set if
            you used the auto detect function.
            </para>
          </listitem>
        </varlistentry>
        
        <varlistentry>
          <term><guilabel>File parser regular expressions</guilabel></term>
          <listitem>
            <para>
            Various regular expressions that are used when parsing the file
            to ensure that the data is found. In most cases, all values
            are taken from the matched configuration and can be left as is.
            </para>
          </listitem>
        </varlistentry>
        
        <varlistentry>
          <term><guilabel>Error handling</guilabel></term>
          <listitem>
            <para>
              A section which contains different options how to
              handle errors when parsing the file. Normally you can
              select if the import should fail as a whole or if
              only the line with the error should be skipped.
            </para>
          </listitem>
        </varlistentry>
      </variablelist>
      
      <para>
        Continue to the next step by clicking the 
        &gbNext; button.
      </para>
      
      <seeother>
        <other external_id="runplugin.configure">The plug-in configuration wizard</other>
      </seeother>   
      </helptext>

    </sect2>
    
    <sect2 id="import_export_data.import.jobqueue">
      <title>Add the import job to the job queue</title>

      <figure id="import_export_data.figures.finish_job">
        <title>Job name and options</title>
        <screenshot>
          <mediaobject>
            <imageobject>
              <imagedata 
                fileref="figures/finish_job.png" format="PNG" />
            </imageobject>
          </mediaobject>
        </screenshot>
      </figure>
      
      <helptext external_id="runplugin.finshjob" 
        title="Set job name and options">
      <para>
        In this window should information about the job be filled in, like name and
        description. Where name is required and need to have valid string as a value. There
        are also two check boxes in this page.
        <variablelist>
          <varlistentry>
            <term>
              <guilabel>Name</guilabel>
            </term>
            <listitem>
              <para>
                Most plug-ins should suggest a name for the job, but you can change it if
                you want to.
              </para>
            </listitem>
          </varlistentry>
          <varlistentry>
            <term>
              <guilabel>Use job agent</guilabel>
            </term>
            <listitem>
              <para>
                This option is only available if the BASE system has been configured with
                job agents and the logged in user has <constant>SELECT_JOBAGENT</constant>
                permission. Select the <guilabel>automatic</guilabel> option to let
                BASE automatically select a job agent or select a specific option
                to force the use of that particular job agent.
              </para>
            </listitem>
          </varlistentry>
          <varlistentry>
            <term>
              <guilabel>Send message</guilabel>
            </term>
            <listitem>
              <para>
                Tick this check box if the job should send you a message when it is
                finished, otherwise untick it
              </para>
            </listitem>
          </varlistentry>
          <varlistentry>
            <term>
              <guilabel>Remove job</guilabel>
            </term>
            <listitem>
              <para>
                If this check box is ticked, the job will be marked as removed when
                it is finished, on condition that it was finished successfully. This
                is only available for import- and export- plugins.
              </para>
            </listitem>
          </varlistentry>
        </variablelist>
      </para>
      <para>
        Clicking on
        &gbFinish;
        when everything is set will end the job configuration and place the job in the job queue. 
        A self-refreshing window appears with information about the
        job's status and execution time. How long time it takes before the job starts to run
        depends on which priority it and the other jobs in the queue have. The job does not
        depend on the status window to be able to run and the window can be
        closed without interrupting the execution.
      </para>
      <tip>
        <title>View job status</title>
        <para>
          A job's status can be viewed at any time by opening it from the job list page,
          <menuchoice>
            <guimenuitem>View</guimenuitem>
            <guimenuitem>Jobs</guimenuitem>
          </menuchoice>.
        </para>
      </tip>
      </helptext>
    </sect2>

    </sect1>

    <sect1 id="import_data.batch">
      <title>Batch import of data</title>

      <para>
        There are in general several possibilities to import data into
        BASE. Bulk data such as reporter information and raw data
        imports are handled by plug-ins created for these tasks. For
        item types that are imported in more moderate quantities a
        suite of batch item importers available
        (<xref linkend="coreplugins.import.batch" />). These importers
        allows the user to create new items in BASE and define item
        properties and associations between items using tab-separated
        (or equivalent) files.
      </para>

      <para>
        The batch importers are available for most users and they may
        have been pre-configured but there is no requirement to
        configure the batch importer plug-ins. Here we assume that no
        plug-in configuration exists for the batch
        importers. Pre-configuration of the importers is really only
        needed for facilities that perform the same imports regularly
        whereas for occasional use the provided wizard is
        sufficient. Configuring the importers follows the route
        described in <xref linkend="plugins.configuration" />.
      </para>

      <para>
        The batch importers either creates new items or updates
        already existing items. In either mode the plugin can set
        values for
        <itemizedlist>
          <listitem>
            <para>
              Simple properties, <emphasis>eg.</emphasis>, string
              values, numeric values, dates, etc.
            </para>
          </listitem>
          <listitem>
            <para>
              Single-item references, <emphasis>eg.</emphasis>,
              protocol, label, software, owner, etc.
            </para>
          </listitem>
          <listitem>
            <para>
              Multi-item references are references to several other
              items of the same type. The extracts of a
              physical bioassay or pooled samples are two examples of
              items that refer to several other items; a physical bioassay
              may contain several extracts and a sample may be
              a pool of several samples. In some cases a multi-item
              reference is bundled with simple
              values, <emphasis>eg.</emphasis>, used quantity of a
              source biomaterial, the position an extract is
              used on, etc. Multi-item references are never removed by
              the importer, only added or updated. Removing an item
              from a multi-item reference is a manual procedure to be
              done using the web interface.
            </para>
          </listitem>
        </itemizedlist>
        The batch importers do not set values for annotations since
        this is handled by the annotation importer
        plug-in (<xref linkend="annotations.massimport" />). However,
        the annotation importer and batch item importers have similar
        behaviour and functionality to minimize the learning cost for
        users.
      </para>

      <para>
        The importer only works with one type of items at each use and can be
        used in a <emphasis>dry-run</emphasis> mode where everything
        is performed as if a real import is taking place, but the work
        (transaction) is not committed to the database. The result of
        the test can be stored to a log file and the user can examine
        the output to see how an actual import would perform. Summary
        results such as the number of items imported and the number of
        failed items are reported after the import is finished, and in
        the case of non-recoverable failure the reason is reported.
      </para>

      <sect2 id="import_data.batch.fileformat">
        <title>File format</title>

        <para>
          For proper and efficient use of the batch importers users
          need to understand how the files to be imported should be
          formatted. The input file must be organised into columns separated by a
          specified character such as a tab or comma character. The
          data header line contains the column headers which defines
          the contents of each column and defines the beginning of
          item data in the file. The item data block continues until
          the end of the file or to an optional data footer line
          defining the end of the data block.
        </para>

        <para>
          When reading data for an item the plug-in must use some
          information for identifying items. Depending on item type
          there are two or three options to select the item identifier
          <itemizedlist>
            <listitem>
              <para>
                Using the internal <property>id</property>. This is
                always unique for a specific BASE server.
              </para>
            </listitem>
            <listitem>
              <para>
                Using the <property>name</property>. This may or may
                not be unique.
              </para>
            </listitem>
            <listitem>
              <para>
                Some items have
                an <property>externalId</property>. This may or may
                not be unique.
              </para>
            </listitem>
            <listitem>
              <para>
                Array slides may have a <property>barcode</property>
                which is similar to
                the <property>externalId</property>.
              </para>
            </listitem>
          </itemizedlist>
          It is important that the identifier selected
          is <emphasis>unique</emphasis> in the file used, or if the
          file is used to update items already existing in BASE the
          identifier should also be unique in BASE for the user
          performing the update. The plug-in will check uniqueness
          when default parameters are used but the user may change the
          default behaviour.
        </para>

        <para>
          Data for a single item may be split into multiple lines. The
          first line contains simple properties and single-item
          references, and the first multi-item reference. If there are
          more multi-item references they should be on the following
          lines with empty values in all other columns, except for the
          column holding the item identifier. The item identifier must
          have the same value on all lines associated with the
          item. Lines containing other data than multi-item references
          will be ignored or may be considered as an error depending
          on plug-in parameter settings. The reason for treating
          copied data entries as an error is to catch situations where
          two items is given the same item identifier by accident.
        </para>

      </sect2>

      <sect2 id="import_data.batch.running">
        <title>Running the item batch importer</title>

        <para>
          This section discuss specific parameters and features of the
          batch importers. The general use of the batch importers
          follow the description outlined in
          <xref linkend="import_data.import" /> and the setting of
          column mapping parameters is assisted with
          the <guilabel>Test with file</guilabel> function described
          in <xref linkend="plugins.configuration.testwithfile"
          />. The column headers are mapped to item properties at each
          use of the plug-in but, as pointed out above, they can also
          be predefined by saving settings as a plug-in
          configuration. The configuration also includes separator
          character and other information that is needed to parse
          files. The ability to save configurations depends on user
          credential and is by default only granted to administrators.
        </para>

        <para>
          The plug-in parameter follows the standard BASE plug-in
          layout and shows help information for selected
          parameters. The list below comments on some of the
          parameters available.
        </para>
        
          <variablelist>
            <varlistentry>
              <term>
                <guilabel>Mode</guilabel>
              </term>
              <listitem>
                <para>
                  Select the mode of the plug-in. The plug-in can
                  create new items and/or update items already
                  existing in BASE. This setting is available to allow
                  the user to make a conscious choice of how to treat
                  missing or already existing items. For example, if
                  the user selects to only update items already
                  existing the plug-in will complain if an item in the
                  file does not exist in BASE (using default error
                  condition treatment). This adds an extra layer of
                  security and diagnostics for the user during import.
                </para>
              </listitem>
            </varlistentry>
            <varlistentry>
              <term>
                <guilabel>Data directory</guilabel>
              </term>
              <listitem>
                <para>
                  This option is only available for items that has support for
                  attaching files (eg. array design, derived bioassay, etc.).
                  This setting is used to resolve file references that doesn't
                  include a complete absolute path.
                </para>
              </listitem>
            </varlistentry>
            <varlistentry>
              <term>
                <guilabel>Identification method</guilabel>
              </term>
              <listitem>
                <para>
                  This parameter defines the method to use to find
                  already existing items. The parameter can only be
                  set to a set of item properties listed in the
                  plug-in parameter dialog. The property selected by
                  the user must be mapped to a column in the file. If
                  it is not set there is obviously no way for the
                  plug-in to identify if an item already exists.
                </para>
              </listitem>
            </varlistentry>
            <varlistentry>
              <term>
                <guilabel>Item subtypes</guilabel>
              </term>
              <listitem>
                <para>
                  Only look for existing items among the selected subtypes. If no subtype
                  is selected all items are searched. If exactly one subtype is selected
                  new items are automatically created with this subtype (unless it is overridden
                  by specific subtype values in the import file). 
                </para>
              </listitem>
            </varlistentry>
            <varlistentry>
              <term>
                <guilabel>Owned by me</guilabel>, <guilabel>Shared to
                me</guilabel>, <guilabel>In current
                project</guilabel>, and <guilabel>Owned by
                others</guilabel>
              </term>
              <listitem>
                <para>
                  Defines the set of items the plug-in should look in
                  when it checks whether an item already exists. The
                  options are the same that are available in list
                  views and the actual set of parameters depends in
                  user credentials.
                </para>
                <para>
                  When <property>id</property> is used as
                  the <guilabel>Identification method</guilabel>, the
                  plug-in looks for the item irrespective the setting
                  of these parameters. Of course, the user still must
                  have proper access to the item referenced.
                </para>
              </listitem>
            </varlistentry>
            <varlistentry>
              <term>
                <guilabel>Column mapping expressions</guilabel>
              </term>
              <listitem>
                <para>
                  Use the <guilabel>Test with file</guilabel> function
                  described in
                  <xref linkend="plugins.configuration.testwithfile"
                  /> to set the column mapping parameters.
                </para>
                <para>
                  When working with biomaterial items, the 
                  <guilabel>Parent type</guilabel> property is used to
                  tell the plug-in how to find parent items. This only
                  has to be set if the parent item is of the same type
                  as the biomaterial being imported since the default
                  is to look for the nearest parent type in the predefined hierarchy.
                  In ascending order the BASE ordering
                  of <emphasis>parent - child - grandchild -
                  ...</emphasis> item relation is <emphasis>biosource
                  - sample - extract</emphasis>.
                </para>
                <para>
                  The values accepted for <guilabel>Parent type</guilabel>
                  are <constant>BIOSOURCE</constant>,
                  <constant>SAMPLE</constant> or <constant>EXTRACT</constant>.
                  Sometimes all items in a file to be imported have the same parent
                  type but there is no column with this information. This can
                  be resolved by setting
                  the <guilabel>Parent type</guilabel> mapping to a
                  constant string (eg. no backslash '\' character).
                </para>
              </listitem>
            </varlistentry>
            <varlistentry>
              <term>
                <guilabel>Permissions</guilabel>
              </term>
              <listitem>
                <para>
                  This is a column mapping that can be used to update the permissions
                  set on items. Normally, new items are only shared to the active project
                  (if any). By naming a permission template, new items are shared using 
                  the permissions from that template instead. Permissions on already existing 
                  items are merged with the permission from the template.
                </para>
              </listitem>
            </varlistentry>
          </variablelist>
          
          <para>
          After setting the parameters,
          select <guilabel>Next</guilabel>. Another parameter dialog
          will appear where error handling options can be set among
          with 
          </para>
          
          <variablelist>
            <varlistentry>
              <term>
                <guilabel>Log file</guilabel>
              </term>
              <listitem>
                <para>
                  Setting this parameter will turn on logging. The
                  plug-in will give detailed information about how the
                  file is parsed. This is useful for resolving file
                  parsing issues.
                </para>
              </listitem>
            </varlistentry>
            <varlistentry>
              <term>
                <guilabel>Dry run</guilabel>
              </term>
              <listitem>
                <para>
                  Enable or disable test run of the plug-in. If
                  enabled the plug-in will parse and simulate an
                  import. When enabling this option you should set
                  the <guilabel>Log file</guilabel> also. The dry run
                  mode allows testing of large imports and updates by
                  creating a log file that can be examined for
                  inconsistencies before actually performing the action
                  without a safety net.
                </para>
              </listitem>
            </varlistentry>
          </variablelist>


        <para>
          During file parsing the plug-in will look for items
          referenced on each line. There are three outcomes of this
          item search
        </para>
        
          <itemizedlist>
            <listitem>
              <para>
                No item is found. Depending on parameter settings this
                may abort the plug-in, the plug-in may ignore the
                line, or a new item is created.
              </para>
            </listitem>
            <listitem>
              <para>
                One item is found. This is the item that is going to
                be updated.
              </para>
            </listitem>
            <listitem>
              <para>
                More than one item is found. Depending on parameter
                settings this may abort the plug-in or the plug-in may
                ignore the line.
              </para>
            </listitem>
          </itemizedlist>

      </sect2>

      <sect2 id="import_data.batch.comments">
        <title>Comments on the item batch importers</title>

        <para>
          The item batch importers are not designed to change or
          create annotations. There is another plug-in for this, see
          <xref linkend="annotations.massimport" /> for an
          introduction to the annotation importer.
        </para>

        <para>
          There is no need to map all columns when running the
          importer. When new items are created usually the only
          mandatory entry is <property>Name</property>, and when
          running the plug-in in update mode only the column defining
          the item identification property needs to be defined. This
          can be utilized when only one or a few properties needs to
          be updated; map only columns that should be changed and the
          plug-in will ignore the other properties and leave them as
          they are already stored in BASE. This also means that if one
          property should be deleted then that property must be mapped
          and the value must be empty in the file. Note, multi-item
          reference cannot be deleted with the batch importer, and
          deletion of multi-item references must be done using the web
          interface.
        </para>

        <para>
          When parent and other relations are created using the
          plug-in the referenced items are properly linked and
          updated. This means that when a quantity that decreases a
          referenced item is used, the referenced item is updated
          accordingly. In consequence, if the relation is removed in a
          later update - maybe wrong parent was referenced - the
          referenced item is restored and any decrease of quantities
          are also reset.
        </para>

        <para>
          A common mistake is to forget to make sure that some of the
          referenced items already exists in BASE, or at least are
          accessible for the user performing the import. Items such as
          protocols and labels must be added before referencing
          them. This is of course also true for other items but during
          batch import one usually follows the natural order of first
          importing biosources, samples, extracts, and so on. In this
          way the parents are always present and may be referenced
          without any issues.
        </para>

      </sect2>

    </sect1>

</chapter>