source: trunk/doc/src/docbook/developerdoc/api_overview.xml @ 5071

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE chapter PUBLIC 
    "-//Dawid Weiss//DTD DocBook V3.1-Based Extension for XML and graphics inclusion//EN" 
    "../../../../lib/docbook/preprocess/dweiss-docbook-extensions.dtd">
<!--
  $Id: api_overview.xml 5071 2009-08-21 06:17:57Z nicklas $

  Copyright (C) 2007 Peter Johansson, Nicklas Nordborg, Martin Svensson

  This file is part of BASE - BioArray Software Environment.
  Available at http://base.thep.lu.se/

  BASE is free software; you can redistribute it and/or
  modify it under the terms of the GNU General Public License
  as published by the Free Software Foundation; either version 3
  of the License, or (at your option) any later version.

  BASE is distributed in the hope that it will be useful,
  but WITHOUT ANY WARRANTY; without even the implied warranty of
  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
  GNU General Public License for more details.

  You should have received a copy of the GNU General Public License
  along with BASE. If not, see <http://www.gnu.org/licenses/>.
-->

<chapter id="api_overview">
  <?dbhtml dir="api"?>
  <title>API overview (how to use and code examples)</title>

  <sect1 id="api_overview.public_api">
    <title>The Public API of BASE</title>
    
    <para>
      Not all public classes and methods in the <filename>BASE2Core.jar</filename>
      and other JAR files shipped with BASE are considered as
      <emphasis>Public API</emphasis>. This is important knowledge
      since we will always try to maintain backwards compatibility
      for classes that are part of the public API. For other
      classes, changes may be introduced at any time without
      notice or specific documentation. In other words:
    </para>
    
    <note>
      <title>Only use the public API when developing plug-ins</title>
      <para>
        This will maximize the chance that you plug-in will continue
        to work with the next BASE release. If you use the non-public API
        you do so at your own risk.
      </para>
    </note>
    
    <para>
      See the <ulink url="http://base.thep.lu.se/chrome/site/doc/api/index.html"
        >javadoc</ulink> for information about
      what parts of the API that contributes to the public API.
      Methods, classes and other elements that have been tagged as 
      <code>@deprecated</code> should be considered as part of the internal API 
      and may be removed in a subsequent release without warning.
    </para>
    
    <para>
      See <xref linkend="appendix.incompatible" /> to read more about
      changes that have been introduced by each release.
    </para>

    <sect2 id="api_overview.compatibility">
      <title>What is backwards compatibility?</title>
      
      <para>
        There is a great article about this subject on <ulink 
        url="http://wiki.eclipse.org/index.php/Evolving_Java-based_APIs"
          >http://wiki.eclipse.org/index.php/Evolving_Java-based_APIs</ulink>.
        This is what we will try to comply with. If you do not want to
        read the entire article, here are some of the most important points:
      </para>
      
      
      <sect3 id="api_overview.compatibility.binary">
        <title>Binary compatibility</title>
        <para>
        <blockquote>
          Pre-existing Client binaries must link and run with new releases of the 
          Component without recompiling.
        </blockquote>
        
        For example:
        <itemizedlist>
        <listitem>
          <para>
            We cannot change the number or types of parameters to a method
            or constructor.
          </para>
        </listitem>
        <listitem>
          <para>
            We cannot add or change methods to interfaces that are intended
            to be implemented by plug-in or client code.
          </para>
        </listitem>
        </itemizedlist>
        </para>       
      </sect3>
      
      <sect3 id="api_overview.compatibility.contract">
        <title>Contract compatibility</title>
        <para>
          <blockquote>
          API changes must not invalidate formerly legal Client code.
          </blockquote>
        
          For example:
          <itemizedlist>
          <listitem>
            <para>
              We cannot change the implementation of a method to do
              things differently than before. For example, allow <constant>null</constant>
              as a return value when it was not allowed before.
            </para>
          </listitem>
          </itemizedlist>
        
          <note>
            <para>
            Sometimes there is a very fine line between what is considered a
            bug and what is considered a feature. For example, if the
            actual implementation does not do what the javadoc says,
            do we change the code or do we change the documentation?
            This has to be considered from case to case and depends on 
            the age of the code and if we expect plug-ins and clients to be
            affected by it or not.
            </para>
          </note>
        </para>
      </sect3>
      
      <sect3 id="api_overview.compatibility.source">
        <title>Source code compatibility</title>
        <para>
        This is not an important matter and is not always possible to
        achieve. In most cases, the problems are easy to fix.
        Example:
        
        <itemizedlist>
        <listitem>
          <para>
          Adding a class may break a plug-in or client that import
          classes with <constant>.*</constant> if the same class name
          exists in another package.
          </para>
        </listitem>
        </itemizedlist>
        </para>
      </sect3>
    </sect2>
  </sect1>

  <sect1 id="api_overview.data_api" chunked="1">
    <title>The database schema and the Data Layer API</title>

    <para>
      This section gives an overview of the entire data layer API. 
      The figure below show how different modules relate to each other.
    </para>
  
    <figure id="data_api.figures.overview">
      <title>Data layer overview</title>
      <screenshot>
        <mediaobject>
          <imageobject>
            <imagedata 
              align="center"
              scalefit="1" width="100%"
              fileref="figures/uml/datalayer.overview.png" format="PNG" />
          </imageobject>
        </mediaobject>
      </screenshot>
    </figure>

    <sect2 id="data_api.basic">
      <title>Basic classes and interfaces</title>
      
      <para>
        This document contains information about the basic classes and interfaces in this package. 
        They are important since all data-layer classes must inherit from one of the already 
        existing abstract base classes or implement one or more of the
        existing interfaces. They contain code that is common to all classes, 
        for example implementations of the <methodname>equals()</methodname>
        and <methodname>hashCode()</methodname> methods or how to link with the owner of an 
        item.
      </para>
      
      <sect3 id="data_api.basic.uml">
        <title>UML diagram</title>
        
        <figure id="data_api.figures.basic">
          <title>Basic classes and interfaces</title>
          <screenshot>
            <mediaobject>
              <imageobject>
                <imagedata 
                  align="center"
                  fileref="figures/uml/datalayer.basic.png" format="PNG" />
              </imageobject>
            </mediaobject>
          </screenshot>
        </figure>
      </sect3>
      
      <sect3 id="data_api.basic.classes">
        <title>Classes</title>
        
        <variablelist>
        <varlistentry>
          <term><classname docapi="net.sf.basedb.core.data">BasicData</classname></term>
          <listitem>
            <para>
            The root class. It overrides the <methodname>equals()</methodname>, 
            <methodname>hashCode()</methodname> and <methodname>toString()</methodname> methods 
            from the <classname>Object</classname> class. It also defines the 
            <varname>id</varname> and <varname>version</varname> properties.
            All data layer classes must inherit from this class or one of it's subclasses.
            </para>
          </listitem>
        </varlistentry>
        
        <varlistentry>
          <term><classname docapi="net.sf.basedb.core.data">OwnedData</classname></term>
          <listitem>
            <para>
            Extends the <classname>BasicData</classname> class and adds 
            an <varname>owner</varname> property. The owner is a required link to a 
            <classname docapi="net.sf.basedb.core.data">UserData</classname> object, representing the user that
            is the owner of the item.
            </para>
          </listitem>
        </varlistentry>

        <varlistentry>
          <term><classname docapi="net.sf.basedb.core.data">SharedData</classname></term>
          <listitem>
            <para>
            Extends the <classname>OwnedData</classname> class and adds 
            properties (<varname>itemKey</varname> and <varname>projectKey</varname>)
            that holds access permission information for an item.
            Access permissions are held in <classname docapi="net.sf.basedb.core.data">ItemKeyData</classname> and/or 
            <classname docapi="net.sf.basedb.core.data">ProjectKeyData</classname> objects. These objects only exists if 
            the item has been shared.
            </para>
          </listitem>
        </varlistentry>

        <varlistentry>
          <term><classname docapi="net.sf.basedb.core.data">CommonData</classname></term>
          <listitem>
            <para>
            This is a convenience class for items that extends the <classname>SharedData</classname>
            class and implements the <interfacename docapi="net.sf.basedb.core.data">NameableData</interfacename> and 
            <interfacename docapi="net.sf.basedb.core.data">RemoveableData</interfacename> interfaces. This is one of
            the most common situations.
            </para>
          </listitem>
        </varlistentry>

        <varlistentry>
          <term><classname docapi="net.sf.basedb.core.data">AnnotatedData</classname></term>
          <listitem>
            <para>
            This is a convenience class for items that can be annotated. 
            Annotations are held in <classname docapi="net.sf.basedb.core.data">AnnotationSetData</classname> objects. 
            The annotation set only exists if annotations has been created for the item.
            </para>
          </listitem>
        </varlistentry>
        </variablelist>
        
      </sect3>
      
      <sect3 id="data_api.basic.interfaces">
        <title>Interfaces</title>
        
        <variablelist>
        <varlistentry>
          <term><classname docapi="net.sf.basedb.core.data">IdentifiableData</classname></term>
          <listitem>
            <para>
            All items are identifiable, which means that they have a unique <varname>id</varname>. 
            The id is unique for all items of a specific type (ie. class). The id is number 
            that is automatically generated by the database and has no other meaning 
            outside of the application. The <varname>version</varname> property is used for 
            detecting and preventing concurrent modifications to an item. 
            </para>
          </listitem>
        </varlistentry>
        
        <varlistentry>
          <term><classname docapi="net.sf.basedb.core.data">OwnableData</classname></term>
          <listitem>
            <para>
            An ownable item is an item which has an owner. The owner is represented as a 
            required link to a <classname docapi="net.sf.basedb.core.data">UserData</classname> object. 
            </para>
          </listitem>
        </varlistentry>       

        <varlistentry>
          <term><classname docapi="net.sf.basedb.core.data">ShareableData</classname></term>
          <listitem>
            <para>
            A shareable item is an item which can be shared to other users, groups or projects.
            Access permissions are held in <classname docapi="net.sf.basedb.core.data">ItemKeyData</classname> and/or 
            <classname docapi="net.sf.basedb.core.data">ProjectKeyData</classname> objects. 
            </para>
          </listitem>
        </varlistentry>
              
        <varlistentry>
          <term><classname docapi="net.sf.basedb.core.data">NameableData</classname></term>
          <listitem>
            <para>
            A nameable item is an item that has a name (required) and a description
            (optional). The name doesn't have to be unique, except in a few special 
            cases (for example, the name of a file).
            </para>
          </listitem>
        </varlistentry>
        
        <varlistentry>
          <term><classname docapi="net.sf.basedb.core.data">RemovableData</classname></term>
          <listitem>
            <para>
            A removable item is an item that can be flagged as removed. This doesn't 
            remove the information about the item from the database, but can be used by 
            client applications to hide items that the user is not interested in. 
            A trashcan function can be used to either restore or permanently 
            remove items that has the flag set.
            </para>
          </listitem>
        </varlistentry>
                
        <varlistentry>
          <term><classname docapi="net.sf.basedb.core.data">SystemData</classname></term>
          <listitem>
            <para>
            A system item is an item which has an additional id in the form of string. A system id 
            is required when we need to make sure that we can get a specific item without 
            knowing the numeric id. Example of such items are the root user and the everyone group. 
            A system id is generally constructed like: 
            <constant>net.sf.basedb.core.User.ROOT</constant>. The system id:s are defined in the 
            core layer by each item class. 
            </para>
          </listitem>
        </varlistentry>

        <varlistentry>
          <term><classname docapi="net.sf.basedb.core.data">DiskConsumableData</classname></term>
          <listitem>
            <para>
            This interface is used by items which occupies a lot of disk space and 
            should be part of the quota system, for example files. The required 
            <classname docapi="net.sf.basedb.core.data">DiskUsageData</classname> contains information about the size, 
            location, owner etc. of the item.
            </para>
          </listitem>
        </varlistentry>
        
        <varlistentry>
          <term><classname docapi="net.sf.basedb.core.data">AnnotatableData</classname></term>
          <listitem>
            <para>
            This interface is used by items which can be annotated. Annotations are name/value 
            pairs that are attached as extra information to an item. All annotations are
            contained in an <classname docapi="net.sf.basedb.core.data">AnnotationSetData</classname> object. 
            </para>
          </listitem>
        </varlistentry>
        
        <varlistentry>
          <term><classname docapi="net.sf.basedb.core.data">ExtendableData</classname></term>
          <listitem>
            <para>
            This interface is used by items which can have extra administrator-defined 
            columns. The functionality is similar to annotations. It is not as flexible, 
            since it is a global configuration, but has better performance. BASE will
            generate extra database columns to store the data in the tables for items that
            can be extended.
            </para>
          </listitem>
        </varlistentry>
        
        <varlistentry>
          <term><classname docapi="net.sf.basedb.core.data">BatchableData</classname></term>
          <listitem>
            <para>
            This interface is a tagging interface which is used by items that needs batch 
            functionality in the core.
            </para>
          </listitem>
        </varlistentry>
        
        <varlistentry>
          <term><classname docapi="net.sf.basedb.core.data">RegisteredData</classname></term>
          <listitem>
            <para>
            This interface is used by items which registered the date they were
            created in the database. The registration date is set at creation time
            and can't be modified later. Since this didn't exist prior to BASE 2.10,
            null values are allowed on all pre-existing items. Note! For backwards 
            compatibility reasons with existing code in 
            <classname docapi="net.sf.basedb.core.data">BioMaterialEventData</classname>
            the method name is <methodname>getEntryDate()</methodname>.
            </para>
          </listitem>
        </varlistentry>
        
        <varlistentry>
          <term><interfacename docapi="net.sf.basedb.core.data">LoggableData</interfacename></term>
          <listitem>
            <para>
            This is a tagging interface that indicates that the <classname 
            docapi="net.sf.basedb.core.log.db">DbLogManagerFactory</classname> logging
            implementation should log changes made to items that implements it.
            </para>
          </listitem>
        </varlistentry>
        </variablelist>

      </sect3>
    </sect2>
    
    <sect2 id="data_api.authentication">
      <title>User authentication and access control</title>
      
      <para>
         This section gives an overview of user authentication and
         how groups, roles and projects are used for access control
         to items.
      </para>
      
      <sect3 id="data_api.authentication.uml">
        <title>UML diagram</title>
        
        <figure id="data_api.figures.authentication">
          <title>User authentication and access control</title>
          <screenshot>
            <mediaobject>
              <imageobject>
                <imagedata 
                  align="center"
                  scalefit="1" width="100%"
                  fileref="figures/uml/datalayer.authentication.png" format="PNG" />
              </imageobject>
            </mediaobject>
          </screenshot>
        </figure>
      </sect3>
      
      <sect3 id="data_api.authentication.users">
        <title>Users and passwords</title>      
      
        <para>
          The <classname docapi="net.sf.basedb.core.data">UserData</classname> class holds information about users. 
          We keep the passwords in a separate table and use proxies to avoid loading 
          password data each time a user is loaded to minimize security risks. It is
          only if the password needs to be changed that the <classname docapi="net.sf.basedb.core.data">PasswordData</classname>
          object is loaded. The one-to-one mapping between user and password is controlled 
          by the password class, but a cascade attribute on the user class makes sure 
          that the password is deleted when a user is deleted.
        </para>
      </sect3>

      <sect3 id="data_api.authentication.groups">
        <title>Groups, roles and projects</title>     
      
        <para>
          The <classname docapi="net.sf.basedb.core.data">GroupData</classname>, <classname docapi="net.sf.basedb.core.data">RoleData</classname> and 
          <classname docapi="net.sf.basedb.core.data">ProjectData</classname> classes holds information about groups, roles 
          and projects respectively. A user may be a member of any number of groups,
          roles and/or projects. The membership in a project comes with an attached
          permission values. This is the highest permission the user has in the
          project. No matter what permission an item has been shared with the
          user will not get higher permission. Groups may be members of other groups and 
          also in projects.
        </para>
        
        <para>
          Group membership is always accounted for, but the core only allows
          one project at a time to be use, this is the <emphasis>active project</emphasis>.
          When a project is active new items that are created are automatically
          added to that project with the permission given by the 
          <varname>autoPermission</varname> property.
        </para>
              
      </sect3>
      
      <sect3 id="data_api.authentication.keys">
        <title>Keys</title>     
      
        <para>
          The <classname docapi="net.sf.basedb.core.data">KeyData</classname> class and it's subclasses 
          <classname docapi="net.sf.basedb.core.data">ItemKeyData</classname>, <classname docapi="net.sf.basedb.core.data">ProjectKeyData</classname> and 
          <classname docapi="net.sf.basedb.core.data">RoleKeyData</classname>, are used to store information about access 
          permissions to items. To get permission to manipulate an item a user must have 
          access to a key giving that permission. There are three types of keys:
        </para>
        
        <variablelist>
        <varlistentry>
          <term><classname docapi="net.sf.basedb.core.data">ItemKey</classname></term>
          <listitem>
            <para>
            Is used to give a user or group access to a specific item. The item 
            must be a <interfacename docapi="net.sf.basedb.core.data">ShareableData</interfacename> item.
            The permissions are usually set by the owner of the item. Once created an 
            item key cannot be changed. This allows the core to reuse a key if the 
            permissions match exactly, ie. for a given set of users/groups/permissions 
            there can be only one item key object.
            </para>
          </listitem>
        </varlistentry>

        <varlistentry>
          <term><classname docapi="net.sf.basedb.core.data">ProjectKey</classname></term>
          <listitem>
            <para>
            Is used to give members of a project access to a specific item. The item 
            must be a <interfacename docapi="net.sf.basedb.core.data">ShareableData</interfacename> item. Once created a 
            project key cannot be changed. This allows the core to reuse a key if the 
            permissions match exactly, ie. for a given set of projects/permissions 
            there can be only one project key object.
            </para>
          </listitem>
        </varlistentry>

        <varlistentry>
          <term><classname docapi="net.sf.basedb.core.data">RoleKey</classname></term>
          <listitem>
            <para>
            Is used to give a user access to all items of a specific type, ie. 
            <constant>READ</constant> all <constant>SAMPLES</constant>. The installation 
            will make sure that there already exists a role key for each type of item, and 
            it is not possible to add new or delete existing keys. Unlike the other two types 
            this key can be modified.
            </para>
            
            <para>
            A role key is also used to assign permissions to plug-ins. If a plug-in has 
            been specified to use permissions the default is to deny everything. 
            The mapping to the role key is used to grant permissions to the plugin. 
            The <varname>granted</varname> value gives the plugin access to all items 
            of the related item type regardless of if the user that is running the plug-in has the 
            permission or not. The <varname>denied</varname> values denies access to all 
            items of the related item type even if the logged in user has the permission. 
            Permissions that are not granted nor denied are checked against the 
            logged in users regular permissions. Permissions to items that are 
            not linked are always denied.
            </para>
          </listitem>
        </varlistentry>
        </variablelist>
        
      </sect3>

      <sect3 id="data_api.authentication.permissions">
        <title>Permissions</title>
        
        <para>
          The <varname>permission</varname> property appearing in many classes is an 
          integer values describing the permission:
        </para>
        
        <informaltable>
        <tgroup cols="2">
          <colspec colname="value" />
          <colspec colname="permission" />
          <thead>
            <row>
              <entry>Value</entry>
              <entry>Permission</entry>
            </row>
          </thead>
          <tbody>
            <row>
              <entry>1</entry>
              <entry>Read</entry>
            </row>
            <row>
              <entry>3</entry>
              <entry>Use</entry>
            </row>
            <row>
              <entry>7</entry>
              <entry>Restricted write</entry>
            </row>
            <row>
              <entry>15</entry>
              <entry>Write</entry>
            </row>
            <row>
              <entry>31</entry>
              <entry>Delete</entry>
            </row>
            <row>
              <entry>47 (=32+15)</entry>
              <entry>Set owner</entry>
            </row>
            <row>
              <entry>79 (=64+15)</entry>
              <entry>Set permissions</entry>
            </row>
            <row>
              <entry>128</entry>
              <entry>Create</entry>
            </row>
            <row>
              <entry>256</entry>
              <entry>Denied</entry>
            </row>
          </tbody>
        </tgroup>
        </informaltable>
        
        <para>
          The values are constructed so that 
          <constant>READ</constant> -&gt;
          <constant>USE</constant> -&gt;
          <constant>RESTRICTED_WRITE</constant> -&gt;
          <constant>WRITE</constant> -&gt;
          <constant>DELETE</constant>
          are chained in the sense that a higher permission always implies the lower permissions 
          also. The <constant>SET_OWNER</constant> and <constant>SET_PERMISSION</constant>
          both implies <constant>WRITE</constant> permission. The <constant>DENIED</constant>
          permission is only valid for role keys, and if specified it overrides all 
          other permissions.                
        </para>
        
        <para>
          When combining permission for a single item the permission codes for the different 
          paths are OR-ed together. For example a user has a role key with <constant>READ</constant>
          permission for <constant>SAMPLES</constant>, but also an item key with <constant>USE</constant>
          permission for a specific sample. Of course, the resulting permission for that
          sample is <constant>USE</constant>. For other samples the resulting permission is
          <constant>READ</constant>.
        </para>
        
        <para>
          If the user is also a member of a project which has <constant>WRITE</constant>
          permission for the same sample, the user will have <constant>WRITE</constant>
          permission when working with that project.
        </para>
        
        <para>
          The <constant>RESTRICTED_WRITE</constant> permission is in most cases the same 
          as the <constant>WRITE</constant> permission. So far the <constant>RESTRICTED_WRITE</constant>
          permission is only given to users to their own <classname docapi="net.sf.basedb.core.data">UserData</classname>
          object so they can change their address and other contact information, 
          but not quota, expiration date and other administrative information.
        </para>

      </sect3>
    </sect2>

    <sect2 id="data_api.wares">
      <title>Hardware and software</title>
      <para>
         This section gives an overview of hardware and software in BASE.
      </para>
      
      <sect3 id="data_api.wares.uml">
        <title>UML diagram</title>
        
        <figure id="data_api.figures.wares">
          <title>Hardware and software</title>
          <screenshot>
            <mediaobject>
              <imageobject>
                <imagedata 
                  align="center"
                  fileref="figures/uml/datalayer.wares.png" format="PNG" />
              </imageobject>
            </mediaobject>
          </screenshot>
        </figure>
      </sect3>
      
      <sect3 id="data_api.wares.description">
        <title>Hardware and software</title>
        <para>
          BASE is pre-installed with a set of hardware and software types. 
          They are typically used to filter the registered hardware and software
          depending on what a user is doing. For example, when adding raw data
          to BASE a user can select a scanner. The GUI will display the hardware
          that has been registered as <emphasis>scanner</emphasis> hardware types.
          Other hardware types are <emphasis>hybridization station</emphasis>
          and <emphasis>print robot</emphasis>. An administrator may register more
          hardware and software types. 
        </para>
      </sect3>
    </sect2>
    
    <sect2 id="data_api.reporters">
      <title>Reporters</title>
      <para>
         This section gives an overview of hardware and software in BASE.
      </para>
      
      <sect3 id="data_api.reporters.uml">
        <title>UML diagram</title>
        
        <figure id="data_api.figures.reporters">
          <title>Reporters</title>
          <screenshot>
            <mediaobject>
              <imageobject>
                <imagedata 
                  align="center"
                  fileref="figures/uml/datalayer.reporters.png" format="PNG" />
              </imageobject>
            </mediaobject>
          </screenshot>
        </figure>
      </sect3>
      
      <sect3 id="data_api.reporters.description">
        <title>Reporters</title>
        <para>
          The <classname docapi="net.sf.basedb.core.data">ReporterData</classname> class holds information about reporters.
          The <property>externalId</property> is a required property that must be unique
          among all reporters. The external ID is the value BASE uses to match 
          reporters when importing data from files.
        </para>
        
        <para>
          The <classname>ReporterData</classname> is an <emphasis>extendable</emphasis>
          class, which means that the server administrator can define additional
          columns (=annotations) in the reporters table. These are accessed with
          the <methodname>ReporterData.getExtended()</methodname> and
          <methodname>ReporterData.setExtended()</methodname> methods.
          See <xref linkend="appendix.extendedproperties" /> for more information about 
          this.
        </para>
        
        <para>
          The <classname>ReporterData</classname> is also a <emphasis>batchable</emphasis>
          class which means that there is no corresponding class in the core
          layer. Client applications and plug-ins should work directly with 
          the <classname>ReporterData</classname> class. To help manage the reporters
          there is the <classname docapi="net.sf.basedb.core">Reporter</classname> and <classname docapi="net.sf.basedb.core">ReporterBatcher</classname>
          classes. The main reason for this
          is to increase the performance and lower the memory usage by bypassing
          internal caching in the core and Hibernate. Performance is also
          increased by the batchers which uses more efficient SQL against the
          database than Hibernate.
        </para>
        
        <para>
          The
          <property>lastUpdate</property>
          property holds the data and time the reporter information was last updated. The
          value is managed automatically by the
          <classname>ReporterBatcher</classname>
          class. That goes for
          <property>lastSource</property>
          property too, which holds information about where the last update comes from. By
          default this is set to the name of the logged in user, but it can be changed by
          calling
          <methodname>ReporterBatcher.setUpdateSource(String source)</methodname>
          before the batcher commits the updates to the database. The source-string 
          should have the format: <synopsis>[ITEM_TYPE]:[ITEM_NAME]</synopsis> where,in 
          the file-case, ITEM_TYPE is File and ITEM_NAME is the file's name.
        </para>
      </sect3>
      
      <sect3 id="data_api.reporters.lists">
        <title>Reporter lists</title>
        
        <para>
          Reporter lists can be used to group reporters that are somehow related
          to each other. This could for example be a list of interesting reporters
          found in the analysis of an experiment. Each reporter in the list may
          optionally be assigned a score. The meaning of the score value is not
          interpreted by BASE.
        </para>
        
      </sect3>
      
      
    </sect2>

    <sect2 id="data_api.quota">
      <title>Quota and disk usage</title>
      <para>
         This section gives an overview of quota system in BASE
         and how the disk usage is kept track of.
      </para>
      
      <sect3 id="data_api.quota.uml">
        <title>UML diagram</title>
        
        <figure id="data_api.figures.quota">
          <title>Quota and disk usage</title>
          <screenshot>
            <mediaobject>
              <imageobject>
                <imagedata 
                  align="center"
                  fileref="figures/uml/datalayer.quota.png" format="PNG" />
              </imageobject>
            </mediaobject>
          </screenshot>
        </figure>
      </sect3>
      
      <sect3 id="data_api.quota.description">
        <title>Quota</title>
        
        <para>
          The <classname docapi="net.sf.basedb.core.data">QuotaData</classname> holds information about a 
          single quota registration. The same quota may be used by many different users
          and groups. This object encapsulates allowed
          quota values for different types of quota types and locations. 
          BASE defines several quota types (file, raw data and experiment),
          and locations (primary, secondary and offline).
        </para>
        
        <para>
          The <property>quotaValues</property> property is a map from 
          <classname docapi="net.sf.basedb.core.data">QuotaIndex</classname> to maximum byte values. 
          This map must contain at least one entry for the total
          quota at the primary location.
        </para>
        
      </sect3>
      
      <sect3 id="data_api.quota.diskusage">
        <title>Disk usage</title>
        
        <para>
          A <interfacename docapi="net.sf.basedb.core.data">DiskConsumableData</interfacename> (for example a file)
          item is automatically linked to a <classname docapi="net.sf.basedb.core.data">DiskUsageData</classname>
          item. This holds information about the number of bytes,
          the location and quota type the item uses. It also holds information
          about which user and group (optional) that should be charged for the disk usage.
          The user is always the owner of the item.
        </para>

      </sect3>
      
    </sect2>

    <sect2 id="data_api.clients">
      <title>Client, session and settings</title>
      <para>
         This section gives an overview of hardware and software in BASE.
      </para>
      
      <sect3 id="data_api.clients.uml">
        <title>UML diagram</title>
        
        <figure id="data_api.figures.clients">
          <title>Client, sessions and settings</title>
          <screenshot>
            <mediaobject>
              <imageobject>
                <imagedata 
                  align="center"
                  scalefit="1" width="100%"
                  fileref="figures/uml/datalayer.clients.png" format="PNG" />
              </imageobject>
            </mediaobject>
          </screenshot>
        </figure>
      </sect3>
      
      <sect3 id="data_api.clients.description">
        <title>Clients</title>
        <para>
          The <classname docapi="net.sf.basedb.core.data">ClientData</classname> class holds information
          about a client application. The <property>externalId</property>
          property is a unique identifier for the application. To avoid ID clashes the ID
          should be constructed in the same way as Java packages, for example 
          <constant>net.sf.basedb.clients.web</constant> is the ID for the
          web client application. 
        </para>
        
        <para>
          A client application doesn't have to be registered with BASE
          to be able to use it. But we recommend it since:
        </para>
        
        <itemizedlist>
        <listitem>
          <para>
            The permission system allows an admin to specify exactly
            which users that may use a specific application.
          </para>
        </listitem>
        
        <listitem>
          <para>
          The application can't store any context-sensitive or application-specific
          settings unless it is registered.
          </para>
        </listitem>
        
        <listitem>
          <para>
          The application can store context-sensitive help in the BASE
          database.
          </para>
        </listitem>
        </itemizedlist>
      </sect3>
      
      <sect3 id="data_api.clients.sessions">
        <title>Sessions</title>
        
        <para>
          A session represents the time between login and logout for a single
          user. The <classname docapi="net.sf.basedb.core.data">SessionData</classname> object is entirely
          managed by the BASE core, and should be considered read-only
          for client applications.
        </para>
            
      </sect3>
      
      <sect3 id="data_api.clients.settings">
        <title>Settings</title>
        
        <para>
          There are two types of settings: context-sensitive settings and regular 
          settings. The regular settings are simple key-value pairs of strings
          and can be used for almost anything. There are four subtypes:
        </para>
        
        <itemizedlist>
        <listitem>
          <para>
          Global default settings: Settings that are used by all users
          and client applications on the BASE server. These settings
          are read-only except for administrators. BASE has not yet defined
          any settings of this type.
          </para>
        </listitem>
        
        <listitem>
          <para>
          User default settings: Settings that are valid for a single user
          for any client application. BASE has not yet defined
          any settings of this type.
          </para>
        </listitem>
        
        <listitem>
          <para>
          Client default settings: Settings that are valid for all users using
          a specific client application. Each client application is responsible
          for defining it's own settings. Settings are read-only except
          for administrators.
          </para>
        </listitem>
        
        <listitem>
          <para>
          User client settings: Settings that are valid for a single user using
          a specific client application. Each client application is responsible
          for defining it's own settings.
          </para>
        </listitem>
        
        </itemizedlist>
        
        <para>
          The context-sensitive settings are designed to hold information 
          about the current status of options related to the listing of items
          of a specific type. This includes:
        </para>
        
        <itemizedlist>
        <listitem>
          <para>
          Current filtering options (as 1 or more <classname docapi="net.sf.basedb.core.data">PropertyFilterData</classname>
          objects).
          </para>
        </listitem>
        
        <listitem>
          <para>
          Which columns and direction to use for sorting. 
          </para>
        </listitem>
        
        <listitem>
          <para>
          The number of items to display on each page, and which page that
          is the current page.
          </para>
        </listitem>
        
        <listitem>
          <para>
          Simple key-value settings related to a given context.
          </para>
        </listitem>
        </itemizedlist>
        
        <para>
          Context-sensitive settings are only accessible if a client
          application has been registered. The settings may be
          named to make it possible to store several presets and to
          quickly switch between them. In any case, BASE maintains a
          current default setting with an empty name. An administrator
          may mark a named setting as public to allow other users to 
          use it.
        </para>
        
      </sect3>
      
      
    </sect2>

    <sect2 id="data_api.files">
      <title>Files and directories</title>

      <para>
        This section covers the details of the BASE file
        system.
      </para>

      <sect3 id="data_api.files.uml">
      <title>UML diagram</title>
      
        <figure id="data_api.figures.files">
          <title>Files and directories</title>
          <screenshot>
            <mediaobject>
              <imageobject>
                <imagedata 
                  align="center"
                  fileref="figures/uml/datalayer.files.png" format="PNG" />
              </imageobject>
            </mediaobject>
          </screenshot>
        </figure>
      </sect3>
      
      <sect3 id="data_api.files.description">
        <title>Description</title>
        
        <para>
          The <classname docapi="net.sf.basedb.core.data">DirectoryData</classname> class holds
          information about directories. Directories are organised in the
          ususal way as as tree structure. All directories must have
          a parent directory, except the system-defined root directory.
        </para>
        
        <para>
          The <classname docapi="net.sf.basedb.core.data">FileData</classname> class holds information about
          a file. The actual file contents is stored on disk in the directory
          specified by the <varname>userfiles</varname> setting in 
          <filename>base.config</filename>. The <varname>internalName</varname>
          property is the name of the file on disk, but this is never exposed to 
          client applications. The filenames and directories
          on the disk doesn't correspond to the the filenames and directories in 
          BASE.
        </para>
        
        <para>
          The <varname>location</varname> property can take three values:
        </para>
        
        <itemizedlist>
        <listitem>
          <para>
          0 = The file is offline, ie. there is no file on the disk
          </para>
        </listitem>
        <listitem>
          <para>
          1 = The file is in primary storage, ie. it is located on the disk
          and can be used by BASE
          </para>
        </listitem>
        <listitem>
          <para>
          2 = The file is in secondary storage, ie. it has been moved to some
          other place and can't be used by BASE immediately.
          </para>
        </listitem>
        </itemizedlist>
        
        <para>
          The <varname>action</varname> property controls how a file is
          moved between primary and seconday storage. It can have the following
          values:
        </para>
        
        <itemizedlist>
        <listitem>
          <para>
          0 = Do nothing
          </para>
        </listitem>
        <listitem>
          <para>
          1 = If the file is in secondary storage, move it back to the primary storage
          </para>
        </listitem>
        <listitem>
          <para>
          2 = If the file is in primary storage, move it to the secondary storage
          </para>
        </listitem>
        </itemizedlist>
        
        <para>
          The actual moving between primary and secondary storage is done by an 
          external program. See 
          <xref linkend="appendix.base.config.secondary" /> and 
          <xref linkend="plugin_developer.other.secondary" /> for more information.
        </para>
      
        <para>
          The <varname>md5</varname> property can be used to check for file
          corruption when it is moved between primary and secondary storage or
          when a user re-uploads a file that has been offline.
        </para>
        
        <para>
          BASE can store files in a compressed format. This is handled internally
          and is not visible to client applications. The <varname>compressed</varname>
          and <varname>diskSize</varname> properties are used to store information
          about this. A file may always be compressed if the users says so, but
          BASE can also do this automatically if the file is uploaded
          to a directory with the <varname>autoCompress</varname> flag set
          or if the file has MIME type with the <varname>autoCompress</varname>
          flag set.
        </para>
        
        <para>
          The <classname docapi="net.sf.basedb.core.data">FileTypeData</classname> class holds information about 
          file types. It is used only to make it easier for users to organise 
          their files.
        </para>
        
        <para>
          The <classname docapi="net.sf.basedb.core.data">MimeTypeData</classname> is used to register mime types and
          map them to file extensions. The information is only used to lookup values 
          when needed. Given the filename we can set the <varname>File.mimeType</varname>
          and <varname>File.fileType</varname> properties. The MIME type is also
          used to decide if a file should be stored in a compressed format or not.
          The extension of a MIME type must be unique. Extensions should be registered 
          without a dot, ie <emphasis>html</emphasis>, not <emphasis>.html</emphasis>.  
        </para>
        
      </sect3>
      
      
    </sect2>
    
    <sect2 id="data_api.platforms">
      <title>Experimental platforms</title>

      <para>
         This section gives an overview of experimental platforms
         and how they are used to enable data storage in files
         instead of in the database.
      </para>
      
      <itemizedlist>
        <title>See also</title>
        <listitem><xref linkend="core_api.data_in_files" /></listitem>
        <listitem><xref linkend="appendix.rawdatatypes.platforms" /></listitem>
        <listitem><xref linkend="plugin_developer.other.datafiles" /></listitem>
      </itemizedlist>
          
      <sect3 id="data_api.platforms.uml">
        <title>UML diagram</title>
        
        <figure id="data_api.figures.platforms">
          <title>Experimental platforms</title>
          <screenshot>
            <mediaobject>
              <imageobject>
                <imagedata 
                  align="center"
                  fileref="figures/uml/datalayer.platforms.png" format="PNG" />
              </imageobject>
            </mediaobject>
          </screenshot>
        </figure>
      </sect3>
      
      <sect3 id="data_api.platforms.platforms">
        <title>Platforms</title>
        
        <para>
          The <classname docapi="net.sf.basedb.core.data">PlatformData</classname> holds information about a
          platform. A platform can have one or more <classname docapi="net.sf.basedb.core.data">PlatformVariant</classname>:s.
          Both the platform and variant are identified by an external ID that
          is fixed and can't be changed. <emphasis>Affymetrix</emphasis>
          is an example of a platform.
          If the <varname>fileOnly</varname> flag is set data for the platform
          can only be stored in files and not imported into the database. If
          the flag is not set data can be imported into the database.
          In the latter case, the <varname>rawDataType</varname> property
          can be used to lock the platform
          to a specific raw data type. If the value is <constant>null</constant>
          the platform can use any raw data type. 
        </para>
        
        <para>
          Each platform and it's variant can be connected to one or more
          <classname docapi="net.sf.basedb.core.data">DataFileTypeData</classname> items. This item 
          describes the kind of files that are used to hold data for
          the platform and/or variant. The file types are re-usable between
          different platforms and variants. Note that a file type may be attached
          to either only a platform or to a platform with a variant. File 
          types attached to platforms are inherited by the variants. The variants
          can only define additional file types, not remove or redefine file types
          that has been attached to the platform.
        </para>
        <para>
          The file type is also identified
          by a fixed, non-changable external ID. The <varname>itemType</varname>
          property tells us what type of item the file holds data for (ie. 
          array design or raw bioassay). It also links to a <classname docapi="net.sf.basedb.core.data">FileType</classname>
          which is the generic type of data in the file. This allows us to query
          the database for, as an example, files with the generic type
          <constant>FileType.RAW_DATA</constant>. If we are in an Affymetrix 
          experiment we will get the CEL file, for another platform we will
          get another file.
        </para>
        <para>
          The <varname>required</varname> flag in <classname docapi="net.sf.basedb.core.data">PlatformFileTypeData</classname>
          is used to signal that the file is a required file. This is not 
          enforeced by the core. It is intended to be used by client applications
          for creating a better GUI and for validation of an experiment.
        </para>

      </sect3>
      
      <sect3 id="data_api.platforms.files">
        <title>FileStoreEnabled items and data files</title>
        
        <para>
          An item must implement the <interfacename docapi="net.sf.basedb.core">FileStoreEnabledData</interfacename>
          interface to be able to store data in files instead of in the database.
          The interface creates a link to a <classname docapi="net.sf.basedb.core.data">FileSetData</classname> object,
          which can hold several <classname docapi="net.sf.basedb.core.data">FileSetMemberData</classname> items.
          Each member points to specific <classname docapi="net.sf.basedb.core.data">FileData</classname> item.
          A file set can only store one file of each <classname docapi="net.sf.basedb.core.data">DataFileTypeData</classname>.
        </para>
        
      </sect3>
    </sect2>

    <sect2 id="data_api.parameters">
      <title>Parameters</title>
      
      <para>
        This section gives an overview the generic parameter
        system in BASE that is used to store annotation values,
        plugin configuration values, job parameter values, etc.
      </para>
      
      <sect3 id="data_api.parameters.uml">
        <title>UML diagram</title>
        
        <figure id="data_api.figures.parameters">
          <title>Parameters</title>
          <screenshot>
            <mediaobject>
              <imageobject>
                <imagedata 
                  align="center"
                  fileref="figures/uml/datalayer.parameters.png" format="PNG" />
              </imageobject>
            </mediaobject>
          </screenshot>
        </figure>
      </sect3>
      
      <sect3 id="data_api.parameters.description">
        <title>Parameters</title>
        
        <para>
          The parameter system is a generic system that can store almost
          any kind of simple values (string, numbers, dates, etc.) and
          also links to other items. The <classname docapi="net.sf.basedb.core.data">ParameterValueData</classname> 
          class is an abstract base class that can hold multiple values (all must be of the
          same type). Unless only a specific type of values should be stored, this is 
          the class that should be used when creating references for storing parameter
          values. It makes it possible for a single relaltion to use any kind of
          values or for a collection reference to mix multiple types of values.
          A typical use case maps a <classname>Map</classname> with the 
          parameter name as the key:
        </para>
        
        <programlisting language="java">
private Map&lt;String, ParameterValueData&lt;?&gt;&gt; configurationValues;
/**
   Link parameter name with it's values.
   @hibernate.map table="`PluginConfigurationValues`" lazy="true" cascade="all"
   @hibernate.collection-key column="`pluginconfiguration_id`"
   @hibernate.collection-index column="`name`" type="string" length="255"
   @hibernate.collection-many-to-many column="`value_id`" 
      class="net.sf.basedb.core.data.ParameterValueData"
*/
public Map&lt;String, ParameterValueData&lt;?&gt;&gt; getConfigurationValues()
{
   return configurationValues;
}
void setConfigurationValues(Map&lt;String, ParameterValueData&lt;?&gt;&gt; configurationValues)
{
   this.configurationValues = configurationValues;
}
</programlisting>
        
      <para>
      Now it is possible for the collection to store all types of values:
      </para>
      
      <programlisting language="java">
Map&lt;String, ParameterValueData&lt;?&gt;&gt; config = ...
config.put("names", new StringParameterValueData("A", "B", "C"));
config.put("sizes", new IntegerParameterValueData(10, 20, 30));

// When you later load those values again you have to cast 
// them to the correct class.
List&lt;String&gt; names = (List&lt;String&gt;)config.get("names").getValues();
List&lt;Integer&gt; sizes = (List&lt;Integer&gt;)config.get("sizes").getValues();
</programlisting>

      </sect3>
      
    </sect2>

    <sect2 id="data_api.annotations">
      <title>Annotations</title>
      
      <para>
        This section gives an overview of how the BASE annotation
        system works.
      </para>
      
      <sect3 id="data_api.annotations.uml">
        <title>UML diagram</title>
        
        <figure id="data_api.figures.annotations">
          <title>Annotations</title>
          <screenshot>
            <mediaobject>
              <imageobject>
                <imagedata 
                  align="center"
                  fileref="figures/uml/datalayer.annotations.png" format="PNG" />
              </imageobject>
            </mediaobject>
          </screenshot>
        </figure>
      </sect3>
      
      <sect3 id="data_api.annotations.description">
        <title>Annotations</title>
        
        <para>
        An item must implement the <interfacename docapi="net.sf.basedb.core.data">AnnotatableData</interfacename>
        interface to be able to use the annotation system. This interface gives
        a link to a <classname docapi="net.sf.basedb.core.data">AnnotationSetData</classname> item. This class
        encapsulates all annotations for the item. There are two types of
        annotations:
        </para>
        
        <itemizedlist>
        <listitem>
          <para>
          <emphasis>Primary annotations</emphasis> are annotations that
          explicitely belong to the item. An annotation set can contain
          only one primary annotation of each annotation type. The primary
          annotation are linked with the <property>annotations</property>
          property. This property is a map with an 
          <classname docapi="net.sf.basedb.core.data">AnnotationTypeData</classname>  as the key.
          </para>
        </listitem>
        
        <listitem>
          <para>
          <emphasis>Inherited annotations</emphasis> are annotations
          that belong to a parent item, but that we want to use on 
          another item as well. Inherited annotations are saved as
          references to either a single annotation or to another
          annotation set. Thus, it is possible for an item to inherit
          multiple annotations of the same annotation type.
          </para>
        </listitem>
        </itemizedlist>
        
        <para>
          The <classname docapi="net.sf.basedb.core.data">AnnotationData</classname> class is also
          just a placeholder. It connects the annotation set and
          annotation type with a <classname docapi="net.sf.basedb.core.data">ParameterValueData</classname>
          object. This is the object that holds the actual annotation
          values.
        </para>
        
      </sect3>
      
      <sect3 id="data_api.annotations.types">
        <title>Annotation types</title>
        
        <para>
        Instances of the <classname docapi="net.sf.basedb.core.data">AnnotationTypeData</classname> class
        defines the various annotations. It must have a <property>valueType</property> 
        property which cannot be changed. The value of this property controls 
        which <classname docapi="net.sf.basedb.core.data">ParameterValueData</classname> subclass is used to store 
        the annotation values, ie. <classname docapi="net.sf.basedb.core.data">IntegerParameterValueData</classname>, 
        <classname docapi="net.sf.basedb.core.data">StringParameterValueData</classname>, etc. 
        The <property>multiplicity</property> property holds the maximum allowed 
        number of values for an annotation, or 0 if an unlimited number is 
        allowed.
        </para>
        
        <para>
        The <property>itemTypes</property> collection holds the codes for 
        the types of items the annotation type can be used on. This is 
        checked when new annotations are created but already existing 
        annotations are not affected if the collection is modified.
        </para>
        
        <para>
        Annotation types with the <property>protocolParameter</property> flag set 
        are treated a bit differently. They will not show up as annotations 
        to items with a type found in the <property>itemTypes</property> collection.
        A protocol parameter should be attached to a protocol. Then, when an item 
        is using that protocol it becomes possible to add annotation values for 
        the annotation types specified as protocol parameters. It doesn't matter 
        if the item's type is found in the <property>itemTypes</property> 
        collection or not.
        </para>
        
        <para>
        The <property>options</property> collection is used to store additional 
        options required by some of the value types, for example a max string 
        length for string annotations or the max and min allowed value for 
        integer annotations.
        </para>
        
        <para>
        The <property>enumeration</property> property is a boolean flag 
        indicating if the allowed values are predefined as an enumeration. 
        In that case those values are found in the <property>enumerationValues</property>
        property. The actual subclass is determined by the <property>valueType</property>
        property.
        </para>
        
        <para>
        Most of the other properties are hints to client applications how 
        to render the input field for the annotation. 
        </para>
        
      </sect3>
      
      <sect3 id="data_api.annotations.units">
        <title>Units</title>
        <para>
        Numerical annotation values can have units. A unit is described by 
        a <classname docapi="net.sf.basedb.core.data">UnitData</classname> object. 
        Each unit belongs to a <classname docapi="net.sf.basedb.core.data">QuantityData</classname> 
        object which defines the class of units. For example, if the quantity is 
        <emphasis>weight</emphasis>, we can have units, <emphasis>kg</emphasis>, 
        <emphasis>mg</emphasis>, <emphasis>µg</emphasis>, etc. The <classname>UnitData</classname>
        contains a factor and offset that relates all units to a common reference
        defined by the <classname>QuantityData</classname> class. For example,
        <emphasis>1 meter</emphasis> is the reference unit for distance, and we
        have <code>1 meter * 0.001 = 1 millimeter</code>. In this case, the factor is 
        <emphasis>0.001</emphasis> and the offset 0. Another example is the relationship between 
        kelvin and Celsius, which is <code>1 kelvin + 273.15 = 1 °Celsius</code>.
        Here, the factor is 1 and the offset is <emphasis>+273.15</emphasis>.
        The <classname
        docapi="net.sf.basedb.core.data">UnitSymbolData</classname>
        is used to make it possible to assign alternative symbols to a single unit.
        This is needed to simplify input where it may be hard to know what to
        type to get <emphasis>m²</emphasis> or <emphasis>°C</emphasis>. Instead, 
        <emphasis>m2</emphasis> and <emphasis>C</emphasis> can be used as 
        alternative symbols.
        </para>
        
        <para>
        The creator of an annotation type may select a 
        <classname>QuantityData</classname>, which can't be changed later, and
        a default <classname>UnitData</classname>. When entering annotation values
        a user may select any unit for the selected quantity (unless annotation type
        owner has limited this by selecting <varname>usableUnits</varname>). Before
        the values are stored in the database, they are converted to the default
        unit. This makes it possible to compare and filter on annotation values
        using different units. For example, filtering with <emphasis>&gt;5mg</emphasis> 
        also finds items that are annotated with <emphasis>2g</emphasis>.
        </para>
        
        <para>
        The core should automatically update the stored annotation values if
        the default unit is changed for an annotation type, or if the reference
        factor for a unit is changed.
        </para>
      </sect3>
      
      <sect3 id="data_api.annotations.categories">
        <title>Categories</title>
        
        <para>
        The <classname docapi="net.sf.basedb.core.data">AnnotationTypeCategoryData</classname> class defines
        categories that are used to group annotation types that are related to 
        each other. This information is mainly useful for client applications 
        when displaying forms for annotating items, that wish to provide a 
        clearer interface when there are many (say 50+) annotations type for 
        an item. An annotation type can belong to more than one category. 
        </para>
        
      </sect3>
      
    </sect2>

    <sect2 id="data_api.protocols">
      <title>Protocols</title>

      <para>
        This section gives an overview of how protocols that describe various
        processes, such as sampling, extraction and scanning, are used in BASE.
      </para>
      
      <sect3 id="data_api.protocols.uml">
        <title>UML diagram</title>
        
        <figure id="data_api.figures.protocols">
          <title>Protocols</title>
          <screenshot>
            <mediaobject>
              <imageobject>
                <imagedata 
                  align="center"
                  fileref="figures/uml/datalayer.protocols.png" format="PNG" />
              </imageobject>
            </mediaobject>
          </screenshot>
        </figure>
      </sect3>
      
      <sect3 id="data_api.protocols.description">
        <title>Protocols</title>
        
        <para>
        A protocol is something that defines a procedure or recipe for some
        kind of action, such as sampling, extraction and scanning. In BASE we only
        store a short name and description. It is possible to attach a file
        that provides a longer description of the procedure.
        </para>
      
      </sect3>
      
      <sect3 id="data_api.protocols.parameters">
        <title>Parameters</title>
        
        <para>
        The procedure described by the protocol may have parameters
        that are set indepentently each time the protocol is used. It
        could for example be a temperature, a time or something else.
        The definition of parameters is done by creating annotation
        types and attaching them to the protocol. It is only possible
        to attach annotation types which has the <property>protocolParameter</property>
        property set to <constant>true</constant>. The same annotation type
        can be used for more than one protocol, but only do this if the 
        parameters actually has the same meaning.
        </para>
      
      </sect3>
      
    </sect2>

    <sect2 id="data_api.plugins">
      <title>Plug-ins, jobs and job agents</title>
      
      <para>
         This section gives an overview of plug-ins, jobs and job agents.
      </para>
      
      <itemizedlist>
        <title>See also</title>
        <listitem><xref linkend="plugins.installation" /></listitem>
        <listitem><xref linkend="installation_upgrade.jobagents" /></listitem>
      </itemizedlist>
      
      <sect3 id="data_api.plugins.uml">
        <title>UML diagram</title>
        
        <figure id="data_api.figures.plugins">
          <title>Plug-ins, jobs and job agents</title>
          <screenshot>
            <mediaobject>
              <imageobject>
                <imagedata 
                  align="center"
                  scalefit="1" width="100%"
                  fileref="figures/uml/datalayer.plugins.png" format="PNG" />
              </imageobject>
            </mediaobject>
          </screenshot>
        </figure>
      </sect3>

      <sect3 id="data_api.plugins.plugins">
        <title>Plug-ins</title>
        
        <para>
          The <classname docapi="net.sf.basedb.core.data">PluginDefinitionData</classname> holds information of the 
          installed plugin classes. Much of the information is copied from the
          plug-in itself from the <classname docapi="net.sf.basedb.core.plugin">About</classname> object and by checking
          which interfaces it implements.
        </para>
        
        <para>
          There are five main types of plug-ins:
        </para>
        
        <itemizedlist>
        <listitem>
          <para>
          IMPORT (mainType = 1): A plug-in that imports data to BASE.
          </para>
        </listitem>
        <listitem>
          <para>
          EXPORT (mainType = 2): A plug-in that exports data from BASE.
          </para>
        </listitem>
        <listitem>
          <para>
          INTENSITY (mainType = 3): A plug-in that calculates intensity values
          from raw data.
          </para>
        </listitem>
        <listitem>
          <para>
          ANALYZE (mainType = 4): A plug-in that analyses data.
          </para>
        </listitem>
        <listitem>
          <para>
          OTHER (mainType = 5): Any other plug-in.
          </para>
        </listitem>
        </itemizedlist>
        
        <para>
          A plug-in may have different configurations. The flags <property>supportsConfigurations</property>
          and <property>requiresConfiguration</property> are used to specify if a plug-in 
          must have or can't have any configurations. Configuration parameter values are 
          versioned. Each time anyone updates a configuration the version number 
          is increased and the parameter values are stored as a new entity. 
          This is required because we want to be able to know exactly which
          parameters a job were using when it was executed. When a job is 
          created we also store the parameter version number 
          (<property>JobData.parameterVersion</property>). This means that even if 
          someone changes the configuration later we will always know which 
          parameters the job used. 
        </para>
        
        <para>
          The <classname docapi="net.sf.basedb.core.data">PluginTypeData</classname> class is ued to group
          plug-ins that share some common functionality, by implementing
          additional (optional) interfaces. For example, the 
          <interfacename docapi="net.sf.basedb.core.plugin">AutoDetectingImporter</interfacename> should be implemented
          by import plug-ins that supports automatic detection of file formats.
          Another example is the <interfacename docapi="net.sf.basedb.core.plugin">AnalysisFilterPlugin</interfacename>
          interface which should be implemented by all analysis plug-ins that
          only filters data.
        </para>

      </sect3>
      
      <sect3 id="data_api.plugins.jobs">
        <title>Jobs</title>
        
        <para>
          A job represents a single invokation of a plug-in to do some work.
          The <classname docapi="net.sf.basedb.core.data">JobData</classname> class holds information about this.
          A job is usuallu executed by a plug-in, but doesn't have to be. The
          <property>status</property> property holds the current state of a job.
        </para>
        
        <itemizedlist>
        <listitem>
          <para>
            UNCONFIGURED (status = 0): The job is not yet ready to be executed.
          </para>
        </listitem>
        <listitem>
          <para>
            WAITING (status = 1): The job is waiting to be executed.
          </para>
        </listitem>
        <listitem>
          <para>
            PREPARING (status = 5): The job is about to be executed but hasn't started yet.
          </para>
        </listitem>
        <listitem>
          <para>
            EXECUTING (status = 2): The job is currently executing.
          </para>
        </listitem>
        <listitem>
          <para>
            DONE (status = 3): The job finished successfully.
          </para>
        </listitem>
        <listitem>
          <para>
            ERROR (status = 4): The job finished with an error.
          </para>
        </listitem>
        </itemizedlist>
      </sect3>

      <sect3 id="data_api.plugins.agents">
        <title>Job agents</title>
        
        <para>
          A job agent is a program running on the same or a different server that 
          is regularly checking for jobs that are waiting to be executed. The
          <classname docapi="net.sf.basedb.core.data">JobAgentData</classname> holds information about a job agent
          and the <classname docapi="net.sf.basedb.core.data">JobAgentSettingsData</classname> links the agent
          with the plug-ins the agent is able to execute. The job agent will only 
          execute jobs that are owner by users or projects that the job agent has 
          been shared to with at least use permission. The <property>priorityBoost</property>
          property can be used to give specific plug-ins higher priority.
          Thus, for a job agent it is possible to:
        </para>
        
        <itemizedlist>
        <listitem>
          <para>
          Specify exactly which plug-ins it will execute. For example, it is possible
          to dedicate one agent to only run one plug-in.
          </para>
        </listitem>
        <listitem>
          <para>
          Give some plug-ins higher priority. For example a job agent that is mainly 
          used for importing data should give higher priority to all import plug-ins.
          Other types of jobs will have to wait until there are no more data to be 
          imported.
          </para>
        </listitem>
        <listitem>
          <para>
          Specify exactly which users/groups/projects that may use the agent. For 
          example, it is possible to dedicate one agent to only run jobs for a certain 
          project.
          </para>
        </listitem>
        </itemizedlist>
        
      </sect3>


    </sect2>
    
    <sect2 id="data_api.biomaterials">
      <title>Biomaterials</title>
      
      <sect3 id="data_api.biomaterials.uml">
        <title>UML diagram</title>
        
        <figure id="data_api.figures.biomaterials">
          <title>Biomaterials</title>
          <screenshot>
            <mediaobject>
              <imageobject>
                <imagedata 
                  align="center"
                  fileref="figures/uml/datalayer.biomaterials.png" format="PNG" />
              </imageobject>
            </mediaobject>
          </screenshot>
        </figure>
      </sect3>
      
      <sect3 id="data_api.biomaterials.description">
        <title>Biomaterials</title>
        
        <para>
          There are four types of biomaterials: <classname docapi="net.sf.basedb.core.data">BioSourceData</classname>, 
          <classname docapi="net.sf.basedb.core.data">SampleData</classname>, <classname docapi="net.sf.basedb.core.data">ExtractData</classname> and 
          <classname docapi="net.sf.basedb.core.data">LabeledExtractData</classname>.
          All four types of are derived from the base class <classname docapi="net.sf.basedb.core.data">BioMaterialData</classname>. 
          The reason for this is that they all share common functionality such as pooling 
          and events. By using a common base class we do not have to create duplicate 
          classes for keeping track of events and parents.
        </para>
        
        <para>
          The <classname docapi="net.sf.basedb.core.data">BioSourceData</classname> is the simplest of the biomaterials. 
          It cannot have parents and can't participate in events. It's only used as a 
          (non-required) parent for samples.
        </para>
        
        <para>
          The <classname docapi="net.sf.basedb.core.data">MeasuredBioMaterialData</classname> class is used as a base 
          class for the other three biomaterial types. It introduces quantity 
          measurements and can store original and remaining quantities. They are 
          both optional. If an original quantity has been specified the core 
          automatically calculates the remaining quantity based on the events a 
          biomaterial participates in.
        </para>
        
        <para>
          All measured biomaterial have at least one event associated with them, 
          the creation event, which holds information about the creation of the 
          biomaterial. A measured biomaterial can be created in three ways:
        </para>
        
        <itemizedlist>
        <listitem>
          <para>
          From a single item of the parent type. Biosource is the parent type of 
          samples, sample is the parent type of extracts, and extract is the
          parent type of labeled extracts. In this case the 
          <property>pooled</property> property is <constant>false</constant>
          and the parent is specified in the <property>parent</property> property. 
          If the parent is not a <classname docapi="net.sf.basedb.core.data">BioSourceData</classname> this information 
          is duplicated, with the addition of an optional used quantity value, in the 
          <property>sources</property> collection of the <classname docapi="net.sf.basedb.core.data">BioMaterialEventData</classname>
          object representing the creation event. It is the responsibility of the 
          core to make sure that everything is properly synchronized and that 
          remaining quantities are calculated.
          </para>
        </listitem>
        
        <listitem>
          <para>
          From one or more items of the same type, i.e pooling.
          In this case the <property>pooled</property> property is <constant>true</constant> 
          and the <property>parent</property> property is null. All source 
          biomaterials are contained in the <property>sources</property> collection. 
          The core is still responsible for keeping everything synchronized and to
          update remaining quantities.
          </para>
        </listitem>
        
        <listitem>
          <para>
          As a standalone biomaterial without parents.
          </para>
        </listitem>
        </itemizedlist>

      </sect3>
      
      <sect3 id="data_api.biomaterials.events">
        <title>Biomaterial events</title>
        
        <para>
          An event represents something that happened to one or more biomaterials, for example 
          the creation of another biomaterial. The <classname docapi="net.sf.basedb.core.data">BioMaterialEventData</classname>
          holds information about entry and event dates, protocols used, the user who is 
          responsible, etc. There are three types of events represented by the <property>eventType</property>
          property.
        </para>
        
        <orderedlist>
        <listitem>
          <para>
          <emphasis>Creation event</emphasis>: This event represents the creation of a (measured) 
          biomaterial. The <property>sources</property> collection contains 
          information about the biomaterials that were used to create the new 
          biomaterial. If the biomaterial is a pooled biomaterial all sources must 
          be of the same type. Otherwise there can only be one source of the parent 
          type. These rules are maintained by the core.
          </para>
        </listitem>
        
        <listitem>
          <para>
          <emphasis>Hybridization event</emphasis>: This event represents the creation 
          of a hybridization. This event type is needed because we want to keep track
          of quantities for labeled extracts. This event has a hybridization as a 
          product instead of a biomaterial. The sources collection can only contain 
          labeled extracts.
          </para>
        </listitem>

        <listitem>
          <para>
          <emphasis>Other event</emphasis>: This event represents some other important
          information about a single biomaterial that affected the remaining quantity. 
          This event type doesn't have any sources.
          </para>
        </listitem>
        </orderedlist>
      </sect3>
  
    </sect2>

    <sect2 id="data_api.plates">
      <title>Array LIMS - plates</title>

      <sect3 id="data_api.plates.uml">
        <title>UML diagram</title>
        
        <figure id="data_api.figures.plates">
          <title>Array LIMS - plates</title>
          <screenshot>
            <mediaobject>
              <imageobject>
                <imagedata 
                  align="center"
                  scalefit="1" width="100%"
                  fileref="figures/uml/datalayer.plates.png" format="PNG" />
              </imageobject>
            </mediaobject>
          </screenshot>
        </figure>
      </sect3>

      <sect3 id="data_api.plates.description">
        <title>Plates</title>
        
        <para>
          The <classname docapi="net.sf.basedb.core.data">PlateData</classname> is the main class holding information 
          about a single plate. The associated <classname docapi="net.sf.basedb.core.data">PlateGeometryData</classname>
          defines how many rows and columns there are on a plate. Since this 
          information is used to create wells, and for various other checks it is 
          not possible to change the number of rows or columns once a geometry has 
          been created. 
        </para>
          
        <para>
          All plates must have a <classname docapi="net.sf.basedb.core.data">PlateTypeData</classname> which defines 
          the geometry and a set of event types (see below).
        </para>
        
        <para>
          If the destroyed flag of a plate is set it is not allowed to use the 
          plate for a plate mapping or to create array designs. However, it 
          is possible to change the flag to not destroyed.
        </para>

        <para>
          The barcode is intended to be used as an external identifier of the plate. 
          But, the core doesn't care about the value or if it is unique or not. 
        </para>
      </sect3>
      
      <sect3 id="data_api.plates.events">
        <title>Plate events</title>

        <para>
          The plate type defines a set of <classname docapi="net.sf.basedb.core.data">PlateEventTypeData</classname>
          objects, each one represening a particular event a plate of this type
          usually goes trough. For a plate of a certain type, it is possible to 
          attach exactly one event of each event type. The event type defines an 
          optional protocol type, which can be used by client applications to 
          filter a list of protocols for the event. The core doesn't check that 
          the selected protocol for an event is of the same protocol type as 
          defined by the event type.
        </para>

        <para>
          The ordinal value can be used as a hint to client applications in 
          which order the events actually are performed in the lab. The core doesn't 
          care about this value or if several event types have the same value. 
        </para>
      </sect3>

      <sect3 id="data_api.plates.mappings">
        <title>Plate mappings</title>
        
        <para>
          A plate can be created either from scratch, with the help of the information
          in a <classname docapi="net.sf.basedb.core.data">PlateMappingData</classname>, from a set of parent plates.
          In the first case it is possible to specify a reporter for each well on the 
          plate. In the second case the mapping code creates all the wells and links 
          them to the parent wells on the parent plates. Once the plate has been saved
          to the database, the wells cannot be modified (because they are used 
          downstream for various validation, etc.)
        </para>
        
        <para>
          The details in a plate mapping are simply coordinates that for each
          destination plate, row and column define a source plate, row and column.
          It is possible for a single source well to be mapped to multiple destination
          wells, but for each destination well only a single source well can be 
          used.
        </para>
        
      </sect3>

    </sect2>

    <sect2 id="data_api.arrays">
      <title>Array LIMS - arrays</title>
      
      <sect3 id="data_api.arrays.uml">
        <title>UML diagram</title>
        
        <figure id="data_api.figures.arrays">
          <title>Array LIMS - arrays</title>
          <screenshot>
            <mediaobject>
              <imageobject>
                <imagedata 
                  align="center"
                  fileref="figures/uml/datalayer.arrays.png" format="PNG" />
              </imageobject>
            </mediaobject>
          </screenshot>
        </figure>
      </sect3>
      
      <sect3 id="data_api.arrays.designs">
        <title>Array designs</title>
        
        <para>
          Array designs are stored in <classname docapi="net.sf.basedb.core.data">ArrayDesignData</classname> objects
          and can be created either as standalone designs or
          from plates. In the first case the features on an array design
          are described by a reporter map. A reporter map is a file
          that maps a coordinate (block, meta-grid, row, column), 
          position or an external ID on an array design to a 
          reporter. Which method to use is given by the 
          <property>ArrayDesign.featureIdentificationMethod</property> property. 
          The coordinate system on an array design is divided into blocks.
          Each block can be identified either by a <property>blockNumber</property>
          or by meta coordinates. This information is stored in
          <classname docapi="net.sf.basedb.core.data">ArrayDesignBlockData</classname> items. Each block
          contains several <classname docapi="net.sf.basedb.core.data">FeatureData</classname> items, each
          one identified by a row and column coordinate. Platforms that doesn't
          divide the array design into blocks or doesn't use the coordinate system at all
          must still create a single super-block that holds all features. 
        </para>
        
        <para>
          Array designs that are created from plates use a print map file
          instead of a reporter map. A print map is similar to a plate mapping
          but maps features (instead of wells) to wells. The file should
          specifify which plate and well a feature is created from. Reporter
          information will automatically be copied by BASE from the well.
        </para>
        
        <para>
          It is also possible to skip the importing of features into the 
          database and just keep the data in the orginal files instead.
          This is typically done for Affymetrix CDF files.
        </para>
        
      </sect3>
      
      <sect3 id="data_api.arrays.slides">
        <title>Array slides</title>
        
        <para>
          The <classname docapi="net.sf.basedb.core.data">ArraySlideData</classname> represents a single
          array. Arrays are usually printed several hundreds in a batch, 
          represented by a <classname docapi="net.sf.basedb.core.data">ArrayBatchData</classname> item.
          The <property>batchIndex</property> is the ordinal number of the
          array in the batch. The <property>barcode</property> can be used
          as a means for external programs to identify the array. BASE doesn't
          care if a value is given or if they are unique or not. If the
          <property>destroyed</property> flag is set it prevents a slide from
          beeing used by a hybridization.
        </para>

      </sect3>
    </sect2>

    <sect2 id="data_api.rawdata">
      <title>Hybridizations and raw data</title>
      
      <sect3 id="data_api.rawdata.uml">
        <title>UML diagram</title>
        
        <figure id="data_api.figures.rawdata">
          <title>Hybridizations and raw data</title>
          <screenshot>
            <mediaobject>
              <imageobject>
                <imagedata 
                  align="center"
                  scalefit="1" width="100%"
                  fileref="figures/uml/datalayer.rawdata.png" format="PNG" />
              </imageobject>
            </mediaobject>
          </screenshot>
        </figure>
      </sect3>
      
      <sect3 id="data_api.rawdata.hybridizations">
        <title>Hybridizations</title>
        
        <para>
        Hybridizations connects the slides from the Array LIMS part
        with labeled extracts from the biomaterials part. The <property>creationEvent</property>
        is used to register which labeled extracts that were used on the hybridization.
        The relation to slides is a one-to-one relation. A slide can only be used on 
        a single hybridization and a hybridization can only use a single slide. The relation
        is optional from both sides.
        </para>

        <para>
        The scanning of the hybridized slide is registered as separate scan events.
        One or more images can optionally be attached to each scan. 
        The images are not used by BASE.
        </para>
        
      </sect3>
      
      <sect3 id="data_api.rawdata.description">
        <title>Raw data</title>
        
        <para>
        A <classname docapi="net.sf.basedb.core.data">RawBioAssayData</classname> object represents
        the raw data that is produced by analysing the image(s) from a
        single scan. You may register which software that was used, the
        protocol and any parameters (through the annotation system).
        </para>

        <para>
        Files with the analysed data values can be attached to the 
        associated <classname docapi="net.sf.basedb.core.data">FileSetData</classname> object. The platform
        and, optionally, the variant has information about the file types
        that can be used for that platform. If the platform file types support 
        metadata extraction, headers, the number of spots, and other 
        information may be automatically extracted from the raw data file(s).
        </para>
        
        <para>
        If the platform support it, raw data can also be imported into the database.
        This is handled by batchers and <classname docapi="net.sf.basedb.core.data">RawData</classname> objects.
        Which table to store the data in depends on the <property>rawDataType</property>
        property. The properties shown for the <classname>RawData</classname> class
        in the diagram are the mandatory properties. Each raw data type defines additional
        properties that are specific to that raw data type.
        </para>
        
      </sect3>
      
      <sect3 id="data_api.rawdata.spotimages">
        <title>Spot images</title>
        
        <para>
        Spot images can be created if you have the original image
        files. BASE can use 1-3 images as sources for the red, green
        and blue channel respectively. The creation of spotimages requires
        that x and y coordinates are given for each raw data spot. The scaling
        and offset values are used to convert the coordinates to pixel
        coordinates. With this information BASE is able to cut out a square
        from the source images that, theoretically, contains a specific spot and 
        nothing else. The spot images are gamma-corrected independently and then
        put together into PNG images that are stored in a zip file. 
        </para>
      </sect3>
      
    </sect2>

    <sect2 id="data_api.experiments">
      <title>Experiments and analysis</title>
      
      
      <sect3 id="data_api.experiments.uml">
        <title>UML diagram</title>
        
        <figure id="data_api.figures.experiments">
          <title>Experiments</title>
          <screenshot>
            <mediaobject>
              <imageobject>
                <imagedata 
                  align="center"
                  scalefit="1" width="75%"
                  fileref="figures/uml/datalayer.experiments.png" format="PNG" />
              </imageobject>
            </mediaobject>
          </screenshot>
        </figure>
      </sect3>
      
      <sect3 id="data_api.experiments.description">
        <title>Experiments</title>
        
        <para>
          The <classname docapi="net.sf.basedb.core.data">ExperimentData</classname> 
          class is used to collect information about a single experiment. It 
          links to any number of <classname docapi="net.sf.basedb.core.data">RawBioAssayData</classname>
          items, which must all be of the same <classname 
          docapi="net.sf.basedb.core">RawDataType</classname>.
        </para>
        
        <para>
          Annotation types that are needed in the analysis must connected to
          the experiment as experimental factors and the annotation values should 
          be set on or inherited by each raw bioassay that is part of the 
          experiment.
        </para>
        
        <para>
          The directory connected to the experiment is the default directory
          where plugins that generate files should store them.
        </para>
      </sect3>
            
      <sect3 id="data_api.experiments.bioassays">
        <title>Bioassay sets, bioassays and transformations</title>
        
        <para>
          Each line of analysis starts with the creation of a <emphasis>root</emphasis>
          <classname docapi="net.sf.basedb.core.data">BioAssaySetData</classname>, 
          which holds the intensities calculated from the raw data. 
          A bioassayset can hold one intensity for each channel. The number of 
          channels is defined by the raw data type. For each raw bioassay used a 
          <classname docapi="net.sf.basedb.core.data">BioAssayData</classname>
          is created.
        </para>
        
        <para>
          Information about the process that calculated the intensities are 
          stored in a <classname docapi="net.sf.basedb.core.data">TransformationData</classname>
          object. The root transformation links with the raw bioassays that are used 
          in this line of analysis and to a <classname 
          docapi="net.sf.basedb.core.data">JobData</classname> which has information
          about which plug-in and parameters that was used in the calculation.
        </para>
      
        <para>
          Once the root bioassayset has been created it is possible to 
          again apply a transformation to it. This time the transformation
          links to a single source bioassayset instead of the raw bioassays. 
          As before, it still links to a job with information about the plug-in and
          parameters that does the actual work. The transformation must make sure 
          that new bioassays are created and linked to the bioassays in the 
          source bioassayset. This above process may be repeated as many times 
          as needed.
        </para>
        
        <para>
          Data to a bioassay set can only be added to it before it has been
          committed to the database. Once the transaction has been committed
          it is no longed possible to add more data or to modify existing
          data.
        </para>
      
      </sect3>

      <sect3 id="data_api.experiments.virtualdb">
        <title>Virtual databases, datacubes, etc.</title>
        
        <para>
          The above processes requires a flexible storage solution for the data. 
          Each experiment is related to a <classname docapi="net.sf.basedb.core.data">VirtualDb</classname>
          object. This object represents the set of tables that are needed to store
          data for the experiment. All tables are created in a special part of the
          BASE database that we call the <emphasis>dynamic database</emphasis>.
          In MySQL the dynamic database is a separate database, in Postgres it is
          a separate schema.
        </para>
        
        <para>
          A virual database is divided into data cubes. A data cube can be seen 
          as a three-dimensional object where each point can hold data that in
          most cases can be interpreted as data for a single spot from an
          array. The coordinates to a point is given by <emphasis>layer</emphasis>,
          <emphasis>column</emphasis> and <emphasis>position</emphasis>. The
          layer and column coordinates are represented by the 
          <classname docapi="net.sf.basedb.core.data">DataCubeLayerData</classname>
          and <classname docapi="net.sf.basedb.core.data">DataCubeColumnData</classname>
          objects. The position coordinate has no separate object associated with
          it.
        </para>
        
        <para>
          Data for a single bioassay set is always stored in a single layer. It
          is possible for more than one bioassay set to use the same layer. This
          usually happens for filtering transformations that doesn't modify the 
          data.  The filtered bioassay set is then linked to a 
          <classname docapi="net.sf.basedb.core.data">DataCubeFilterData</classname>
          object, which has information about which data points that
          passed the filter.
        </para>
        
        <para>
          All data for a bioassay is stored in a single column.
          Two bioassays in different bioassaysets (layers) can only have the same
          column if one is the parent of the other.
        </para>
        
        <para>
          The position coordinate is tied to a reporter.
        </para>
        
        <para>
          A child bioassay set may use the same data cube as it's parent
          bioassay set if all of the following conditions are true:
        </para>
        
        <itemizedlist>
        <listitem>
          <para>
          All positions are linked to the same reporter as the positions
          in the parent bioassay set.
          </para>
        </listitem>
        
        <listitem>
          <para>
          All data points are linked to the same (possible many) raw data
          spots as the corresponding data points in the parent bioassay set.
          </para>
        </listitem>
        
        <listitem>
          <para>
          The bioassays in the child bioassay set each have exactly one
          parent in the parent bioassay set. No parent bioassay may be the
          parent of more than one child bioassay.
          </para>
        </listitem>
        </itemizedlist>
        
        <para>
          If any of the above conditions are not true, a new data cube
          must be created for the child bioassay set. 
        </para>
      </sect3>
      
      <sect3 id="data_api.dynamic.description">
        <title>The dynamic database</title>

        <figure id="data_api.figures.dynamic">
          <title>The dynamic database</title>
          <screenshot>
            <mediaobject>
              <imageobject>
                <imagedata 
                  align="center"
                  fileref="figures/uml/datalayer.dynamic.png" format="PNG" />
              </imageobject>
            </mediaobject>
          </screenshot>
        </figure>
        
        <para>
          Each virtual database consists of several tables. The tables
          are dynamically created when needed. For each table shown in the diagram 
          the # sign is replaced by the id of the virtual database object at run
          time.
        </para>
        
        <para>
          There are no classes in the data layer for these tables and they 
          are not mapped with Hibernate. When we work with these tables we 
          are always using batcher classes and queries that works with integer, 
          floats and strings.
        </para>
        
        <bridgehead>The D#Spot table</bridgehead>
        <para>
          This is the main table which keeps the intensities for a single spot 
          in the data cube. Extra values attached to the spot are kept in separate 
          tables, one for each type of value (D#SpotInt, D#SpotFloat and D#SpotString).
        </para>
        
        <bridgehead>The D#Pos table</bridgehead>
        <para>
          This table stores the reporter id for each position in a cube. 
          Extra values attached to the position are kept in separate tables, 
          one for each type of value (D#PosInt, D#PosFloat and D#PosString).
        </para>
        
        <bridgehead>The D#Filter table</bridgehead>
        <para>
          This table stores the coordinates for the spots that remain after 
          filtering. Note that each filter is related to a bioassayset which 
          gives the cube and layer values. Each row in the filter table then 
          adds the column and position allowing us to find the spots in the 
          D#Spot table.
        </para>
        
        <bridgehead>The D#RawParents table</bridgehead>
        <para>
          This table holds mappings for a spot to the raw data it is calculated 
          from. We don't need the layer coordinate since all layers in a cube 
          must have the same mapping to raw data.
        </para>
        
      </sect3>      

      
    </sect2>
    
    <sect2 id="data_api.misc">
      <title>Other classes</title>
      
      <sect3 id="data_api.misc.uml">
        <title>UML diagram</title>
        
        <figure id="data_api.figures.misc">
          <title>Other classes</title>
          <screenshot>
            <mediaobject>
              <imageobject>
                <imagedata 
                  align="center"
                  fileref="figures/uml/datalayer.misc.png" format="PNG" />
              </imageobject>
            </mediaobject>
          </screenshot>
        </figure>
      </sect3>
      
    </sect2>

  </sect1>
  
  <sect1 id="api_overview.core_api" chunked="1">
    <title>The Core API</title>
    
    <para>
      This section gives an overview of various parts of the core API.
    </para>
    
    <sect2 id="core_api.data_in_files">
      <title>Using files to store data</title>
      
      <para>
        BASE 2.5 introduced the possibility to use files to store data instead
        of importing it into the database. Files can be attached
        to any item that implements the <interfacename docapi="net.sf.basedb.core">FileStoreEnabled</interfacename>
        interface. Currently this is <classname docapi="net.sf.basedb.core">RawBioAssay</classname>
        and <classname docapi="net.sf.basedb.core">ArrayDesign</classname>. The
        ability to store data in files is not a replacement for storing data in the
        database. It is possible (for some platforms/raw data types) to have data in 
        files and in the database at the same time. We would have liked to enforce 
        that (raw) data is always present in files, but this will not be backwards compatible
        with older installations, so there are three cases:
      </para>
      
      <itemizedlist>
      <listitem>
        <para>
        Data in files only
        </para>
      </listitem>
      <listitem>
        <para>
        Data in the database only
        </para>
      </listitem>
      <listitem>
        <para>
        Data in both files and in the database
        </para>
      </listitem>
      </itemizedlist>
      
      <para>
        Not all three cases are supported for all types of data. This is controlled
        by the <classname docapi="net.sf.basedb.core">Platform</classname> class, which may disallow 
        that data is stored in the database. To check this call
        <methodname>Platform.isFileOnly()</methodname> and/or
        <methodname>Platform.getRawDataType()</methodname>. If the <methodname>isFileOnly()</methodname>
        method returns <constant>true</constant>, the platform can't store data in 
        the database. If the value is <constant>false</constant> more information
        can be obtained by calling <methodname>getRawDataType()</methodname>,
        which may return:
      </para>
      
      <itemizedlist>
      <listitem>
        <para>
          <constant>null</constant>: The platform can store data with any
          raw data type in the database. 
        </para>
      </listitem>
      <listitem>
        <para>
        A <classname docapi="net.sf.basedb.core">RawDataType</classname> that has <code>isStoredInDb() == true</code>:
        The platform can store data in the database but only data with the specified raw
        data type.
        </para>
      </listitem>
      <listitem>
        <para>
        A <classname docapi="net.sf.basedb.core">RawDataType</classname> that has <code>isStoredInDb() == false</code>:
        The platform can't store data in the database.
        </para>
      </listitem>
      </itemizedlist>

      <para>
        One major change from earlier BASE versions is that the registration of raw data types
        has changed. The <filename>raw-data-types.xml</filename> file should
        only be used for raw data types that are stored in the database. The
        <sgmltag>storage</sgmltag> tag has been deprecated and BASE will refuse
        to start if it finds a raw data type definitions with <code>storage="file"</code>.
      </para>
      
      <para>
        For backwards compatibility reasons, each <classname docapi="net.sf.basedb.core">Platform</classname>
        that can only store data in files will create "virtual" raw data type
        objects internally. These raw data types all return <constant>false</constant>
        from the <methodname>RawDataType.isStoredInDb()</methodname>
        method. They also have a back-link to the platform/variant that
        created it: <methodname>RawDataType.getPlatform()</methodname>
        and <methodname>RawDataType.getVariant()</methodname>. These two methods
        will always return <constant>null</constant> when called on a raw data type 
        that can be stored in the database.
      </para>
      
      <itemizedlist>
        <title>See also</title>
        <listitem><xref linkend="data_api.platforms" /></listitem>
        <listitem><xref linkend="plugin_developer.other.datafiles" /></listitem>
        <listitem><xref linkend="appendix.rawdatatypes.platforms" /></listitem>
        <listitem>
          <xref linkend="appendix.incompatible.2.5" /> in
          <xref linkend="appendix.incompatible" />
        </listitem>
      </itemizedlist>
      
      <sect3 id="core_api.data_in_files.diagram">
        <title>Diagram of classes and methods</title>
        <figure id="core_api.figures.data_in_files">
          <title>Store data in files</title>
          <screenshot>
            <mediaobject>
              <imageobject>
                <imagedata 
                  align="center"
                  scalefit="1" width="100%"
                  fileref="figures/uml/corelayer.datainfiles.png" format="PNG" />
              </imageobject>
            </mediaobject>
          </screenshot>
        </figure>
        
        <para>
          This is rather large set of classes and methods. The ultimate goal
          is to be able to create links between a <classname docapi="net.sf.basedb.core">RawBioAssay</classname>
          / <classname docapi="net.sf.basedb.core">ArrayDesign</classname> and <classname docapi="net.sf.basedb.core">File</classname>
          items and to provide some metadata about the files. 
          The <classname docapi="net.sf.basedb.core">FileStoreUtil</classname> class is one of the most
          important ones. It is intended to make it easy for plug-in (and other)
          developers to access the files without having to mess with platform
          or file type objects. The API is best described
          by a set of use-case examples.
        </para>
        
      </sect3>
      
      <sect3 id="core_api.data_in_files.ask">
        <title>Use case: Asking the user for files for a given item</title>

        <para>
          A client application must know what types of files it makes sense 
          to ask the user for. In some cases, data may be split into more than 
          one file so we need a generic way to select files.
        </para>
        
        <para>
          Given that we have a <interfacename docapi="net.sf.basedb.core">FileStoreEnabled</interfacename>
          item we want to find out which <classname docapi="net.sf.basedb.core">DataFileType</classname>
          items that can be used for that item. The 
          <methodname>DataFileType.getQuery(FileStoreEnabled)</methodname>
          can be used for this. Internally, the method uses the result from 
          <methodname>FileStoreEnabled.getPlatform()</methodname>
          and <methodname>FileStoreEnabled.getVariant()</methodname>
          methods to restrict the query to only return file types for
          a given platform and/or variant. If the item doesn't have
          a platform or variant the query will return all file types
          that are associated with the given item type. In any case, we get a list 
          of <classname>DataFileType</classname> items, each one representing a 
          specific file type that we should ask the user about. Examples:
        </para>

        <orderedlist>
        <listitem>
          <para>
          The <constant>Affymetrix</constant> platform defines <constant>CEL</constant>
          as a raw data file and <constant>CDF</constant> as an array design (reporter map)
          file. If we have a <classname docapi="net.sf.basedb.core">RawBioAssay</classname> the query will only return
          the CEL file type and the client can ask the user for a CEL file.
          </para>
        </listitem>
        <listitem>
          <para>
          The <constant>Generic</constant> platform defines <constant>PRINT_MAP</constant>
          and <constant>REPORTER_MAP</constant> for array designs. If we have
          an <classname docapi="net.sf.basedb.core">ArrayDesign</classname> the query will return those two
          items.
          </para>
        </listitem>
        </orderedlist>
      
        <para>
          It might also be interesting to know the currently selected file
          for each file type and if the platform has set the <varname>required</varname>
          flag for a particular file type. Here is a simple code example
          that may be useful to start from:
        </para>
      
        <programlisting language="java">
DbControl dc = ...
FileStoreEnabled item = ...
Platform platform = item.getPlatform();
PlatformVariant variant = item.getVariant();

// Get list of DataFileTypes used by the platform
ItemQuery&lt;DataFileType&gt; query =
   DataFileType.getQuery(item);
List&lt;DataFileType&gt; types = query.list(dc);

// Always check hasFileSet() method first to avoid
// creating the file set if it doesn't exists
FileSet fileSet = item.hasFileSet() ? 
   null : item.getFileSet();
   
for (DataFileType type : types)
{
   // Get the current file, if any
   FileSetMember member = fileSet == null || !fileSet.hasMember(type) ?
      null : fileSet.getMember(type);
   File current = member == null ? 
      null : member.getFile();
   
   // Check if a file is required by the platform
   PlatformFileType pft = platform == null ? 
      null : platform.getFileType(type, variant);
   boolean isRequired = pft == null ? 
      false : pft.isRequired();
      
   // Now we can do something with this information to
   // let the user select a file ...
}
</programlisting>
      
        <note>
          <title>Also remember to catch PermissionDeniedException</title>
          <para>
            The above code may look complicated, but this is mostly because
            of all checks for <constant>null</constant> values. Remember
            that many things are optional and may return <constant>null</constant>.
            Another thing to look out for is 
            <exceptionname>PermissionDeniedException</exceptionname>:s. The logged in
            user may not have access to all items. The above example doesn't include
            any code for this since it would have made it too complex.
          </para>
        </note>
      </sect3>
      
      <sect3 id="core_api.data_in_files.link">
        <title>Use case: Link, validate and extract metadata from the selected files</title>
        <para>
          When the user has selected the file(s) we must store the links
          to them in the database. This is done with a <classname docapi="net.sf.basedb.core">FileSet</classname>
          object. A file set can contain any number of files. The only limitation
          is that it can only contain one file for each file type.
          Call <methodname>FileSet.setMember()</methodname> to store
          a file in the file set. If a file already exists for the given file type
          it is replaced, otherwise a new entry is created. The following
          program example assumes that we have a map where <classname docapi="net.sf.basedb.core">File</classname>:s
          are related to <classname docapi="net.sf.basedb.core">DataFileType</classname>:s. When all files
          have been added we call <methodname>FileSet.validate()</methodname>
          to validate the files and extract metadata.
        </para>
        
        <programlisting language="java">
DbControl dc = ...
FileStoreEnabled item = ...
Map&lt;DataFileType, File&gt; files = ...

// Store the selected files in the fileset
FileSet fileSet = item.getFileSet();
for (Map.Entry&lt;DataFileType, File&gt; entry : files)
{
   DataFileType type = entry.getKey();
   File file = entry.getValue();
   fileSet.setMember(type, file);
}

// Validate the files and extract metadata
fileSet.validate(dc, true);
</programlisting>

        <para>
          Validation and extraction of metadata is important since we want
          data in files to be equivalent to data in the database. The validation
          and metadata extraction is done by the core when the 
          <methodname>FileSet.validate()</methodname> is called. 
          The process is partly pluggable since each <classname docapi="net.sf.basedb.core">DataFileType</classname> 
          can name a class that should do the validation and/or metadata extraction.
        </para>

        <note>
          <para>
          The <methodname>FileSet.validate()</methodname> only validates
          the files where the file types have specified plug-ins that can
          do the validation and metadata extraction. The method doesn't
          throw any exceptions. Instead, all validation errors
          are returned a list of <classname>Throwable</classname>:s. The
          validation result is also stored for each file and can be access
          with <methodname>FileSetMember.isValid()</methodname> and
          <methodname>FileSetMember.getErrorMessage()</methodname>.
          </para>
        </note>

        <para>
          Here is the general outline of what is going on in the core:
        </para>

        <orderedlist>
        <listitem>
          <para>
          The core checks the <classname docapi="net.sf.basedb.core">DataFileType</classname> of all
          members in the file set and creates <classname docapi="net.sf.basedb.core.filehandler">DataFileValidator</classname>
          and <classname docapi="net.sf.basedb.core.filehandler">DataFileMetadataReader</classname> objects. Only one instance
          of each class is created. If the file set contains members which has the
          same validator or metadata reader, they will all share the same instance.
          </para>
        </listitem>
        
        <listitem>
          <para>
          Each validator/metadata reader class is initialised with calls to
          <methodname>DataFileHandler.setItem()</methodname> and 
          <methodname>DataFileHandler.setFile()</methodname>.
          </para>
        </listitem>
        
        <listitem>
          <para>
          Each validator is called. The result of the validation is saved for each
          file and can be retreieved by <methodname>FileSetMember.isValid()</methodname>
          and <methodname>FileSetMember.getErrorMessage()</methodname>.
          </para>
        </listitem>
        
        <listitem>
          <para>
          Each metadata reader is called, unless the metadata reader is the same class
          as the validator and the validation failed. If the metadata reader is a
          different class, it is called even if the validation failed.
          </para>
        </listitem>
        </orderedlist>

        <note>
          <title>Only one instance of each validator class is created</title>
          <para>
          The validation/metadata extraction is not done until all files have been 
          added to the fileset. If the same validator/meta data reader is
          used for more than one file, the same instance is reused. Ie. 
          the <methodname>setFile()</methodname> is called one time
          for each file/file type pair. The <methodname>validate()</methodname>
          and <methodname>extractMetadata()</methodname> methods are only
          called once.
          </para>
        </note>
        
        <para>
          All validators and meta data extractors should extend
          the <classname docapi="net.sf.basedb.core.filehandler">AbstractDataFileHandler</classname> class. The reason
          is that we may want to add more methods to the <interfacename docapi="net.sf.basedb.core.filehandler">DataFileHandler</interfacename>
          interface in the future. The <classname docapi="net.sf.basedb.core.filehandler">AbstractDataFileHandler</classname> will
          be used to provide default implementations for backwards compatibility.
        </para>
        
      </sect3>
      
      <sect3 id="core_api.data_in_files.import">
        <title>Use case: Import data into the database</title>
        
        <para>
          This should be done by existing plug-ins in the same way as before.
          A slight modification is needed since it is good if the importers
          are made aware of already selected files in the <classname docapi="net.sf.basedb.core">FileSet</classname>
          to provide good default values. The <classname docapi="net.sf.basedb.core">FileStoreUtil</classname>
          class is very useful in cases like this:
        </para>
        
        <programlisting language="java">
RawBioAssay rba = ...
DbControl dc = ...

// Get the current raw data file, if any
List&lt;File&gt; rawDataFiles = 
   FileStoreUtil.getGenericDataFiles(dc, rba, FileType.RAW_DATA);
File defaultFile = rawDataFiles.size() > 0 ?
   rawDataFiles.get(0) : null;
   
// Create parameter asking for input file - use current as default
PluginParameter&lt;File&gt; fileParameter = new PluginParameter&lt;File&gt;(
   "file",
   "Raw data file",
   "The file that contains the raw data that you want to import",
   new FileParameterType(defaultFile, true, 1)
);
</programlisting>

      <para>
        An import plug-in should also save the file that was used to the file set:
      </para>
      
      <programlisting language="java">
RawBioassay rba = ...
// The file the user selected to import from
File rawDataFile = (File)job.getValue("file");

// Save the file to the fileset. The method will check which file 
// type the platform uses as the raw data type. As a fallback the
// GENERIC_RAW_DATA type is used
FileStoreUtil.setGenericDataFile(dc, rba, FileType.RAW_DATA, 
   DataFileType.GENERIC_RAW_DATA, rawDataFile);
</programlisting>

      </sect3>
      
      <sect3 id="core_api.data_in_files.experiments">
        <title>Use case: Using raw data from files in an experiment</title>
        
        <para>
          Just as before, an experiment is still locked to a single
          <classname docapi="net.sf.basedb.core">RawDataType</classname>. This is a design issue that
          would break too many things if changed. If data is stored in files
          the experiment is also locked to a single <classname docapi="net.sf.basedb.core">Platform</classname>.
          This has been designed to have as little impact on existing
          plug-ins as possible. In most cases, the plug-ins will continue
          to work as before.
        </para>
        
        <para>
          A plug-in (using data from the database that needs to check if it can
          be used within an experiment can still do:
        </para>
        
        <programlisting language="java">
Experiment e = ...
RawDataType rdt = e.getRawDataType();
if (rdt.isStoredInDb())
{
   // Check number of channels, etc...
   // ... run plug-in code ...
}
</programlisting>
        
        <para>
          A newer plug-in which uses data from files should do:
        </para>
        
        <programlisting language="java">
Experiment e = ...
DbControl dc = ...
RawDataType rdt = e.getRawDataType();
if (!rdt.isStoredInDb())
{
   // Check that platform/variant is supported
   Platform p = rdt.getPlatform(dc);
   PlatformVariant v = rdt.getVariant(dc);
   // ...

   // Get data files
   File aFile = FileStoreUtil.getDataFile(dc, ...);
   
   // ... run plug-in code ...
}
</programlisting>
        
      </sect3>
      
    </sect2>
    
    <sect2 id="core_api.signals">
      <title>Sending signals (to plug-ins)</title>
    
      <para>
        BASE has a simple system for sending signals between different parts of
        a system. This signalling system was initially developed to be able to
        kill plug-ins that a user for some reason wanted to abort. The signalling
        system as such is not limited to this and it can be used for other purposes 
        as well. Signals can of course be handled internally in a single JVM but
        also sent externally to other JVM:s running on the same or a different 
        computer. The transport mechanism for signals is decoupled from the actual
        handling of them. If you want to, you could implement a signal transporter
        that sends signal as emails and the target plug-in would never know.
      </para>
      
      <para>
        The remainder of this section will focus mainly on the sending and
        transportation of signals. For more information about handling signals
        on the receiving end, see <xref linkend="plugin_developer.signals" />.
      </para>
      
      <sect3 id="core_api.signals.diagram">
        <title>Diagram of classes and methods</title>
        <figure id="core_api.figures.signals">
          <title>The signalling system</title>
          <screenshot>
            <mediaobject>
              <imageobject>
                <imagedata 
                  align="center"
                  scalefit="1" width="100%"
                  fileref="figures/uml/corelayer.signals.png" format="PNG" />
              </imageobject>
            </mediaobject>
          </screenshot>
        </figure>
      
        <para>
          The signalling system is rather simple. An object that wish
          to receieve signals must implement the 
          <interfacename docapi="net.sf.basedb.core.signal"
          >SignalTarget</interfacename>. It's only method 
          is <methodname>getSignalHandler()</methodname>. A 
          <interfacename docapi="net.sf.basedb.core.signal"
          >SignalHandler</interfacename> is an object that 
          knows what to do when a signal is delivered to it. The target object
          may implement the <interfacename>SignalHandler</interfacename> itself
          or use one of the existing handlers. 
        </para>
        
        <para>
          The difficult part here is to be aware that a signal is usually
          delivered by a separate thread. The target object must be aware
          of this and know how to handle multiple threads. As an example we
          can use the <classname docapi="net.sf.basedb.core.signal"
          >ThreadSignalHandler</classname> which simply
          calls <code>Thread.interrupt()</code> to deliver a signal. The target 
          object that uses this signal handler it must know that it should check
          <code>Thread.interrupted()</code> at regular intervals from the main
          thread. If that method returns true, it means that the <constant>ABORT</constant>
          signal has been delivered and the main thread should clean up and exit as
          soon as possible.
        </para>
        
        <para>
          Even if a signal handler could be given directly to the party
          that may be interested in sending a signal to the target this
          is not recommended. This would only work when sending signals
          within the same virtual machine. The signalling system includes
          <interfacename docapi="net.sf.basedb.core.signal"
          >SignalTransporter</interfacename> and
          <interfacename docapi="net.sf.basedb.core.signal"
          >SignalReceiver</interfacename> objects that are used
          to decouple the sending of signals with the handling of signals. The
          implementation usually comes in pairs, for example 
          <classname docapi="net.sf.basedb.core.signal"
          >SocketSignalTransporters</classname> and <classname 
          docapi="net.sf.basedb.core.signal">SocketSignalReceiver</classname>.
        </para>
        
        <para>
          Setting up the transport mechanism is usually a system responsibility.
          Only the system know what kind of transport that is appropriate for it's current 
          setup. Ie. should signals be delievered by TCP/IP sockets, only internally, or 
          should a delivery mechanism based on web services be implemented? 
          If a system wants to receive signals it must create an appropriate
          <interfacename>SignalReceiver</interfacename> object. Within BASE the 
          internal job queue set up it's own signalling system that can be used to
          send signals (eg. kill) running jobs. The job agents do the same but uses
          a different implementation. See <xref linkend="appendix.base.config.jobqueue" />
          for more information about how to configure the internal job queue's 
          signal receiver. In both cases, there is only one signal receiver instance
          active in the system.
        </para>
        
        <para>
          Let's take the internal job queue as an example. Here is how it works:
        </para>
        
        <itemizedlist>
        <listitem>
          <para>
          When the internal job queue is started, it will also create a signal
          receiver instance according to the settings in <filename>base.config</filename>.
          The default is to create <classname docapi="net.sf.basedb.core.signal"
          >LocalSignalReceiver</classname>
          which can only be used inside the same JVM. If needed, this can
          be changed to a <classname docapi="net.sf.basedb.core.signal"
          >SocketSignalReceiver</classname> or any other
          user-provided implementation.
          </para>
        </listitem>
        
        <listitem>
          <para>
          When the job queue has found a plug-in to execute it will check if
          it also implements the <interfacename docapi="net.sf.basedb.core.signal"
          >SignalTarget</interfacename>
          interface. If it does, a signal handler is created and registered
          with the signal receiver. This is actually done by the BASE core 
          by calling <methodname>PluginExecutionRequest.registerSignalReceiver()</methodname>
          which also makes sure that the the ID returned from the registration is
          stored in the database together with the job item representing the 
          plug-in to execute.
          </para>
        </listitem>
        
        <listitem>
          <para>
          Now, when the web client see's a running job which has a non-empty 
          signal transporter property, the <guilabel>Abort</guilabel>
          button is activated. If the user clicks this button the BASE core
          uses the information in the database to create 
          <interfacename docapi="net.sf.basedb.core.signal"
          >SignalTransporter</interfacename> object. This
          is simply done by calling <code>Job.getSignalTransporter()</code>.
          The created signal transporter knows how to send a signal 
          to the signal receiver it was first registered with. When the
          signal arrives at the receiver it will find the handler for it
          and call <code>SignalHandler.handleSignal()</code>. This will in it's turn
          trigger some action in the signal target which soon will abort what
          it is doing and exit.
          </para>
        </listitem>
        </itemizedlist>
        
        
      </sect3>
    
    </sect2>
    
  </sect1>

  <sect1 id="api_overview.query_api">
    <title>The Query API</title>
    <para>
      This documentation is only available in the old format.
      See <ulink url="http://base.thep.lu.se/chrome/site/doc/historical/development/overview/query/index.html"
        >http://base.thep.lu.se/chrome/site/doc/historical/development/overview/query/index.html</ulink>
    </para>
    
  </sect1>
  
  <sect1 id="api_overview.dynamic_and_batch_api">
    <title>Analysis and the Dynamic and Batch API:s</title>
    <para>
      This documentation is only available in the old format.
      See <ulink url="http://base.thep.lu.se/chrome/site/doc/historical/development/overview/dynamic/index.html"
        >http://base.thep.lu.se/chrome/site/doc/historical/development/overview/dynamic/index.html</ulink>
    </para>
  </sect1>

  <sect1 id="api_overview.extensions">
    <title>Extensions API</title>
    
    <sect2 id="api_overview.extensions.core">
      <title>The core part</title>
    
      <para>
        The <emphasis>Extensions API</emphasis> is divided into two parts. A core
        part and a web client specific part. The core part can be found in the
        <package>net.sf.basedb.util.extensions</package> package and it's sub-packages,
        and consists of three sub-parts:
      </para>
      
      <itemizedlist>
      <listitem>
        <para>
        A set of interface definitions which forms the core of the Extensions API.
        The interfaces defines, for example, what an <interfacename 
        docapi="net.sf.basedb.util.extensions">Extension</interfacename> is and
        what an <interfacename 
        docapi="net.sf.basedb.util.extensions">ActionFactory</interfacename> should do.
        </para>
      </listitem>
      
      <listitem>
        <para>
        A <classname docapi="net.sf.basedb.util.extensions">Registry</classname> that is 
        used to keep track of installed extensions. The registry also provides 
        functionality for invoking and using the extensions.
        </para>
      </listitem>
      
      <listitem>
        <para>
        Utility classes that are useful when implementation a client application
        that can be extendable. The most useful example is the <classname
        docapi="net.sf.basedb.util.extensions.xml">XmlLoader</classname> which can
        read extension definitions from XML files and create the proper factories,
        etc.
        </para>
      </listitem>
      </itemizedlist>
      
      <figure id="core_api.figures.extensions_core">
        <title>The core part of the Extensions API</title>
        <screenshot>
          <mediaobject>
            <imageobject>
              <imagedata 
                align="center"
                fileref="figures/uml/corelayer.extensions_core.png" format="PNG" />
            </imageobject>
          </mediaobject>
        </screenshot>
      </figure>
      
      <para>
        The <classname docapi="net.sf.basedb.util.extensions">Registry</classname> 
        is one of the main classes in the extension system. All extension points and
        extensions must be registered before they can be used. Typically, you will
        first register extension points and then extensions, beacuse an extension
        can't be registered until the extension point it is extending has been
        registered.
      </para>
      
      <para>
        An <interfacename docapi="net.sf.basedb.util.extensions">ExtensionPoint</interfacename>
        is an ID and a definition of an <interfacename docapi="net.sf.basedb.util.extensions">Action</interfacename>
        class. The other options (name, description, renderer factory, etc.) are optional.
        An <interfacename docapi="net.sf.basedb.util.extensions">Extension</interfacename>
        that extends a specific extension point must provide an 
        <interfacename docapi="net.sf.basedb.util.extensions">ActionFactory</interfacename>
        instance that can create actions of the type the extension point requires.
      </para>
      
      <example id="core_api.example.extensions_core">
        <title>The menu extensions point</title>
        <para>
        The <code>net.sf.basedb.clients.web.menu.extensions</code> extension point 
        requires <interfacename 
        docapi="net.sf.basedb.clients.web.extensions.menu">MenuItemAction</interfacename>
        objects. An extension for this extension point must provide a factory that
        can create <classname>MenuItemAction</classname>:s. BASE ships with default
        factory implementations, for example the <classname 
        docapi="net.sf.basedb.clients.web.extensions.menu">FixedMenuItemFactory</classname>
        class, but an extension may provide it's own factory implementation if it wants to.
        </para>
      </example>
      
      <para>
        Call the <methodname>Registry.useExtensions()</methodname> method
        to use extensions from one or several extension points. This method will
        find all extensions for the given extension points. If a filter is given,
        it checks if any of the extensions or extension points has been disabled.
        It will then call <methodname>ActionFactory.prepareContext()</methodname>
        for all remaining extensions. This gives the action factory a chance to
        also disable the extension, for example, if the logged in user doesn't
        have a required permission. The action factory may also set attributes
        on the context. The attributes can be anything that the extension point 
        may make use of. Check the documentation for the specific extension point
        for information about which attributes it supports. If there are
        any renderer factories, their <methodname>RendererFactory.prepareContext()</methodname>
        is also called. They have the same possibility of setting attributes
        on the context, but can't disable an extension.
      </para>
      
      <para>
        After this, an <classname 
        docapi="net.sf.basedb.util.extensions">ExtensionsInvoker</classname>
        object is created and returned to the extension point. Note that
        the <methodname>ActionFactory.getActions()</methodname> has not been 
        called yet, so we don't know if the extensions are actually
        going to generate any actions. The <methodname>ActionFactory.getActions()</methodname>
        is not called until we have got ourselves an 
        <classname docapi="net.sf.basedb.util.extensions">ActionIterator</classname>
        from the <methodname>ExtensionsInvoker.iterate()</methodname> method and
        starts to iterate. The call to <methodname>ActionIterator.hasNext()</methodname>
        will propagate down to <methodname>ActionFactory.getActions()</methodname>
        and the generated actions are then available with the 
        <methodname>ActionIterator.next()</methodname> method.
      </para>
      
      <para>
        The <methodname>ExtensionsInvoker.renderDefault()</methodname>
        and <methodname>ExtensionsInvoker.render()</methodname> are 
        just convenience methods that will make it easer to render
        the actions. The first method will of course only work if the
        extension point is providing a renderer factory, that can
        create the default renderer.
      </para>
      
      <note>
        <title>Be aware of multi-threading issues</title>
        <para>
          When you are creating extensions you must be aware that
          multiple threads may access the same objects at the same time.
          In particular, any action factory or renderer factory has to be
          thread-safe, since only one exists for each extension. 
          Action and renderer objects should be thread-safe if the
          factories re-use the same objects.
        </para>
      </note>
    
    </sect2>
    
    <sect2 id="api_overview.extensions.web">
      <title>The web client part</title>
    
      <para>
        The web client specific parts of the Extensions API can be found
        in the <package>net.sf.basedb.client.web.extensions</package> package
        and it's subpackages. The top-level package contains classes used to 
        administrate the extension system. Here is for example the 
        <classname docapi="net.sf.basedb.client.web.extensions">ExtensionsControl</classname> 
        class which is the master controller for the web client extensions. It:
      </para>
      
      <itemizedlist>
      <listitem>
        <para>
        Keeps track of installed extensions and which JAR or XML file they are
        installed from.
        </para>
      </listitem>
      
      <listitem>
        <para>
        Can, manually or automatically, find and install new or
        updated extensions and uninstall deleted extensions.
        </para>
      </listitem>
      
      <listitem>
        <para>
        Adds permission control to the extension system, so that only an 
        administrator is allowed to change settings, enable/disable extensions, 
        etc.
        </para>
      </listitem>
      </itemizedlist>
      
      <para>
        In the top-level package there are also some abstract classes that may
        be useful to extend for developers creating their own extensions.
        For example, we recommend that all action factories extend the <classname 
        docapi="net.sf.basedb.client.web.extensions">AbstractJspActionFactory</classname>
        class.
      </para>
      
      <para>
        The sub-packages to <package>net.sf.basedb.client.web.extensions</package>
        are mostly specific to a single extension point or to a specific type of
        extension point. The <package>net.sf.basedb.client.web.extensions.menu</package>
        package, for example, contains classes that are/can be used for extensions 
        adding menu items to the <menuchoice><guimenu>Extensions</guimenu></menuchoice>
        menu.
      </para>
      
      <figure id="core_api.figures.extensions_web">
        <title>The web client part of the Extensions API</title>
        <screenshot>
          <mediaobject>
            <imageobject>
              <imagedata 
                align="center"
                fileref="figures/uml/corelayer.extensions_web.png" format="PNG" />
            </imageobject>
          </mediaobject>
        </screenshot>
      </figure>
    
      <para>
        When the Tomcat web server is starting up, the <classname 
        docapi="net.sf.basedb.client.web.extensions">ExtensionsServlet</classname>
        is automatically loaded. This servlet has as two purposes:
      </para>
      
      <itemizedlist>
      <listitem>
        <para>
        Initialise the extensions system by calling 
        <methodname>ExtensionsControl.init()</methodname>. This will result in
        an initial scan for installed extensions, which is equivalent to doing
        a manual scan with the force update setting to false. This means that
        the extension system is up an running as soon as the first user log's
        in to BASE.
        </para>
      </listitem>
      
      <listitem>
        <para>
        Act as a proxy for custom servlets defined by the extensions. URL:s
        ending with <code>.servlet</code> has been mapped to the 
        <classname>ExtensionsServlet</classname>. When a request is made it
        will extract the name of the extension's JAR file from the
        URL, get the corresponding <classname 
        docapi="net.sf.basedb.client.web.extensions">ExtensionsFile</classname>
        and <classname docapi="net.sf.basedb.client.web.extensions">ServletWrapper</classname>
        and then invoke the custom servlet. More information can be found in
        <xref linkend="extensions_developer.servlets" />.
        </para>
      </listitem>
      
      </itemizedlist>
      
      <para>
        Using extensions only involves calling the 
        <methodname>ExtensionsControl.createContext()</methodname> and
        <methodname>ExtensionsControl.useExtensions()</methodname> methods. This
        returns an <classname docapi="net.sf.basedb.util.extensions">ExtensionsInvoker</classname> 
        object as described in the previous section.
      </para>
      
      <para>
        To render the actions it is possible to either use the 
        <methodname>ExtensionsInvoker.iterate()</methodname> method
        and generate HTML from the information in each action. Or 
        (the better way) is to use a renderer together with the
        <classname docapi="net.sf.basedb.clients.web.taglib.extensions">Render</classname>
        taglib.
      </para>
      
      <para>
        To get information about the installed extensions,  
        change settings, enabled/disable extensions, performing a manual
        scan, etc. use the <methodname>ExtensionsControl.get()</methodname>
        method. This will create a permission-controlled object. All
        users has read permission, administrators has write permission.
      </para>
      
      <note>
        <para>
          The permission we check for is WRITE permission on the
          web client item. This means it is possible to give a user
          permissions to manage the extension system by assigning
          WRITE permission to the web client entry in the database.
          Do this from <menuchoice>
            <guimenu>Administrate</guimenu>
            <guimenuitem>Clients</guimenuitem>
          </menuchoice>.
        </para>
      </note>
    
      <para>
        The <classname docapi="net.sf.basedb.clients.web.extensions">XJspCompiler</classname>
        is mapped to handle the compilation <code>.xjsp</code> files
        which are regular JSP files with a different extension. This feature is 
        experimental and requires installing an extra JAR into Tomcat's lib
        directory. See <xref linkend="admin.extensions.xjspcompiler" /> for 
        more information.
      </para>
    
    </sect2>
    
  </sect1>

  <sect1 id="api_overview.other_api">
    <title>Other useful classes and methods</title>
    <para>
      TODO
    </para>
  </sect1>
  
</chapter>