source: trunk/doc/src/docbook/developerdoc/api_overview.xml @ 3838

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE chapter PUBLIC 
    "-//Dawid Weiss//DTD DocBook V3.1-Based Extension for XML and graphics inclusion//EN" 
    "../../../../lib/docbook/preprocess/dweiss-docbook-extensions.dtd">
<!--
  $Id: api_overview.xml 3838 2007-10-16 11:38:56Z nicklas $

  Copyright (C) 2007 Peter Johansson, Nicklas Nordborg, Martin Svensson

  This file is part of BASE - BioArray Software Environment.
  Available at http://base.thep.lu.se/

  BASE is free software; you can redistribute it and/or
  modify it under the terms of the GNU General Public License
  as published by the Free Software Foundation; either version 2
  of the License, or (at your option) any later version.

  BASE is distributed in the hope that it will be useful,
  but WITHOUT ANY WARRANTY; without even the implied warranty of
  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
  GNU General Public License for more details.

  You should have received a copy of the GNU General Public License
  along with this program; if not, write to the Free Software
  Foundation, Inc., 59 Temple Place - Suite 330,
  Boston, MA  02111-1307, USA.
-->

<chapter id="api_overview">
  <?dbhtml dir="api"?>
  <title>API overview (how to use and code examples)</title>

  <sect1 id="api_overview.public_api">
    <title>The Public API of BASE</title>
    
    <para>
      Not all public classes and methods in the <filename>BASE2Core.jar</filename>
      and other JAR files shipped with BASE are considered as
      <emphasis>Public API</emphasis>. This is important knowledge
      since we will always try to maintain backwards compatibility
      for classes that are part of the public API. For other
      classes, changes may be instroduced at any time without
      notice or specific documentation. In other words:
    </para>
    
    <note>
      <title>Only use the public API when developing plug-ins</title>
      <para>
        This will maximize the chance that you plug-in will continue
        to work with the next BASE release. If you use the non-public API
        you do so at your own risk.
      </para>
    </note>
    
    <para>
      See the <ulink url="http://base.thep.lu.se/chrome/site/doc/api/index.html"
        >javadoc</ulink> for information about
      what parts of the API that contributes to the public API.
      Methods, classes and other elements that have been tagged as 
      <code>@deprecated</code> should be considered as part of the internal API 
      and may be removed in a subsequent relase without warning.
    </para>
    
    <para>
      See <xref linkend="appendix.incompatible" /> to read more about
      changes that have been introduced by each release.
    </para>

    <sect2 id="api_overview.compatibility">
      <title>What is backwards compatibility?</title>
      
      <para>
        There is a great article about this subject on <ulink 
        url="http://wiki.eclipse.org/index.php/Evolving_Java-based_APIs"
          >http://wiki.eclipse.org/index.php/Evolving_Java-based_APIs</ulink>.
        This is what we will try to comply with. If you do not want to
        read the entire article, here are some of the most important points:
      </para>
      
      
      <sect3 id="api_overview.compatibility.binary">
        <title>Binary compatibility</title>
        <para>
        <blockquote>
          Pre-existing Client binaries must link and run with new releases of the 
          Component without recompiling.
        </blockquote>
        
        For example:
        <itemizedlist>
        <listitem>
          <para>
            We cannot change the number or types of parameters to a method
            or constructor.
          </para>
        </listitem>
        <listitem>
          <para>
            We cannot add or change methods to interfaces that are intended
            to be implemented by plug-in or client code.
          </para>
        </listitem>
        </itemizedlist>
        </para>       
      </sect3>
      
      <sect3 id="api_overview.compatibility.contract">
        <title>Contract compatibility</title>
        <para>
          <blockquote>
          API changes must not invalidate formerly legal Client code.
          </blockquote>
        
          For example:
          <itemizedlist>
          <listitem>
            <para>
              We cannot change the implementation of a method to do
              things differently than before. For example, allow <constant>null</constant>
              as a return value when it was not allowed before.
            </para>
          </listitem>
          </itemizedlist>
        
          <note>
            <para>
            Sometimes there is a very fine line between what is considered a
            bug and what is considered a feature. For example, if the
            actual implementation does not do what the javadoc says,
            do we change the code or do we change the documentation?
            This has to be considered from case to case and depends on 
            the age of the code and if we expect plug-ins and clients to be
            affected by it or not.
            </para>
          </note>
        </para>
      </sect3>
      
      <sect3 id="api_overview.compatibility.source">
        <title>Source code compatibility</title>
        <para>
        This is not an important matter and is not always possible to
        achieve. In most cases, the problems are easy to fix.
        Example:
        
        <itemizedlist>
        <listitem>
          <para>
          Adding a class may break a plug-in or client that import
          classes with <constant>.*</constant> if the same class name
          exists in another package.
          </para>
        </listitem>
        </itemizedlist>
        </para>
      </sect3>
    </sect2>
  </sect1>

  <sect1 id="api_overview.data_api" chunked="1">
    <title>The database schema and the Data Layer API</title>

    <para>
      This section gives an overview of the entire data layer API. 
      The figure below show how different modules relate to each other.
    </para>
    
    <note>
      All information has not yet been transfered from the old documentation.
      The old documentation is available at
      <ulink url="http://base.thep.lu.se/chrome/site/doc/development/overview/data/index.html"
        >http://base.thep.lu.se/chrome/site/doc/development/overview/data/index.html</ulink>
    </note>
    
    <figure id="data_api.figures.overview">
      <title>Data layer overview</title>
      <screenshot>
        <mediaobject>
          <imageobject>
            <imagedata 
              fileref="figures/uml/datalayer.overview.png" format="PNG" />
          </imageobject>
        </mediaobject>
      </screenshot>
    </figure>

    <sect2 id="data_api.basic">
      <title>Basic classes and interfaces</title>
      
      <para>
        This document contains information about the basic classes and interfaces in this package. 
        They are important since all data-layer classes must inherit from one of the already 
        existing abstract base classes or implement one or more of the
        existing interfaces. They contain code that is common to all classes, 
        for example implementations of the <methodname>equals()</methodname>
        and <methodname>hashCode()</methodname> methods or how to link with the owner of an 
        item.
      </para>
      
      <sect3 id="data_api.basic.uml">
        <title>UML diagram</title>
        
        <figure id="data_api.figures.basic">
          <title>Basic classes and interfaces</title>
          <screenshot>
            <mediaobject>
              <imageobject>
                <imagedata 
                  fileref="figures/uml/datalayer.basic.png" format="PNG" />
              </imageobject>
            </mediaobject>
          </screenshot>
        </figure>
      </sect3>
      
      <sect3 id="data_api.basic.classes">
        <title>Classes</title>
        
        <variablelist>
        <varlistentry>
          <term><classname>BasicData</classname></term>
          <listitem>
            <para>
            The root class. It overrides the <methodname>equals()</methodname>, 
            <methodname>hashCode()</methodname> and <methodname>toString()</methodname> methods 
            from the <classname>Object</classname> class. It also defines the 
            <varname>id</varname> and <varname>version</varname> properties.
            All data layer classes must inherit from this class or one of it's subclasses.
            </para>
          </listitem>
        </varlistentry>
        
        <varlistentry>
          <term><classname>OwnedData</classname></term>
          <listitem>
            <para>
            Extends the <classname>BasicData</classname> class and adds 
            an <varname>owner</varname> property. The owner is a required link to a 
            <classname>UserData</classname> object, representing the user that
            is the owner of the item.
            </para>
          </listitem>
        </varlistentry>

        <varlistentry>
          <term><classname>SharedData</classname></term>
          <listitem>
            <para>
            Extends the <classname>OwnedData</classname> class and adds 
            properties (<varname>itemKey</varname> and <varname>projectKey</varname>)
            that holds access permission information for an item.
            Access permissions are held in <classname>ItemKeyData</classname> and/or 
            <classname>ProjectKeyData</classname> objects. These objects only exists if 
            the item has been shared.
            </para>
          </listitem>
        </varlistentry>

        <varlistentry>
          <term><classname>CommonData</classname></term>
          <listitem>
            <para>
            This is a convenience class for items that extends the <classname>SharedData</classname>
            class and implements the <interfacename>NameableData</interfacename> and 
            <interfacename>RemoveableData</interfacename> interfaces. This is one of
            the most common situations.
            </para>
          </listitem>
        </varlistentry>

        <varlistentry>
          <term><classname>AnnotatedData</classname></term>
          <listitem>
            <para>
            This is a convenience class for items that can be annotated. 
            Annotations are held in <classname>AnnotationSetData</classname> objects. 
            The annotation set only exists if annotations has been created for the item.
            </para>
          </listitem>
        </varlistentry>
        </variablelist>
        
      </sect3>
      
      <sect3 id="data_api.basic.interfaces">
        <title>Interfaces</title>
        
        <variablelist>
        <varlistentry>
          <term><classname>IdentifiableData</classname></term>
          <listitem>
            <para>
            All items are identifiable, which means that they have a unique <varname>id</varname>. 
            The id is unique for all items of a specific type (ie. class). The id is number 
            that is automatically generated by the database and has no other meaning 
            outside of the application. The <varname>version</varname> property is used for 
            detecting and preventing concurrent modifications to an item. 
            </para>
          </listitem>
        </varlistentry>
        
        <varlistentry>
          <term><classname>OwnableData</classname></term>
          <listitem>
            <para>
            An ownable item is an item which has an owner. The owner is represented as a 
            required link to a <classname>UserData</classname> object. 
            </para>
          </listitem>
        </varlistentry>       

        <varlistentry>
          <term><classname>ShareableData</classname></term>
          <listitem>
            <para>
            A shareable item is an item which can be shared to other users, groups or projects.
            Access permissions are held in <classname>ItemKeyData</classname> and/or 
            <classname>ProjectKeyData</classname> objects. 
            </para>
          </listitem>
        </varlistentry>
              
        <varlistentry>
          <term><classname>NameableData</classname></term>
          <listitem>
            <para>
            A nameable item is an item that has a name (required) and a description
            (optional). The name doesn't have to be unique, except in a few special 
            cases (for example, the name of a file).
            </para>
          </listitem>
        </varlistentry>
        
        <varlistentry>
          <term><classname>RemovableData</classname></term>
          <listitem>
            <para>
            A removable item is an item that can be flagged as removed. This doesn't 
            remove the information about the item from the database, but can be used by 
            client applications to hide items that the user is not interested in. 
            A trashcan function can be used to either restore or permanently 
            remove items that has the flag set.
            </para>
          </listitem>
        </varlistentry>
                
        <varlistentry>
          <term><classname>SystemData</classname></term>
          <listitem>
            <para>
            A system item is an item which has an additional id in the form of string. A system id 
            is required when we need to make sure that we can get a specific item without 
            knowing the numeric id. Example of such items are the root user and the everyone group. 
            A system id is generally constructed like: 
            <constant>net.sf.basedb.core.User.ROOT</constant>. The system id:s are defined in the 
            core layer by each item class. 
            </para>
          </listitem>
        </varlistentry>

        <varlistentry>
          <term><classname>DiskConsumableData</classname></term>
          <listitem>
            <para>
            This interface is used by items which occupies a lot of disk space and 
            should be part of the quota system, for example files. The required 
            <classname>DiskUsageData</classname> contains information about the size, 
            location, owner etc. of the item.
            </para>
          </listitem>
        </varlistentry>
        
        <varlistentry>
          <term><classname>AnnotatableData</classname></term>
          <listitem>
            <para>
            This interface is used by items which can be annotated. Annotations are name/value 
            pairs that are attached as extra information to an item. All annotations are
            contained in an <classname>AnnotationSetData</classname> object. 
            </para>
          </listitem>
        </varlistentry>
        
        <varlistentry>
          <term><classname>ExtendableData</classname></term>
          <listitem>
            <para>
            This interface is used by items which can have extra administrator-defined 
            columns. The functionality is similar to annotations. It is not as flexible, 
            since it is a global configuration, but has better performance. BASE will
            generate extra database columns to store the data in the tables for items that
            can be extended.
            </para>
          </listitem>
        </varlistentry>
        
        <varlistentry>
          <term><classname>BatchableData</classname></term>
          <listitem>
            <para>
            This interface is a tagging interface which is used by items that needs batch 
            functionality in the core.
            </para>
          </listitem>
        </varlistentry>
        </variablelist>

      </sect3>
    </sect2>
    
    <sect2 id="data_api.authentication">
      <title>User authentication and access control</title>
      
      <para>
         This section gives an overview of user authentication and
         how groups, roles and projects are used for access control
         to items.
      </para>
      
      <sect3 id="data_api.authentication.uml">
        <title>UML diagram</title>
        
        <figure id="data_api.figures.authentication">
          <title>User authentication and access control</title>
          <screenshot>
            <mediaobject>
              <imageobject>
                <imagedata 
                  fileref="figures/uml/datalayer.authentication.png" format="PNG" />
              </imageobject>
            </mediaobject>
          </screenshot>
        </figure>
      </sect3>
      
      <sect3 id="data_api.authentication.users">
        <title>Users and passwords</title>      
      
        <para>
          The <classname>UserData</classname> class holds information about users. 
          We keep the passwords in a separate table and use proxies to avoid loading 
          password data each time a user is loaded to minimize security risks. It is
          only if the password needs to be changed that the <classname>PasswordData</classname>
          object is loaded. The one-to-one mapping between user and password is controlled 
          by the password class, but a cascade attribute on the user class makes sure 
          that the password is deleted when a user is deleted.
        </para>
      </sect3>

      <sect3 id="data_api.authentication.groups">
        <title>Groups, roles and projects</title>     
      
        <para>
          The <classname>GroupData</classname>, <classname>RoleData</classname> and 
          <classname>ProjectData</classname> classes holds information about groups, roles 
          and projects respectively. A user may be a member of any number of groups,
          roles and/or projects. The membership in a project comes with an attached
          permission values. This is the highest permission the user has in the
          project. No matter what permission an item has been shared with the
          user will not get higher permission. Groups may be members of other groups and 
          also in projects.
        </para>
        
      </sect3>
      
      <sect3 id="data_api.authentication.keys">
        <title>Keys</title>     
      
        <para>
          The <classname>KeyData</classname> class and it's subclasses 
          <classname>ItemKeyData</classname>, <classname>ProjectKeyData</classname> and 
          <classname>RoleKeyData</classname>, are used to store information about access 
          permissions to items. To get permission to manipulate an item a user must have 
          access to a key giving that permission. There are three types of keys:
        </para>
        
        <variablelist>
        <varlistentry>
          <term><classname>ItemKey</classname></term>
          <listitem>
            <para>
            Is used to give a user or group access to a specific item. The item 
            must be a <interfacename>ShareableData</interfacename> item.
            The permissions are usually set be the owner of the item. Once created an 
            item key cannot be changed. This allows the core to reuse a key if the 
            permissions match exactly, ie. for a given set of users/groups/permissions 
            there can be only one item key object.
            </para>
          </listitem>
        </varlistentry>

        <varlistentry>
          <term><classname>ProjectKey</classname></term>
          <listitem>
            <para>
            Is used to give members of a project access to a specific item. The item 
            must be a <interfacename>ShareableData</interfacename> item. Once created a 
            project key cannot be changed. This allows the core to reuse a key if the 
            permissions match exactly, ie. for a given set of projects/permissions 
            there can be only one project key object.
            </para>
          </listitem>
        </varlistentry>

        <varlistentry>
          <term><classname>RoleKey</classname></term>
          <listitem>
            <para>
            Is used to give a user access to all items of a specific type, ie. 
            <constant>READ</constant> all <constant>SAMPLES</constant>. The installation 
            will make sure that there already exists a role key for each type of item, and 
            it is not possible to add new or delete existing keys. Unlike the other two types 
            this key can be modified.
            </para>
            
            <para>
            A role key is also used to assign permissions to plug-ins. If a plug-in has 
            been specified to use permissions the default is to deny everything. 
            The mapping to the role key is used to grant permissions to the plugin. 
            The <varname>granted</varname> value gives the plugin access to all items 
            of the related item type regardless of if the user that is running the plug-in has the 
            permission or not. The <varname>denied</varname> values denies access to all 
            items of the related item type even if the logged in user has the permission. 
            Permissions that are not granted nor denied are checked against the 
            logged in users regular permissions. Permissions to items that are 
            not linked are always denied.
            </para>
          </listitem>
        </varlistentry>
        </variablelist>
        
      </sect3>

      <sect3 id="data_api.authentication.permissions">
        <title>Permissions</title>
        
        <para>
          The <varname>permission</varname> property appearing in many classes is an 
          integer values describing the permission:
        </para>
        
        <informaltable>
        <tgroup cols="2">
          <colspec colname="value" />
          <colspec colname="permission" />
          <thead>
            <row>
              <entry>Value</entry>
              <entry>Permission</entry>
            </row>
          </thead>
          <tbody>
            <row>
              <entry>1</entry>
              <entry>Read</entry>
            </row>
            <row>
              <entry>3</entry>
              <entry>Use</entry>
            </row>
            <row>
              <entry>7</entry>
              <entry>Restricted write</entry>
            </row>
            <row>
              <entry>15</entry>
              <entry>Write</entry>
            </row>
            <row>
              <entry>31</entry>
              <entry>Delete</entry>
            </row>
            <row>
              <entry>47 (=32+15)</entry>
              <entry>Set owner</entry>
            </row>
            <row>
              <entry>79 (=64+15)</entry>
              <entry>Set permissions</entry>
            </row>
            <row>
              <entry>128</entry>
              <entry>Create</entry>
            </row>
            <row>
              <entry>256</entry>
              <entry>Denied</entry>
            </row>
          </tbody>
        </tgroup>
        </informaltable>
        
        <para>
          The values are constructed so that 
          <constant>READ</constant> -&gt;
          <constant>USE</constant> -&gt;
          <constant>RESTRICTED_WRITE</constant> -&gt;
          <constant>WRITE</constant> -&gt;
          <constant>DELETE</constant>
          are chained in the sense that a higher permission always implies the lower permissions 
          also. The <constant>SET_OWNER</constant> and <constant>SET_PERMISSION</constant>
          both implies <constant>WRITE</constant> permission. The <constant>DENIED</constant>
          permission is only valid for role keys, and if specified it overrides all 
          other permissions.                
        </para>
        
        <para>
          When combining permission for a single item the permission codes for the different 
          paths are OR-ed together. For example a user has a role key with <constant>READ</constant>
          permission for <constant>SAMPLES</constant>, but also an item key with <constant>USE</constant>
          permission for a specific sample. Of course, the resulting permission for that
          sample is <constant>USE</constant>. For other samples the resulting permission is
          <constant>READ</constant>.
        </para>
        
        <para>
          If the user is also a member of a project which has <constant>WRITE</constant>
          permission for the same sample, the user will have <constant>WRITE</constant>
          permission when working with that project.
        </para>
        
        <para>
          The <constant>RESTRICTED_WRITE</constant> permission is in most cases the same 
          as the <constant>WRITE</constant> permission. So far the <constant>RESTRICTED_WRITE</constant>
          permission is only given to users to their own <classname>UserData</classname>
          object so they can change their address and other contact information, 
          but not quota, expiration date and other administrative information.
        </para>

      </sect3>
    </sect2>

    <sect2 id="data_api.wares">
      <title>Hardware and software</title>
    </sect2>
    
    <sect2 id="data_api.reporters">
      <title>Reporters</title>
    </sect2>

    <sect2 id="data_api.quota">
      <title>Quota and disk usage</title>
    </sect2>

    <sect2 id="data_api.sessions">
      <title>Client, session and settings</title>
    </sect2>

    <sect2 id="data_api.files">
      <title>Files and directories</title>

      <para>
        This section covers the details of the BASE file
        system.
      </para>

      <sect3 id="data_api.files.uml">
      <title>UML diagram</title>
      
        <figure id="data_api.figures.files">
          <title>Files and directories</title>
          <screenshot>
            <mediaobject>
              <imageobject>
                <imagedata 
                  fileref="figures/uml/datalayer.files.png" format="PNG" />
              </imageobject>
            </mediaobject>
          </screenshot>
        </figure>
      </sect3>
      
      <sect3 id="data_api.files.description">
        <title>Description</title>
        
        <para>
          The <classname>DirectoryData</classname> class holds
          information about directories. Directories are organised in the
          ususal way as as tree structure. All directories must have
          a parent directory, except the system-defined root directory.
        </para>
        
        <para>
          The <classname>FileData</classname> class holds information about
          a file. The actual file contents is stored on disk in the directory
          specified by the <varname>userfiles</varname> setting in 
          <filename>base.config</filename>. The <varname>internalName</varname>
          property is the name of the file on disk, but this is never exposed to 
          client applications. The filenames and directories
          on the disk doesn't correspond to the the filenames and directories in 
          BASE.
        </para>
        
        <para>
          The <varname>location</varname> property can take three values:
        </para>
        
        <itemizedlist>
        <listitem>
          <para>
          0 = The file is offline, ie. there is no file on the disk
          </para>
        </listitem>
        <listitem>
          <para>
          1 = The file is in primary storage, ie. it is located on the disk
          and can be used by BASE
          </para>
        </listitem>
        <listitem>
          <para>
          2 = The file is in secondary storage, ie. it has been moved to some
          other place and can't be used by BASE immediately.
          </para>
        </listitem>
        </itemizedlist>
        
        <para>
          The <varname>action</varname> property controls how a file is
          moved between primary and seconday storage. It can have the following
          values:
        </para>
        
        <itemizedlist>
        <listitem>
          <para>
          0 = Do nothing
          </para>
        </listitem>
        <listitem>
          <para>
          1 = If the file is in secondary storage, move it back to the primary storage
          </para>
        </listitem>
        <listitem>
          <para>
          2 = If the file is in primary storage, move it to the secondary storage
          </para>
        </listitem>
        </itemizedlist>
        
        <para>
          The actual moving between primary and secondary storage is done by an 
          external program. See 
          <xref linkend="appendix.base.config.secondary" /> and 
          <xref linkend="plugin_developer.other.secondary" /> for more information.
        </para>
      
        <para>
          The <varname>md5</varname> property can be used to check for file
          corruption when it is moved between primary and secondary storage or
          when a user re-uploads a file that has been offline.
        </para>
        
        <para>
          BASE can store files in a compressed format. This is handled internally
          and is not visible to client applications. The <varname>compressed</varname>
          and <varname>diskSize</varname> properties are used to store information
          about this. A file may always be compressed if the users says so, but
          BASE can also do this automatically if the file is uploaded
          to a directory with the <varname>autoCompress</varname> flag set
          or if the file has MIME type with the <varname>autoCompress</varname>
          flag set.
        </para>
        
        <para>
          The <classname>FileTypeData</classname> class holds information about 
          file types. It is used only to make it easier for users to organise 
          their files.
        </para>
        
        <para>
          The <classname>MimeTypeData</classname> is used to register mime types and
          map them to file extensions. The information is only used to lookup values 
          when needed. Given the filename we can set the <varname>File.mimeType</varname>
          and <varname>File.fileType</varname> properties. The MIME type is also
          used to decide if a file should be stored in a compressed format or not.
          The extension of a MIME type must be unique. Extensions should be registered 
          without a dot, ie <emphasis>html</emphasis>, not <emphasis>.html</emphasis>.  
        </para>
        
      </sect3>
      
      
    </sect2>
    
    <sect2 id="data_api.platforms">
      <title>Experimental platforms</title>

      <para>
         This section gives an overview of experimental platforms
         and how they are used to enable data storage in files
         instead of in the database.
      </para>
      
      <itemizedlist>
        <title>See also</title>
        <listitem><xref linkend="core_api.data_in_files" /></listitem>
        <listitem><xref linkend="appendix.rawdatatypes.platforms" /></listitem>
      </itemizedlist>
          
      <sect3 id="data_api.platforms.uml">
        <title>UML diagram</title>
        
        <figure id="data_api.figures.platforms">
          <title>Experimental platforms</title>
          <screenshot>
            <mediaobject>
              <imageobject>
                <imagedata 
                  fileref="figures/uml/datalayer.platforms.png" format="PNG" />
              </imageobject>
            </mediaobject>
          </screenshot>
        </figure>
      </sect3>
      
      <sect3 id="data_api.platforms.platforms">
        <title>Platforms</title>
        
        <para>
          The <classname>PlatformData</classname> holds information about a
          platform. A platform can have one or more <classname>PlatformVariant</classname>:s.
          Both the platform and variant are identified by an external ID that
          is fixed and can't be changed. <emphasis>Affymetrix</emphasis>
          and <emphasis>Illumina</emphasis> are examples of platforms.
          If the <varname>fileOnly</varname> flag is set data for the platform
          can only be stored in files and not imported into the database. If
          the flag is not set data can be imported into the database.
          The <varname>rawDataType</varname> can be used to lock the platform
          to a specific raw data type. If the value is <constant>null</constant>
          the platform can use any raw data type. 
        </para>
        
        <para>
          Each platform and it's variant can be connected to one or more
          <classname>DataFileTypeData</classname> items. This item 
          describes the kind of files that are used to hold data for
          the platform and/or variant. The file types are re-usable between
          different platforms and variants. Note that a file type may be attached
          to either only a platform or to a platform with a variant. File 
          types attached to platforms are inherited by the variants. The variants
          can only define additional file types, not remove or redefine file types
          that has been attached to the platform.
        </para>
        <para>
          The file type is also identified
          by a fixed, non-changable external ID. The <varname>itemType</varname>
          property tells us what type of item the file holds data for (ie. 
          array design or raw bioassay). It also links to a <classname>FileType</classname>
          which is the generic type of data in the file. This allows to query
          the database for, as an example, for files with the generic type
          <constant>FileType.RAW_DATA</constant>. If we are in an Affymetrix 
          experiment we will get the CEL file, for another platform we will
          get another file.
        </para>
        <para>
          The <varname>required</varname> flag in <classname>PlatformFileTypeData</classname>
          is used to signal that the file is a required file. This will, however, not be
          enforeced by the core. It is intended to be used by client applications
          for creating a better GUI and/or validation of an experiment.
        </para>

      </sect3>
      
      <sect3 id="data_api.platforms.files">
        <title>FileStoreEnabled items and data files</title>
        
        <para>
          An item must implement the <interfacename>FileStoreEnabledData</interfacename>
          interface to be able to store data in files instead of in the database.
          The interface creates a link to a <classname>FileSetData</classname> object,
          which is can hold several <classname>FileSetMemberData</classname> items.
          Each member points to specific <classname>FileData</classname> item.
          A file set can only store one file of each <classname>DataFileTypeData</classname>.
        </para>
        
      </sect3>
    </sect2>

    <sect2 id="data_api.protocols">
      <title>Protocols</title>
    </sect2>

    <sect2 id="data_api.parameters">
      <title>Parameters</title>
    </sect2>

    <sect2 id="data_api.annotations">
      <title>Annotations</title>
    </sect2>

    <sect2 id="data_api.plugins">
      <title>Plug-ins, jobs and job agents</title>
    </sect2>
    
    <sect2 id="data_api.biomaterials">
      <title>Biomaterials</title>
    </sect2>

    <sect2 id="data_api.plates">
      <title>Array LIMS - plates</title>
    </sect2>

    <sect2 id="data_api.arrays">
      <title>Array LIMS - arrays</title>
    </sect2>

    <sect2 id="data_api.rawdata">
      <title>Hybridizations and raw data</title>
    </sect2>

    <sect2 id="data_api.experiments">
      <title>Experiments and analysis</title>
    </sect2>
    
    <sect2 id="data_api.misc">
      <title>Other classes</title>
    </sect2>

  </sect1>
  
  <sect1 id="api_overview.core_api" chunked="1">
    <title>The Core API</title>
    
    <para>
      This section gives an overview of various parts of the core API.
    </para>
    
    <sect2 id="core_api.data_in_files">
      <title>Using files to store data</title>
      
      <para>
        This section is about how BASE can use files to store data instead
        of importing it into the database. Files can be attached
        to any item that implements the <interfacename>FileStoreEnabled</interfacename>
        interface. Currently this is <classname>RawBioAssay</classname>
        and <classname>ArrayDesign</classname>. The
        ability to store data in files is not a replacement for storing data in the
        database. It is possible (for some platforms/raw data types) to have data in 
        files and in the database at the same time. We would have liked to enforce 
        that (raw) data is always present in files, but this will not be backwards compatible
        with older installations, so there are three cases:
      </para>
      
      <itemizedlist>
      <listitem>
        <para>
        Data in files only
        </para>
      </listitem>
      <listitem>
        <para>
        Data in the database only
        </para>
      </listitem>
      <listitem>
        <para>
        Data in both files and in the database
        </para>
      </listitem>
      </itemizedlist>
      
      <para>
        Not all three cases are supported for all types of data. This is controlled
        by the <classname>Platform</classname> class, which may disallow 
        that data is stored in the database. To check this call
        <methodname>Platform.isFileOnly()</methodname> and/or
        <methodname>Platform.getRawDataType()</methodname>. If the <methodname>isFileOnly()</methodname>
        method returns <constant>true</constant>, the platform can't store data in 
        the database. If the value is <constant>false</constant> more information
        can be obtained by calling <methodname>getRawDataType()</methodname>,
        which may return:
      </para>
      
      <itemizedlist>
      <listitem>
        <para>
          <constant>null</constant>: The platform can store data with any
          raw data type in the database. 
        </para>
      </listitem>
      <listitem>
        <para>
        A <classname>RawDataType</classname> that has <code>isStoredInDb() == true</code>:
        The platform can store data in the database but only data with the specified raw
        data type.
        </para>
      </listitem>
      <listitem>
        <para>
        A <classname>RawDataType</classname> that has <code>isStoredInDb() == false</code>:
        The platform can't store data in the database.
        </para>
      </listitem>
      </itemizedlist>

      <para>
        One major modification is that the registration of raw data types
        has changed. The <filename>raw-data-types.xml</filename> file should
        only be used for raw data types that are stored in the database. The
        <sgmltag>storage</sgmltag> tag has been deprecated and BASE will refuse
        to start if it finds a raw data type definitions with <code>storage="file"</code>.
      </para>
      
      <para>
        For backwards compatibility reasons, each <classname>Platform</classname>
        that can only store data in files will create "virtual" raw data type
        objects internally. These raw data types all return <constant>false</constant>
        from if the method <methodname>RawDataType.isStoredInDb()</methodname>
        is called. They also have a back-link to the platform/variant that
        created it: <methodname>RawDataType.getPlatform()</methodname>
        and <methodname>RawDataType.getVariant()</methodname>. Theese two methods
        will always return null when called on a raw data type that can be
        stored in the database.
      </para>
      
      <itemizedlist>
        <title>See also</title>
        <listitem><xref linkend="data_api.platforms" /></listitem>
        <listitem><xref linkend="appendix.rawdatatypes.platforms" /></listitem>
        <listitem>
          <xref linkend="appendix.incompatible.2.5" xrefstyle=""/> in
          <xref linkend="appendix.incompatible" />
        </listitem>
      </itemizedlist>
      
      <sect3 id="core_api.data_in_files.diagram">
        <title>Diagram of classes and methods</title>
        <figure id="core_api.figures.data_in_files">
          <title>Store data in files</title>
          <screenshot>
            <mediaobject>
              <imageobject>
                <imagedata 
                  fileref="figures/uml/corelayer.datainfiles.png" format="PNG" />
              </imageobject>
            </mediaobject>
          </screenshot>
        </figure>
      </sect3>
      
      <sect3 id="core_api.data_in_files.ask">
        <title>Asking the user for files</title>

        <para>
          A client application must know what types of files it makes sense 
          to ask the user for. In some cases, data may be split into more than 
          one file so we need a generic way to select files.
        </para>
        
        <para>
          Given that we have a <interfacename>FileStoreEnabled</interfacename>
          item we want to find out which <classname>DataFileType</classname>
          items that can be used for that item. The 
          <methodname>DataFileType.getQuery(FileStoreEnabled)</methodname>
          can be used for this. Internally, the method uses the result from 
          <methodname>FileStoreEnabled.getPlatform()</methodname>
          and <methodname>FileStoreEnabled.getVariant()</methodname>
          methods to restrict the query to only return file types for
          a given platform and/or variant. If the item doesn't have
          a platform or variant the query will return all file types
          that are associated with the given item type. In any case, we get a list 
          of <classname>DataFileType</classname> items, each one representing a 
          specific file type that we should ask the user about. Examples:
        </para>

        <orderedlist>
        <listitem>
          <para>
          The <constant>Affymetrix</constant> platform defines <constant>CEL</constant>
          as a raw data file and <constant>CDF</constant> as an array design (feature)
          file. If we have a <classname>RawBioAssay</classname> the query will only return
          the CEL file type and the client can ask the user for a CEL file.
          </para>
        </listitem>
        <listitem>
          <para>
          The <constant>Generic</constant> platform defines <constant>PRINT_MAP</constant>
          and <constant>REPORTER_MAP</constant> for array designs. If we have
          an <classname>ArrayDesign</classname> the query will return those two
          items.
          </para>
        </listitem>
        </orderedlist>
      
        <para>
          It might also be interesting to know the currently selected file
          for each file type and if the platform has set the <varname>required</varname>
          flag for a particular file type. Here is a simple code example
          that may be useful to start from:
        </para>
      
        <programlisting language="java">
DbControl dc = ...
FileStoreEnabled item = ...
Platform platform = item.getPlatform();
PlatformVariant variant = item.getVariant();

// Get list of DataFileTypes used by the platform
ItemQuery&lt;DataFileType&gt; query =
   FileStoreUtil.getQuery(item);
List&lt;DataFileType&gt; types = query.list(dc);

// Always check hasFileSet() method first to avoid
// creating the file set if it doesn't exists
FileSet fileSet = item.hasFileSet() ? 
   null : item.getFileSet();
   
for (DataFileType type : types)
{
   // Get the current file, if any
   FileSetMember member = fileSet == null || !fileSet.hasMember(type) ?
      null : fileSet.getMember(type);
   File current = member == null ? 
      null : member.getFile();
   
   // Check if a file is required by the platform
   PlatformFileType pft = platform == null ? 
      null : platform.getFileType(type, variant);
   boolean isRequired = pft == null ? 
      false : pft.isRequired();
      
   // Now we can do something with this information to
   // let the user select a file ...
}
</programlisting>
      
        <note>
          <title>Also remember to catch PermissionDeniedException</title>
          <para>
            The above code may look complicated, but this is mostly because
            of all checks for <constant>null</constant> values. Remember
            that many things are optional and may return <constant>null</constant>.
            Another thing to look out for is 
            <exceptionname>PermissionDeniedException</exceptionname>:s. The logged in
            user may not have access to all items. The above example doesn't include
            any code for this since it would have made it too complex.
          </para>
        </note>
      </sect3>
      
      <sect3 id="core_api.data_in_files.link">
        <title>Link, validate and extract metadata from the selected files</title>
        <para>
          When the user has selected the file(s) we must store the links
          to them in the database. This is done with a <classname>FileSet</classname>.
          object. A file set can contain any number of files. The only limitation
          is that it can only contain one file for each file type.
          Call <methodname>FileSet.setMember()</methodname> to store
          a file in the set. If a file already exists for the given file type
          it is replaced, otherwise a new entry is created. The following
          program example assumes that we have a map where <classname>File</classname>:s
          are related to <classname>DataFileType</classname>:s
          
        </para>
        
        <programlisting language="java">
DbControl dc = ...
FileStoreEnabled item = ...
Map&lt;DataFileType, File&gt; files = ...

// Store the selected files in the fileset
FileSet fileSet = item.getFileSet();
for (Map.Entry&lt;DataFileType, File&gt; entry : files)
{
   DataFileType type = entry.getKey();
   File file = entry.getValue();
   fileSet.setMember(type, file);
}

// Validate the files and extract metadata
fileSet.validate(dc, true);
</programlisting>
      </sect3>
      
      <sect3 id="core_api.data_in_files.validate">
        <title>How the core validate the files and extracts metadata</title>
        
        <para>
          Validation and extraction of metadata is important since we want
          data in files to be equivalent to data in the database. The validation
          and metadata extraction is done by the core when the 
          <methodname>FileSet.validate()</methodname>. 
          The process is partly pluggable since each <classname>DataFileType</classname> 
          can name a class that should do the validation and/or metadata extraction.
          Here is the general outline of what is going on:
        </para>
        
        
        <orderedlist>
        <listitem>
          <para>
          The core checks the <classname>DataFileType</classname> of all
          members in the file set and create <classname>DataFileValidator</classname>
          and <classname>DataFileMetadataReader</classname> objects. Only one instance
          of each class is created. If the file set contains members which has the
          same validator or metadata reader, they will all share the same instance.
          </para>
        </listitem>
        
        <listitem>
          <para>
          Each validator/metadata reader class is initialised with calls to
          <methodname>DataFileHandler.setItem()</methodname> and 
          <methodname>DataFileHandler.setFile()</methodname>.
          </para>
        </listitem>
        
        <listitem>
          <para>
          Each validator is called. The result of the validation is saved for each
          file and can be retreieved by <methodname>FileSetMember.isValid()</methodname>
          and <methodname>FileSetMember.getErrorMessage()</methodname>.
          </para>
        </listitem>
        
        <listitem>
          <para>
          Each metadata reader is called, unless the metadata reader is the same class
          as the validator and the validation failed. If the metadata reader is a
          different class, it is called even if the validation failed.
          </para>
        </listitem>
        </orderedlist>

        <note>
          <title>Only one instance of each validator class is created</title>
          <para>
          The validation/metadata extraction is not done until all files have been 
          added to the fileset. If the same validator/meta data extractor is
          used for more than one file, the same instance is reused. Ie. 
          the <methodname>setFile()</methodname> is called one time
          for each file/file type pair. The <methodname>validate()</methodname>
          and <methodname>extractMetadata()</methodname> methods are only
          called once.
          </para>
        </note>
        
        <para>
          All validators and meta data extractors should extend
          the <classname>AbstractDataFileHandler</classname> class. The reason
          is that we may want to add more methods to the <interfacename>DataFileHandler</interfacename>
          interface in the future. The <classname>AbstractDataFileHandler</classname> will
          be used to provide default implementations for backwards compatibility.
        </para>
        
      </sect3>
      
      <sect3 id="core_api.data_in_files.import">
        <title>Import data into the database</title>
        
        <para>
          This should be done by existing plug-ins in the same way as before.
          A slight modification is needed since it is good if the importers
          are made aware of already selected files in the <classname>FileSet</classname>
          to provide good default values. Something like this.
        </para>
        
        <programlisting language="java">
RawBioAssay rba = ...
DbControl dc = ...

// Get the current raw data file, if any
List&lt;File&gt; rawDataFiles = FileStoreUtil.getGenericDataFiles(dc, rba, FileType.RAW_DATA);
File defaultFile = rawDataFiles.size() > 0 ?
   rawDataFiles.get(0) : null;
   
// Create parameter asking for input file - use current as default
PluginParameter&lt;File&gt; fileParameter = new PluginParameter&lt;File&gt;(
   "file",
   "Raw data file",
   "The file that contains the raw data that you want to import",
   new FileParameterType(defaultFile, true, 1)
);
</programlisting>

      <para>
        An import plug-in should also save the file that was used to the file set:
      </para>
      
      <programlisting language="java">
RawBioassay rba = ...
// The file the user selected to import from
File rawDataFile = (File)job.getValue("file");

// Save the file to the fileset. The method will check which file 
// type the platform uses as the raw data type. As a fallback the
// GENERIC_RAW_DATA type is used
FileStoreUtil.setGenericDataFile(dc, rba, FileType.RAW_DATA, 
   DataFileType.GENERIC_RAW_DATA, rawDataFile);
</programlisting>

      </sect3>
      
      <sect3 id="core_api.data_in_files.experiments">
        <title>Using raw data from files in an experiment</title>
        
        <para>
          Just as before, an experiment is still locked to a single
          <classname>RawDataType</classname>. This is a design issue that
          would break too many things if changed. If data is stored in files
          the experiment is also locked to a single <classname>Platform</classname>.
          This has been designed to have as little impact on existing
          plug-ins as possible. In most cases, the plug-ins will continue
          to work as before.
        </para>
        
        <para>
          A plug-in (using data from the database that needs to check if it can
          be used within an experiment can still do:
        </para>
        
        <programlisting language="java">
Experiment e = ...
RawDataType rdt = e.getRawDataType();
if (rdt.isStoredInDb())
{
   // Check number of channels, etc...
   // ... run plug-in code ...
}
</programlisting>
        
        <para>
          A newer plug-in which uses data from files should do:
        </para>
        
        <programlisting language="java">
Experiment e = ...
DbControl dc = ...
RawDataType rdt = e.getRawDataType();
if (!rdt.isStoredInDb())
{
   Platform p = rdt.getPlatform(dc);
   PlatformVariant v = rdt.getVariant(dc);
   // Check that platform/variant is supported
   // ... run plug-in code ...
}
</programlisting>
        
      </sect3>
      
    </sect2>
  </sect1>

  <sect1 id="api_overview.query_api">
    <title>The Query API</title>
    <para>
      This documentation is only available in the old format.
      See <ulink url="http://base.thep.lu.se/chrome/site/doc/development/overview/query/index.html"
        >http://base.thep.lu.se/chrome/site/doc/development/overview/query/index.html</ulink>
    </para>
    
  </sect1>
  
  <sect1 id="api_overview.dynamic_and_batch_api">
    <title>Analysis and the Dynamic and Batch API:s</title>
    <para>
      This documentation is only available in the old format.
      See <ulink url="http://base.thep.lu.se/chrome/site/doc/development/overview/dynamic/index.html"
        >http://base.thep.lu.se/chrome/site/doc/development/overview/dynamic/index.html</ulink>
    </para>
  </sect1>

  <sect1 id="api_overview.other_api">
    <title>Other useful classes and methods</title>
    <para>
      TODO
    </para>
  </sect1>
  
</chapter>