source: trunk/doc/src/docbook/developer/base_api.xml @ 7640

Last change on this file since 7640 was 7640, checked in by Nicklas Nordborg, 3 years ago

References #2136: Remove support for spot images

Updated documentation.

  • Property svn:eol-style set to native
  • Property svn:keywords set to Id
File size: 142.9 KB
Line 
1<?xml version="1.0" encoding="UTF-8"?>
2<!DOCTYPE chapter PUBLIC
3    "-//Dawid Weiss//DTD DocBook V3.1-Based Extension for XML and graphics inclusion//EN"
4    "../../../../lib/docbook/preprocess/dweiss-docbook-extensions.dtd">
5<!--
6  $Id: base_api.xml 7640 2019-03-11 13:13:29Z nicklas $
7
8  Copyright (C) 2007 Peter Johansson, Nicklas Nordborg, Martin Svensson
9
10  This file is part of BASE - BioArray Software Environment.
11  Available at http://base.thep.lu.se/
12
13  BASE is free software; you can redistribute it and/or
14  modify it under the terms of the GNU General Public License
15  as published by the Free Software Foundation; either version 3
16  of the License, or (at your option) any later version.
17
18  BASE is distributed in the hope that it will be useful,
19  but WITHOUT ANY WARRANTY; without even the implied warranty of
20  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
21  GNU General Public License for more details.
22
23  You should have received a copy of the GNU General Public License
24  along with BASE. If not, see <http://www.gnu.org/licenses/>.
25-->
26
27<chapter id="base_api">
28  <?dbhtml dir="api" filename="index.html" ?>
29  <title>The BASE API</title>
30
31  <sect1 id="base_api.public">
32    <?dbhtml filename="public_api.html" ?>
33    <title>The Public API of BASE</title>
34   
35    <para>
36      Not all public classes and methods in the <filename>base-*.jar</filename>
37      files and other JAR files shipped with BASE are considered as
38      <emphasis>Public API</emphasis>. This is important knowledge
39      since we will always try to maintain backwards compatibility
40      for classes that are part of the public API. For other
41      classes, changes may be introduced at any time without
42      notice or specific documentation. In other words:
43    </para>
44   
45    <note>
46      <title>Only use the public API when developing plug-ins and extensions</title>
47      <para>
48        This will maximize the chance that your code will continue
49        to work with the next BASE release. If you use the non-public API
50        you do so at your own risk.
51      </para>
52    </note>
53   
54    <para>
55      See the <ulink url="../../../api/index.html"
56        >BASE API javadoc</ulink> for information about
57      what parts of the API that contributes to the public API.
58      Methods, classes and other elements that have been tagged as
59      <code>@deprecated</code> should be considered as part of the internal API
60      and may be removed in a subsequent release without warning.
61    </para>
62   
63    <para>
64      Keeping the backwards compatibility is an aim only. It may not
65      always be possible. See <xref linkend="appendix.incompatible" /> to
66      read more about changes that have been introduced by each release
67      that may affect existing code.
68    </para>
69
70    <sect2 id="base_api.compatibility">
71      <title>What is backwards compatibility?</title>
72     
73      <para>
74        There is a great article about this subject on <ulink 
75        url="http://wiki.eclipse.org/index.php/Evolving_Java-based_APIs"
76          >http://wiki.eclipse.org/index.php/Evolving_Java-based_APIs</ulink>.
77        This is what we will try to comply with. If you do not want to
78        read the entire article, here are some of the most important points:
79      </para>
80     
81     
82      <sect3 id="core_api.compatibility.binary">
83        <title>Binary compatibility</title>
84        <para>
85        <blockquote>
86          Pre-existing Client binaries must link and run with new releases of the
87          Component without recompiling.
88        </blockquote>
89       
90        For example:
91        <itemizedlist>
92        <listitem>
93          <para>
94            We cannot change the number or types of parameters to a method
95            or constructor.
96          </para>
97        </listitem>
98        <listitem>
99          <para>
100            We cannot add or change methods to interfaces that are intended
101            to be implemented by plug-in or client code.
102          </para>
103        </listitem>
104        </itemizedlist>
105        </para>       
106      </sect3>
107     
108      <sect3 id="base_api.compatibility.contract">
109        <title>Contract compatibility</title>
110        <para>
111          <blockquote>
112          API changes must not invalidate formerly legal Client code.
113          </blockquote>
114       
115          For example:
116          <itemizedlist>
117          <listitem>
118            <para>
119              We cannot change the implementation of a method to do
120              things differently than before. For example, allow <constant>null</constant>
121              as a return value when it was not allowed before.
122            </para>
123          </listitem>
124          </itemizedlist>
125       
126          <note>
127            <para>
128            Sometimes there is a very fine line between what is considered a
129            bug and what is considered a feature. For example, if the
130            actual implementation does not do what the javadoc says,
131            do we change the code or do we change the documentation?
132            This has to be considered from case to case and depends on
133            the age of the code and if we expect plug-ins and clients to be
134            affected by it or not.
135            </para>
136          </note>
137        </para>
138      </sect3>
139     
140      <sect3 id="base_api.compatibility.source">
141        <title>Source code compatibility</title>
142        <para>
143        This is not an important matter and is not always possible to
144        achieve. In most cases, the problems are easy to fix.
145        Example:
146       
147        <itemizedlist>
148        <listitem>
149          <para>
150          Adding a class may break a plug-in or client that import
151          classes with <constant>.*</constant> if the same class name
152          exists in another package.
153          </para>
154        </listitem>
155        </itemizedlist>
156        </para>
157      </sect3>
158    </sect2>
159  </sect1>
160
161  <sect1 id="base_api.data" chunked="1">
162    <?dbhtml filename="data_api.html" ?>
163    <title>The Data Layer API</title>
164
165    <para>
166      This section gives an overview of the entire data layer API.
167      The figure below show how different modules relate to each other.
168    </para>
169 
170    <figure id="data_api.figures.overview">
171      <title>Data layer overview</title>
172      <screenshot>
173        <mediaobject>
174          <imageobject>
175            <imagedata 
176              align="center"
177              scalefit="1" width="100%"
178              fileref="figures/uml/datalayer.overview.png" format="PNG" />
179          </imageobject>
180        </mediaobject>
181      </screenshot>
182    </figure>
183
184    <sect2 id="data_api.basic">
185      <title>Basic classes and interfaces</title>
186     
187      <para>
188        This document contains information about the basic classes and interfaces in this package.
189        They are important since all data-layer classes must inherit from one of the already
190        existing abstract base classes or implement one or more of the
191        existing interfaces. They contain code that is common to all classes,
192        for example implementations of the <methodname>equals()</methodname>
193        and <methodname>hashCode()</methodname> methods or how to link with the owner of an
194        item.
195      </para>
196     
197        <figure id="data_api.figures.basic">
198          <title>Basic classes and interfaces</title>
199          <screenshot>
200            <mediaobject>
201              <imageobject>
202                <imagedata 
203                  align="center"
204                  fileref="figures/uml/datalayer.basic.png" format="PNG" />
205              </imageobject>
206            </mediaobject>
207          </screenshot>
208        </figure>
209     
210      <sect3 id="data_api.basic.classes">
211        <title>Classes</title>
212       
213        <variablelist>
214        <varlistentry>
215          <term><classname docapi="net.sf.basedb.core.data">BasicData</classname></term>
216          <listitem>
217            <para>
218            The root class. It overrides the <methodname>equals()</methodname>,
219            <methodname>hashCode()</methodname> and <methodname>toString()</methodname> methods
220            from the <classname>Object</classname> class. It also defines the
221            <varname>id</varname> and <varname>version</varname> properties.
222            All data layer classes must inherit from this class or one of it's subclasses.
223            </para>
224          </listitem>
225        </varlistentry>
226       
227        <varlistentry>
228          <term><classname docapi="net.sf.basedb.core.data">OwnedData</classname></term>
229          <listitem>
230            <para>
231            Extends the <classname>BasicData</classname> class and adds
232            an <varname>owner</varname> property. The owner is a required link to a
233            <classname docapi="net.sf.basedb.core.data">UserData</classname> object, representing the user that
234            is the owner of the item.
235            </para>
236          </listitem>
237        </varlistentry>
238
239        <varlistentry>
240          <term><classname docapi="net.sf.basedb.core.data">SharedData</classname></term>
241          <listitem>
242            <para>
243            Extends the <classname>OwnedData</classname> class and adds
244            properties (<varname>itemKey</varname> and <varname>projectKey</varname>)
245            that holds access permission information for an item.
246            Access permissions are held in <classname docapi="net.sf.basedb.core.data">ItemKeyData</classname> and/or
247            <classname docapi="net.sf.basedb.core.data">ProjectKeyData</classname> objects. These objects only exists if
248            the item has been shared.
249            </para>
250          </listitem>
251        </varlistentry>
252
253        <varlistentry>
254          <term><classname docapi="net.sf.basedb.core.data">CommonData</classname></term>
255          <listitem>
256            <para>
257            This is a convenience class for items that extends the <classname>SharedData</classname>
258            class and implements the <interfacename docapi="net.sf.basedb.core.data">NameableData</interfacename> and
259            <interfacename docapi="net.sf.basedb.core.data">RemoveableData</interfacename> interfaces. This is one of
260            the most common situations.
261            </para>
262          </listitem>
263        </varlistentry>
264
265        <varlistentry>
266          <term><classname docapi="net.sf.basedb.core.data">AnnotatedData</classname></term>
267          <listitem>
268            <para>
269            This is a convenience class for items that can be annotated.
270            Annotations are held in <classname docapi="net.sf.basedb.core.data">AnnotationSetData</classname> objects.
271            The annotation set only exists if annotations has been created for the item.
272            </para>
273          </listitem>
274        </varlistentry>
275        </variablelist>
276       
277      </sect3>
278     
279      <sect3 id="data_api.basic.interfaces">
280        <title>Interfaces</title>
281       
282        <variablelist>
283        <varlistentry>
284          <term><classname docapi="net.sf.basedb.core.data">IdentifiableData</classname></term>
285          <listitem>
286            <para>
287            All items are identifiable, which means that they have a unique <varname>id</varname>.
288            The id is unique for all items of a specific type (ie. class). The id is number
289            that is automatically generated by the database and has no other meaning
290            outside of the application. The <varname>version</varname> property is used for
291            detecting and preventing concurrent modifications to an item.
292            </para>
293          </listitem>
294        </varlistentry>
295       
296        <varlistentry>
297          <term><classname docapi="net.sf.basedb.core.data">OwnableData</classname></term>
298          <listitem>
299            <para>
300            An ownable item is an item which has an owner. The owner is represented as a
301            required link to a <classname docapi="net.sf.basedb.core.data">UserData</classname> object.
302            </para>
303          </listitem>
304        </varlistentry>       
305
306        <varlistentry>
307          <term><classname docapi="net.sf.basedb.core.data">ShareableData</classname></term>
308          <listitem>
309            <para>
310            A shareable item is an item which can be shared to other users, groups or projects.
311            Access permissions are held in <classname docapi="net.sf.basedb.core.data">ItemKeyData</classname> and/or
312            <classname docapi="net.sf.basedb.core.data">ProjectKeyData</classname> objects.
313            </para>
314          </listitem>
315        </varlistentry>
316             
317        <varlistentry>
318          <term><classname docapi="net.sf.basedb.core.data">NameableData</classname></term>
319          <listitem>
320            <para>
321            A nameable item is an item that has a name (required) and a description
322            (optional). The name doesn't have to be unique, except in a few special
323            cases (for example, the name of a file).
324            </para>
325          </listitem>
326        </varlistentry>
327       
328        <varlistentry>
329          <term><classname docapi="net.sf.basedb.core.data">RemovableData</classname></term>
330          <listitem>
331            <para>
332            A removable item is an item that can be flagged as removed. This doesn't
333            remove the information about the item from the database, but can be used by
334            client applications to hide items that the user is not interested in.
335            A trashcan function can be used to either restore or permanently
336            remove items that has the flag set.
337            </para>
338          </listitem>
339        </varlistentry>
340               
341        <varlistentry>
342          <term><classname docapi="net.sf.basedb.core.data">SystemData</classname></term>
343          <listitem>
344            <para>
345            A system item is an item which has an additional id in the form of string. A system id
346            is required when we need to make sure that we can get a specific item without
347            knowing the numeric id. Example of such items are the root user and the everyone group.
348            A system id is generally constructed like:
349            <constant>net.sf.basedb.core.User.ROOT</constant>. The system id:s are defined in the
350            core layer by each item class.
351            </para>
352          </listitem>
353        </varlistentry>
354
355        <varlistentry>
356          <term><classname docapi="net.sf.basedb.core.data">DiskConsumableData</classname></term>
357          <listitem>
358            <para>
359            This interface is used by items which occupies a lot of disk space and
360            should be part of the quota system, for example files. The required
361            <classname docapi="net.sf.basedb.core.data">DiskUsageData</classname> contains information about the size,
362            location, owner etc. of the item.
363            </para>
364          </listitem>
365        </varlistentry>
366       
367        <varlistentry>
368          <term><classname docapi="net.sf.basedb.core.data">AnnotatableData</classname></term>
369          <listitem>
370            <para>
371            This interface is used by items which can be annotated. Annotations are name/value
372            pairs that are attached as extra information to an item. All annotations are
373            contained in an <classname docapi="net.sf.basedb.core.data">AnnotationSetData</classname> object.
374            </para>
375          </listitem>
376        </varlistentry>
377       
378        <varlistentry>
379          <term><classname docapi="net.sf.basedb.core.data">ExtendableData</classname></term>
380          <listitem>
381            <para>
382            This interface is used by items which can have extra administrator-defined
383            columns. The functionality is similar to annotations. It is not as flexible,
384            since it is a global configuration, but has better performance. BASE will
385            generate extra database columns to store the data in the tables for items that
386            can be extended.
387            </para>
388          </listitem>
389        </varlistentry>
390       
391        <varlistentry>
392          <term><classname docapi="net.sf.basedb.core.data">BatchableData</classname></term>
393          <listitem>
394            <para>
395            This interface is a tagging interface which is used by items that needs batch
396            functionality in the core.
397            </para>
398          </listitem>
399        </varlistentry>
400       
401        <varlistentry>
402          <term><interfacename docapi="net.sf.basedb.core.data">RegisteredData</interfacename></term>
403          <listitem>
404            <para>
405            This interface is used by items which registered the date they were
406            created in the database. The registration date is set at creation time
407            and can't be modified later. Since this didn't exist prior to BASE 2.10,
408            null values are allowed on all pre-existing items. Note! For backwards
409            compatibility reasons with existing code in
410            <classname docapi="net.sf.basedb.core.data">BioMaterialEventData</classname>
411            the method name is <methodname>getEntryDate()</methodname>.
412            </para>
413          </listitem>
414        </varlistentry>
415       
416        <varlistentry>
417          <term><interfacename docapi="net.sf.basedb.core.data">LoggableData</interfacename></term>
418          <listitem>
419            <para>
420            This is a tagging interface that indicates that the <classname 
421            docapi="net.sf.basedb.core.log.db">DbLogManagerFactory</classname> logging
422            implementation should log changes made to items that implements it.
423            </para>
424          </listitem>
425        </varlistentry>
426
427        <varlistentry>
428          <term><interfacename docapi="net.sf.basedb.core.data">FileStoreEnabledData</interfacename></term>
429          <listitem>
430            <para>
431            This interface is implemented by all items that can have files with related data
432            attached to them. The file types that can be used for a specific item are usually
433            determined by the main type, the subtype or platform.
434            </para>
435          </listitem>
436        </varlistentry>
437
438        <varlistentry>
439          <term><interfacename docapi="net.sf.basedb.core.data">SubtypableData</interfacename></term>
440          <listitem>
441            <para>
442            This interface should be implemented by all items that can be subtyped.
443            Unless otherwise noted the subtype is always an optional link to
444            a <classname docapi="net.sf.basedb.core.data">ItemSubtypeData</classname>.
445            item. In the simplest form, the subtype is a kind of an annotation, but
446            for items that also implements the <interfacename 
447            docapi="net.sf.basedb.core.data">FileStoreEnabledData</interfacename>
448            interface, the subtype can be used to specify the file types that
449            are applicable for each item.
450            </para>
451          </listitem>
452        </varlistentry>
453        </variablelist>
454
455      </sect3>
456    </sect2>
457   
458    <sect2 id="data_api.authentication">
459      <title>User authentication and access control</title>
460     
461      <para>
462         This section gives an overview of user authentication and
463         how groups, roles and projects are used for access control
464         to items.
465      </para>
466     
467        <figure id="data_api.figures.authentication">
468          <title>User authentication and access control</title>
469          <screenshot>
470            <mediaobject>
471              <imageobject>
472                <imagedata 
473                  align="center"
474                  scalefit="1" width="100%"
475                  fileref="figures/uml/datalayer.authentication.png" format="PNG" />
476              </imageobject>
477            </mediaobject>
478          </screenshot>
479        </figure>
480     
481      <sect3 id="data_api.authentication.users">
482        <title>Users and passwords</title>     
483     
484        <para>
485          The <classname docapi="net.sf.basedb.core.data">UserData</classname> class holds information about users.
486          We keep the passwords in a separate table and use proxies to avoid loading
487          password data each time a user is loaded to minimize security risks. It is
488          only if the password needs to be changed that the <classname docapi="net.sf.basedb.core.data">PasswordData</classname>
489          object is loaded. The one-to-one mapping between user and password is controlled
490          by the password class, but a cascade attribute on the user class makes sure
491          that the password is deleted when a user is deleted.
492        </para>
493      </sect3>
494
495      <sect3 id="data_api.authentication.groups">
496        <title>Groups, roles, projects and permission template</title>     
497     
498        <para>
499          The <classname docapi="net.sf.basedb.core.data">GroupData</classname>,
500          <classname docapi="net.sf.basedb.core.data">RoleData</classname> and
501          <classname docapi="net.sf.basedb.core.data">ProjectData</classname> classes holds
502          information about groups, roles
503          and projects respectively. A user may be a member of any number of groups,
504          roles and/or projects. New users are automatically added as members of all
505          groups and roles that has the <varname>default</varname> property set.
506        </para>
507         
508        <para>
509          The membership in a project comes with an attached
510          permission values. This is the highest permission the user has in the
511          project. No matter what permission an item has been shared with the
512          user will not get higher permission. Groups may be members of other groups and
513          also in projects. A <classname docapi="net.sf.basedb.core.data">PermissionTemplateData</classname>
514          is just a holder for permissions that users can use when sharing items. The
515          template is never part of the actual permission control mechanism.
516        </para>
517       
518        <para>
519          Group membership is always accounted for, but the core only allows
520          one project at a time to be use, this is the <emphasis>active project</emphasis>.
521          When a project is active new items that are created are automatically
522          shared according to the settings for the project. There are two cases.
523          If the project has a permission template, the new item is given the same
524          permissions as the template has. If the project doesn't have a permission
525          template, the new item is shared to the active project with the permission
526          given by the <varname>autoPermission</varname> property. Note that in the
527          first case the new item may or may not be shared to the active project
528          depending on if the template is shared to the project or not.
529        </para>
530       
531        <para>
532          Note that the permission template is only used (by the core) when creating
533          new items. The permissions held by the template are copied and when the new item
534          has been saved to the database there is no longer any reference back to
535          the template that was used to create it. This means that changes to the
536          template does not affect already existing items and that the template
537          can be deleted without problems.
538        </para>
539      </sect3>
540     
541      <sect3 id="data_api.authentication.keys">
542        <title>Keys</title>     
543     
544        <para>
545          The <classname docapi="net.sf.basedb.core.data">KeyData</classname> class and it's subclasses
546          <classname docapi="net.sf.basedb.core.data">ItemKeyData</classname>, <classname docapi="net.sf.basedb.core.data">ProjectKeyData</classname> and
547          <classname docapi="net.sf.basedb.core.data">RoleKeyData</classname>, are used to store information about access
548          permissions to items. To get permission to manipulate an item a user must have
549          access to a key giving that permission. There are three types of keys:
550        </para>
551       
552        <variablelist>
553        <varlistentry>
554          <term><classname docapi="net.sf.basedb.core.data">ItemKey</classname></term>
555          <listitem>
556            <para>
557            Is used to give a user or group access to a specific item. The item
558            must be a <interfacename docapi="net.sf.basedb.core.data">ShareableData</interfacename> item.
559            The permissions are usually set by the owner of the item. Once created an
560            item key cannot be changed. This allows the core to reuse a key if the
561            permissions match exactly, ie. for a given set of users/groups/permissions
562            there can be only one item key object.
563            </para>
564          </listitem>
565        </varlistentry>
566
567        <varlistentry>
568          <term><classname docapi="net.sf.basedb.core.data">ProjectKey</classname></term>
569          <listitem>
570            <para>
571            Is used to give members of a project access to a specific item. The item
572            must be a <interfacename docapi="net.sf.basedb.core.data">ShareableData</interfacename> item. Once created a
573            project key cannot be changed. This allows the core to reuse a key if the
574            permissions match exactly, ie. for a given set of projects/permissions
575            there can be only one project key object.
576            </para>
577          </listitem>
578        </varlistentry>
579
580        <varlistentry>
581          <term><classname docapi="net.sf.basedb.core.data">RoleKey</classname></term>
582          <listitem>
583            <para>
584            Is used to give a user access to all items of a specific type, ie.
585            <constant>READ</constant> all <constant>SAMPLES</constant>. The installation
586            will make sure that there already exists a role key for each type of item, and
587            it is not possible to add new or delete existing keys. Unlike the other two types
588            this key can be modified.
589            </para>
590           
591            <para>
592            A role key is also used to assign permissions to plug-ins. If a plug-in has
593            been specified to use permissions the default is to deny everything.
594            The mapping to the role key is used to grant permissions to the plugin.
595            The <varname>granted</varname> value gives the plugin access to all items
596            of the related item type regardless of if the user that is running the plug-in has the
597            permission or not. The <varname>denied</varname> values denies access to all
598            items of the related item type even if the logged in user has the permission.
599            Permissions that are not granted nor denied are checked against the
600            logged in users regular permissions. Permissions to items that are
601            not linked are always denied.
602            </para>
603          </listitem>
604        </varlistentry>
605        </variablelist>
606       
607      </sect3>
608
609      <sect3 id="data_api.authentication.permissions">
610        <title>Permissions</title>
611       
612        <para>
613          The <varname>permission</varname> property appearing in many classes is an
614          integer values describing the permission:
615        </para>
616       
617        <informaltable>
618        <tgroup cols="2">
619          <colspec colname="value" />
620          <colspec colname="permission" />
621          <thead>
622            <row>
623              <entry>Value</entry>
624              <entry>Permission</entry>
625            </row>
626          </thead>
627          <tbody>
628            <row>
629              <entry>1</entry>
630              <entry>Read</entry>
631            </row>
632            <row>
633              <entry>3</entry>
634              <entry>Use</entry>
635            </row>
636            <row>
637              <entry>7</entry>
638              <entry>Restricted write</entry>
639            </row>
640            <row>
641              <entry>15</entry>
642              <entry>Write</entry>
643            </row>
644            <row>
645              <entry>31</entry>
646              <entry>Delete</entry>
647            </row>
648            <row>
649              <entry>47 (=32+15)</entry>
650              <entry>Set owner</entry>
651            </row>
652            <row>
653              <entry>79 (=64+15)</entry>
654              <entry>Set permissions</entry>
655            </row>
656            <row>
657              <entry>128</entry>
658              <entry>Create</entry>
659            </row>
660            <row>
661              <entry>256</entry>
662              <entry>Denied</entry>
663            </row>
664          </tbody>
665        </tgroup>
666        </informaltable>
667       
668        <para>
669          The values are constructed so that
670          <constant>READ</constant> -&gt;
671          <constant>USE</constant> -&gt;
672          <constant>RESTRICTED_WRITE</constant> -&gt;
673          <constant>WRITE</constant> -&gt;
674          <constant>DELETE</constant>
675          are chained in the sense that a higher permission always implies the lower permissions
676          also. The <constant>SET_OWNER</constant> and <constant>SET_PERMISSION</constant>
677          both implies <constant>WRITE</constant> permission. The <constant>DENIED</constant>
678          permission is only valid for role keys, and if specified it overrides all
679          other permissions.               
680        </para>
681       
682        <para>
683          When combining permission for a single item the permission codes for the different
684          paths are OR-ed together. For example a user has a role key with <constant>READ</constant>
685          permission for <constant>SAMPLES</constant>, but also an item key with <constant>USE</constant>
686          permission for a specific sample. Of course, the resulting permission for that
687          sample is <constant>USE</constant>. For other samples the resulting permission is
688          <constant>READ</constant>.
689        </para>
690       
691        <para>
692          If the user is also a member of a project which has <constant>WRITE</constant>
693          permission for the same sample, the user will have <constant>WRITE</constant>
694          permission when working with that project.
695        </para>
696       
697        <para>
698          The <constant>RESTRICTED_WRITE</constant> permission is in most cases the same
699          as the <constant>WRITE</constant> permission. So far the <constant>RESTRICTED_WRITE</constant>
700          permission is only given to users to their own <classname docapi="net.sf.basedb.core.data">UserData</classname>
701          object so they can change their address and other contact information,
702          but not quota, expiration date and other administrative information.
703        </para>
704
705      </sect3>
706    </sect2>
707
708    <sect2 id="data_api.reporters">
709      <title>Reporters</title>
710      <para>
711         This section gives an overview of reporters in BASE.
712      </para>
713     
714        <figure id="data_api.figures.reporters">
715          <title>Reporters</title>
716          <screenshot>
717            <mediaobject>
718              <imageobject>
719                <imagedata 
720                  align="center"
721                  fileref="figures/uml/datalayer.reporters.png" format="PNG" />
722              </imageobject>
723            </mediaobject>
724          </screenshot>
725        </figure>
726     
727      <sect3 id="data_api.reporters.description">
728        <title>Reporters</title>
729        <para>
730          The <classname docapi="net.sf.basedb.core.data">ReporterData</classname> class holds information about reporters.
731          The <property>externalId</property> is a required property that must be unique
732          among all reporters. The external ID is the value BASE uses to match
733          reporters when importing data from files.
734        </para>
735       
736        <para>
737          The <classname>ReporterData</classname> is an <emphasis>extendable</emphasis>
738          class, which means that the server administrator can define additional
739          columns (=annotations) in the reporters table. These are accessed with
740          the <methodname>ReporterData.getExtended()</methodname> and
741          <methodname>ReporterData.setExtended()</methodname> methods.
742          See <xref linkend="appendix.extendedproperties" /> for more information about
743          this.
744        </para>
745       
746        <para>
747          The <classname>ReporterData</classname> is also a <emphasis>batchable</emphasis>
748          class which means that there is no corresponding class in the core
749          layer. Client applications and plug-ins should work directly with
750          the <classname>ReporterData</classname> class. To help manage the reporters
751          there is the <classname docapi="net.sf.basedb.core">Reporter</classname> and <classname docapi="net.sf.basedb.core">ReporterBatcher</classname>
752          classes. The main reason for this
753          is to increase the performance and lower the memory usage by bypassing
754          internal caching in the core and Hibernate. Performance is also
755          increased by the batchers which uses more efficient SQL against the
756          database than Hibernate.
757        </para>
758       
759        <para>
760          The
761          <property>lastUpdate</property>
762          property holds the data and time the reporter information was last updated. The
763          value is managed automatically by the
764          <classname>ReporterBatcher</classname>
765          class. That goes for
766          <property>lastSource</property>
767          property too, which holds information about where the last update comes from. By
768          default this is set to the name of the logged in user, but it can be changed by
769          calling
770          <methodname>ReporterBatcher.setUpdateSource(String source)</methodname>
771          before the batcher commits the updates to the database. The source-string
772          should have the format: <synopsis>[ITEM_TYPE]:[ITEM_NAME]</synopsis> where,in
773          the file-case, ITEM_TYPE is File and ITEM_NAME is the file's name.
774        </para>
775      </sect3>
776     
777      <sect3 id="data_api.reporters.lists">
778        <title>Reporter lists</title>
779       
780        <para>
781          Reporter lists can be used to group reporters that are somehow related
782          to each other. This could for example be a list of interesting reporters
783          found in the analysis of an experiment. Each reporter in the list may
784          optionally be assigned a score. The meaning of the score value is not
785          interpreted by BASE.
786        </para>
787       
788      </sect3>
789     
790     
791    </sect2>
792
793    <sect2 id="data_api.quota">
794      <title>Quota and disk usage</title>
795      <para>
796         This section gives an overview of quota system in BASE
797         and how the disk usage is kept track of.
798      </para>
799     
800        <figure id="data_api.figures.quota">
801          <title>Quota and disk usage</title>
802          <screenshot>
803            <mediaobject>
804              <imageobject>
805                <imagedata 
806                  align="center"
807                  fileref="figures/uml/datalayer.quota.png" format="PNG" />
808              </imageobject>
809            </mediaobject>
810          </screenshot>
811        </figure>
812     
813      <sect3 id="data_api.quota.description">
814        <title>Quota</title>
815       
816        <para>
817          The <classname docapi="net.sf.basedb.core.data">QuotaData</classname> holds information about a
818          single quota registration. The same quota may be used by many different users
819          and groups. This object encapsulates allowed
820          quota values for different types of quota types and locations.
821          BASE defines several quota types (file, raw data and experiment),
822          and locations (primary, external and offline).
823        </para>
824       
825        <para>
826          The <property>quotaValues</property> property is a set containing
827          <classname docapi="net.sf.basedb.core.data">QuotaIndex</classname> which hold the
828          maximum byte values for a specified quota type and location. The set must contain at
829          least one entry for the total quota at the primary location.
830        </para>
831       
832      </sect3>
833     
834      <sect3 id="data_api.quota.diskusage">
835        <title>Disk usage</title>
836       
837        <para>
838          A <interfacename docapi="net.sf.basedb.core.data">DiskConsumableData</interfacename> (for example a file)
839          item is automatically linked to a <classname docapi="net.sf.basedb.core.data">DiskUsageData</classname>
840          item. This holds information about the number of bytes,
841          the location and quota type the item uses. It also holds information
842          about which user and group (optional) that should be charged for the disk usage.
843          The user is always the owner of the item.
844        </para>
845
846      </sect3>
847     
848    </sect2>
849
850    <sect2 id="data_api.clients">
851      <title>Client, session and settings</title>
852      <para>
853         This section gives an overview of client applications, sessions and settings.
854      </para>
855     
856        <figure id="data_api.figures.clients">
857          <title>Client, sessions and settings</title>
858          <screenshot>
859            <mediaobject>
860              <imageobject>
861                <imagedata 
862                  align="center"
863                  scalefit="1" width="100%"
864                  fileref="figures/uml/datalayer.clients.png" format="PNG" />
865              </imageobject>
866            </mediaobject>
867          </screenshot>
868        </figure>
869
870      <sect3 id="data_api.clients.description">
871        <title>Clients</title>
872        <para>
873          The <classname docapi="net.sf.basedb.core.data">ClientData</classname> class holds information
874          about a client application. The <property>externalId</property>
875          property is a unique identifier for the application. To avoid ID clashes the ID
876          should be constructed in the same way as Java packages, for example
877          <constant>net.sf.basedb.clients.web</constant> is the ID for the
878          web client application.
879        </para>
880       
881        <para>
882          A client application doesn't have to be registered with BASE
883          to be able to use it. But we recommend it since:
884        </para>
885       
886        <itemizedlist>
887        <listitem>
888          <para>
889            The permission system allows an admin to specify exactly
890            which users that may use a specific application.
891          </para>
892        </listitem>
893       
894        <listitem>
895          <para>
896          The application can't store any context-sensitive or application-specific
897          settings unless it is registered.
898          </para>
899        </listitem>
900       
901        <listitem>
902          <para>
903          The application can store context-sensitive help in the BASE
904          database.
905          </para>
906        </listitem>
907        </itemizedlist>
908      </sect3>
909     
910      <sect3 id="data_api.clients.sessions">
911        <title>Sessions</title>
912       
913        <para>
914          A session represents the time between login and logout for a single
915          user. The <classname docapi="net.sf.basedb.core.data">SessionData</classname> object is entirely
916          managed by the BASE core, and should be considered read-only
917          for client applications.
918        </para>
919           
920      </sect3>
921     
922      <sect3 id="data_api.clients.settings">
923        <title>Settings</title>
924       
925        <para>
926          There are two types of settings: context-sensitive settings and regular
927          settings. The regular settings are simple key-value pairs of strings
928          and can be used for almost anything. There are four subtypes:
929        </para>
930       
931        <itemizedlist>
932        <listitem>
933          <para>
934          Global default settings: Settings that are used by all users
935          and client applications on the BASE server. These settings
936          are read-only except for administrators. BASE has not yet defined
937          any settings of this type.
938          </para>
939        </listitem>
940       
941        <listitem>
942          <para>
943          User default settings: Settings that are valid for a single user
944          for any client application. BASE has not yet defined
945          any settings of this type.
946          </para>
947        </listitem>
948       
949        <listitem>
950          <para>
951          Client default settings: Settings that are valid for all users using
952          a specific client application. Each client application is responsible
953          for defining it's own settings. Settings are read-only except
954          for administrators.
955          </para>
956        </listitem>
957       
958        <listitem>
959          <para>
960          User client settings: Settings that are valid for a single user using
961          a specific client application. Each client application is responsible
962          for defining it's own settings.
963          </para>
964        </listitem>
965       
966        </itemizedlist>
967       
968        <para>
969          The context-sensitive settings are designed to hold information
970          about the current status of options related to the listing of items
971          of a specific type. This includes:
972        </para>
973       
974        <itemizedlist>
975        <listitem>
976          <para>
977          Current filtering options (as 1 or more <classname docapi="net.sf.basedb.core.data">PropertyFilterData</classname>
978          objects).
979          </para>
980        </listitem>
981       
982        <listitem>
983          <para>
984          Which columns and direction to use for sorting.
985          </para>
986        </listitem>
987       
988        <listitem>
989          <para>
990          The number of items to display on each page, and which page that
991          is the current page.
992          </para>
993        </listitem>
994       
995        <listitem>
996          <para>
997          Simple key-value settings related to a given context.
998          </para>
999        </listitem>
1000        </itemizedlist>
1001       
1002        <para>
1003          Context-sensitive settings are only accessible if a client
1004          application has been registered. The settings may be
1005          named to make it possible to store several presets and to
1006          quickly switch between them. In any case, BASE maintains a
1007          current default setting with an empty name. An administrator
1008          may mark a named setting as public to allow other users to
1009          use it.
1010        </para>
1011       
1012      </sect3>
1013     
1014     
1015    </sect2>
1016
1017    <sect2 id="data_api.files">
1018      <title>Files and directories</title>
1019
1020      <para>
1021        This section covers the details of the BASE file
1022        system.
1023      </para>
1024
1025        <figure id="data_api.figures.files">
1026          <title>Files and directories</title>
1027          <screenshot>
1028            <mediaobject>
1029              <imageobject>
1030                <imagedata 
1031                  align="center"
1032                  fileref="figures/uml/datalayer.files.png" format="PNG" />
1033              </imageobject>
1034            </mediaobject>
1035          </screenshot>
1036        </figure>
1037     
1038        <para>
1039          The <classname docapi="net.sf.basedb.core.data">DirectoryData</classname> class holds
1040          information about directories. Directories are organised in the
1041          ususal way as as tree structure. All directories must have
1042          a parent directory, except the system-defined root directory.
1043        </para>
1044       
1045        <para>
1046          The <classname docapi="net.sf.basedb.core.data">FileData</classname> class holds information about
1047          a file. The actual file contents is stored on disk in the directory
1048          specified by the <varname>userfiles</varname> setting in
1049          <filename>base.config</filename>. The <varname>internalName</varname>
1050          property is the name of the file on disk, but this is never exposed to
1051          client applications. The filenames and directories
1052          on the disk doesn't correspond to the the filenames and directories in
1053          BASE.
1054        </para>
1055       
1056        <para>
1057          The <varname>url</varname> property is used for file items which are stored in
1058          an external location. In this case there is no local file data on the
1059          BASE server.
1060        </para>
1061       
1062        <para>
1063          The <varname>location</varname> property can take three values:
1064        </para>
1065       
1066        <itemizedlist>
1067        <listitem>
1068          <para>
1069          0 = The file is offline, ie. there is no file on the disk
1070          </para>
1071        </listitem>
1072        <listitem>
1073          <para>
1074          1 = The file is in primary storage, ie. it is located on the disk
1075          and can be used by BASE
1076          </para>
1077        </listitem>
1078        <listitem>
1079          <para>
1080          3 = The file is an external file whose location is referenced by the
1081          <varname>url</varname> property. If the file is protected by passwords
1082          or certificates the file item may reference a
1083          <classname docapi="net.sf.basedb.core.data">FileServerData</classname>
1084          object. Note that an external file in most cases can be used by client
1085          applications/plug-ins as if the file was stored locally on the BASE
1086          server.
1087          </para>
1088        </listitem>
1089        </itemizedlist>
1090
1091        <para>
1092          The <varname>md5</varname> property can be used to check for file
1093          corruption when a user re-uploads a file that has been offline.
1094        </para>
1095       
1096        <para>
1097          BASE can store files in a compressed format. This is handled internally
1098          and is not visible to client applications. The <varname>compressed</varname>
1099          and <varname>compressedSize</varname> properties are used to store information
1100          about this. A file may always be compressed if the users says so, but
1101          BASE can also do this automatically if the file is uploaded
1102          to a directory with the <varname>autoCompress</varname> flag set
1103          or if the file has MIME type with the <varname>autoCompress</varname>
1104          flag set.
1105        </para>
1106       
1107        <para>
1108          The <classname docapi="net.sf.basedb.core.data">FileServerData</classname> class
1109          holds information about an external file server. If the <varname>connectionManagerFactory</varname>
1110          isn't set BASE automatically selects a factory based on the URL of the file. There is
1111          built-in support for HTTP and HTTPS, but it is possible to install extensions for
1112          support for other protocols. The <varname>host</varname> property can be set
1113          to override the host part of the URL from the file. See <xref 
1114          linkend="extensions_developer.connection_manager" /> for more
1115          information about connection managers.
1116        </para>
1117       
1118        <para>
1119          The <varname>username</varname> and <varname>password</varname> properties are used if
1120          the server requires the user to be logged in. BASE has built-in support for Basic and
1121          Digest authentication. The <varname>serverCertificate</varname> can be used with HTTPS
1122          servers that uses a non-trusted certificate to tell BASE to trust the server anyway.
1123          In most cases, this is only needed if the server uses a self-signed certificate, but could, for
1124          example, also be used if a trusted site has forgot to renew an expired certificate.
1125          The server certificate should be an X.509 certificate in either binary or text format.
1126          The <varname>clientCertificate</varname> and <varname>clientPassword</varname>
1127          properties are used for servers that require that users present a valid client
1128          certificate before they are allowed access. The client certificate is usually issued
1129          by the server operator and must be in PKCS #12 format.
1130        </para>
1131       
1132        <para>
1133          The <classname docapi="net.sf.basedb.core.data">FileTypeData</classname> class holds information about
1134          file types. It is used only to make it easier for users to organise
1135          their files.
1136        </para>
1137       
1138        <para>
1139          The <classname docapi="net.sf.basedb.core.data">MimeTypeData</classname> is used to register mime types and
1140          map them to file extensions. The information is only used to lookup values
1141          when needed. Given the filename we can set the <varname>File.mimeType</varname>
1142          and <varname>File.fileType</varname> properties. The MIME type is also
1143          used to decide if a file should be stored in a compressed format or not.
1144          The extension of a MIME type must be unique. Extensions should be registered
1145          without a dot, ie <emphasis>html</emphasis>, not <emphasis>.html</emphasis>
1146        </para> 
1147     
1148    </sect2>
1149   
1150    <sect2 id="data_api.platforms">
1151      <title>Experimental platforms and item subtypes</title>
1152
1153      <para>
1154         This section gives an overview of experimental platforms
1155         and how they are used to enable data storage in files
1156         instead of in the database. In some senses item subtypes
1157         are related to platforms so they are also included here.
1158      </para>
1159     
1160      <itemizedlist>
1161        <title>See also</title>
1162        <listitem><xref linkend="core_api.data_in_files" /></listitem>
1163        <listitem><xref linkend="appendix.rawdatatypes.platforms" /></listitem>
1164        <listitem><xref linkend="extensions_developer.fileset_validator" /></listitem>
1165      </itemizedlist>
1166         
1167        <figure id="data_api.figures.platforms">
1168          <title>Experimental platforms and item subtypes</title>
1169          <screenshot>
1170            <mediaobject>
1171              <imageobject>
1172                <imagedata 
1173                  align="center"
1174                  fileref="figures/uml/datalayer.platforms.png" format="PNG" />
1175              </imageobject>
1176            </mediaobject>
1177          </screenshot>
1178        </figure>
1179
1180      <sect3 id="data_api.platforms.platforms">
1181        <title>Platforms</title>
1182       
1183        <para>
1184          The <classname docapi="net.sf.basedb.core.data">PlatformData</classname> holds information about a
1185          platform. A platform can have one or more <classname docapi="net.sf.basedb.core.data">PlatformVariant</classname>:s.
1186          Both the platform and variant are identified by an external ID that
1187          is fixed and can't be changed. <emphasis>Affymetrix</emphasis>
1188          is an example of a platform.
1189          If the <varname>fileOnly</varname> flag is set data for the platform
1190          can only be stored in files and not imported into the database. If
1191          the flag is not set data can be imported into the database.
1192          In the latter case, the <varname>rawDataType</varname> property
1193          can be used to lock the platform
1194          to a specific raw data type. If the value is <constant>null</constant>
1195          the platform can use any raw data type.
1196        </para>
1197       
1198        <para>
1199          Each platform and it's variant can be connected to one or more
1200          <classname docapi="net.sf.basedb.core.data">DataFileTypeData</classname> items. This item
1201          describes the kind of files that are used to hold data for
1202          the platform and/or variant. The file types are re-usable between
1203          different platforms and variants. Note that a file type may be attached
1204          to either only a platform or to a platform with a variant. File
1205          types attached to platforms are inherited by the variants. The variants
1206          can only define additional file types, not remove or redefine file types
1207          that has been attached to the platform.
1208        </para>
1209        <para>
1210          The file type is also identified
1211          by a fixed, non-changable external ID. The <varname>itemType</varname>
1212          property tells us what type of item the file holds data for (ie.
1213          array design or raw bioassay). It also links to a <classname 
1214          docapi="net.sf.basedb.core.data">ItemSubtype</classname>
1215          which is the generic type of data in the file. This allows us to query
1216          the database for, as an example, files with the generic type
1217          <constant>FileType.RAW_DATA</constant>. If we are in an Affymetrix
1218          experiment we will get the CEL file, for another platform we will
1219          get another file.
1220        </para>
1221        <para>
1222          The <varname>required</varname> flag in <classname docapi="net.sf.basedb.core.data">PlatformFileTypeData</classname>
1223          is used to signal that the file is a required file. This is not
1224          enforced by the core. It is intended to be used by client applications
1225          for creating a better GUI and for validation of an experiment.
1226        </para>
1227        <para>
1228          The <varname>allowMultiple</varname> flag in <classname 
1229          docapi="net.sf.basedb.core.data">PlatformFileTypeData</classname>
1230          controls if it should be possible to store more than one file of
1231          the given type in file type. Again, this is not enforced by the core,
1232          but only a recommendation to client applications. The setting is
1233          also used for validation of an experiment.
1234        </para>
1235
1236      </sect3>
1237     
1238      <sect3 id="data_api.platforms.subtypes">
1239        <title>Item subtypes</title>
1240       
1241        <para>
1242          The <classname docapi="net.sf.basedb.core.data">ItemSubtypeData</classname> 
1243          class describes a subtype for a main <varname>itemType</varname>. In the simplest
1244          form the subtype is a kind of annotation that is used mainly for creating a
1245          better user experience. If the main item type is also implementing the
1246          <interfacename docapi="net.sf.basedb.core.data">FileStoreEnabledData</interfacename> 
1247          interface, it is possible to
1248          register associations to the file types that can be used together with a given
1249          item subtype. The <varname>required</varname> and <varname>allowMultiple</varname>
1250          have are used in the same way as in the <classname>PlatformFileTypeData</classname>
1251          class.
1252        </para>
1253       
1254        <para>
1255          A subtype can be related to other subtypes. This is used to "chain" together
1256          groups of item subtypes. For example, <constant>Hybridization</constant>
1257          is a subtype for <constant>PHYSICALBIOASSAY</constant>, which is related to
1258          the <constant>Labeled extract (EXTRACT)</constant> subtype which is related to
1259          the <constant>Label (TAG)</constant> subtype. In addition, there are also
1260          several protocol and hardware subetypes mixed into this. The relationship between
1261          subtypes makes it possible for client applications to filter out unrelated stuff,
1262          and to validate experiments.
1263        </para>
1264       
1265      </sect3>
1266     
1267      <sect3 id="data_api.platforms.files">
1268        <title>FileStoreEnabled items and data files</title>
1269       
1270        <para>
1271          An item must implement the <interfacename docapi="net.sf.basedb.core.data">FileStoreEnabledData</interfacename>
1272          interface to be able to store data in files instead of in the database.
1273          The interface creates a link to a <classname docapi="net.sf.basedb.core.data">FileSetData</classname> object,
1274          which can hold several <classname docapi="net.sf.basedb.core.data">FileSetMemberData</classname> items.
1275          Each member points to specific <classname docapi="net.sf.basedb.core.data">FileData</classname> item.
1276        </para>
1277       
1278      </sect3>
1279    </sect2>
1280
1281    <sect2 id="data_api.parameters">
1282      <title>Parameters</title>
1283     
1284      <para>
1285        This section gives an overview the generic parameter
1286        system in BASE that is used to store annotation values,
1287        plugin configuration values, job parameter values, etc.
1288      </para>
1289     
1290        <figure id="data_api.figures.parameters">
1291          <title>Parameters</title>
1292          <screenshot>
1293            <mediaobject>
1294              <imageobject>
1295                <imagedata 
1296                  align="center"
1297                  fileref="figures/uml/datalayer.parameters.png" format="PNG" />
1298              </imageobject>
1299            </mediaobject>
1300          </screenshot>
1301        </figure>
1302     
1303        <para>
1304          The parameter system is a generic system that can store almost
1305          any kind of simple values (string, numbers, dates, etc.) and
1306          also links to other items. It is, for example, used to store configuration
1307          parameters to plug-ins and jobs as well as annotation values to annotatable items.
1308          The <classname docapi="net.sf.basedb.core.data">ParameterValueData</classname> 
1309          class is an abstract base class that can hold multiple values (all must be of the
1310          same type). Unless only a specific type of values should be stored, this is
1311          the class that should be used when creating references for storing parameter
1312          values. It makes it possible for a single relation to use any kind of
1313          values or for a collection reference to mix multiple types of values.
1314          A typical use case maps a <classname>Map</classname> with the
1315          parameter name as the key:
1316        </para>
1317       
1318        <programlisting language="java">
1319private Map&lt;String, ParameterValueData&lt;?&gt;&gt; configurationValues;
1320/**
1321   Link parameter name with it's values.
1322   @hibernate.map table="`PluginConfigurationValues`" lazy="true" cascade="all"
1323   @hibernate.collection-key column="`pluginconfiguration_id`"
1324   @hibernate.collection-index column="`name`" type="string" length="255"
1325   @hibernate.collection-many-to-many column="`value_id`"
1326      class="net.sf.basedb.core.data.ParameterValueData"
1327*/
1328public Map&lt;String, ParameterValueData&lt;?&gt;&gt; getConfigurationValues()
1329{
1330   return configurationValues;
1331}
1332void setConfigurationValues(Map&lt;String, ParameterValueData&lt;?&gt;&gt; configurationValues)
1333{
1334   this.configurationValues = configurationValues;
1335}
1336</programlisting>
1337       
1338      <para>
1339      Now it is possible for the collection to store all types of values:
1340      </para>
1341     
1342      <programlisting language="java">
1343Map&lt;String, ParameterValueData&lt;?&gt;&gt; config = ...
1344config.put("names", new StringParameterValueData("A", "B", "C"));
1345config.put("sizes", new IntegerParameterValueData(10, 20, 30));
1346
1347// When you later load those values again you have to cast
1348// them to the correct class.
1349List&lt;String&gt; names = (List&lt;String&gt;)config.get("names").getValues();
1350List&lt;Integer&gt; sizes = (List&lt;Integer&gt;)config.get("sizes").getValues();
1351</programlisting>
1352
1353    </sect2>
1354
1355    <sect2 id="data_api.annotations">
1356      <title>Annotations</title>
1357     
1358      <para>
1359        This section gives an overview of how the BASE annotation
1360        system works.
1361      </para>
1362     
1363        <figure id="data_api.figures.annotations">
1364          <title>Annotations</title>
1365          <screenshot>
1366            <mediaobject>
1367              <imageobject>
1368                <imagedata 
1369                  align="center"
1370                  fileref="figures/uml/datalayer.annotations.png" format="PNG" />
1371              </imageobject>
1372            </mediaobject>
1373          </screenshot>
1374        </figure>
1375     
1376      <sect3 id="data_api.annotations.description">
1377        <title>Annotations</title>
1378       
1379        <para>
1380        An item must implement the <interfacename docapi="net.sf.basedb.core.data">AnnotatableData</interfacename>
1381        interface to be able to use the annotation system. This interface gives
1382        a link to a <classname docapi="net.sf.basedb.core.data">AnnotationSetData</classname> item. This class
1383        encapsulates all annotations for the item. There are two types of
1384        annotations:
1385        </para>
1386       
1387        <itemizedlist>
1388        <listitem>
1389          <para>
1390          <emphasis>Primary annotations</emphasis> are annotations that
1391          explicitely belong to the item. An annotation set can contain
1392          only one primary annotation of each annotation type. The primary
1393          annotation are linked with the <property>annotations</property>
1394          property. This property is a map with an
1395          <classname docapi="net.sf.basedb.core.data">AnnotationTypeData</classname>  as the key.
1396          </para>
1397        </listitem>
1398       
1399        <listitem>
1400          <para>
1401          <emphasis>Inherited annotations</emphasis> are annotations
1402          that belong to a parent item, but that we want to use on
1403          another item as well. Inherited annotations are saved as
1404          references to either a single annotation or to another
1405          annotation set. Thus, it is possible for an item to inherit
1406          multiple annotations of the same annotation type.
1407          </para>
1408        </listitem>
1409        </itemizedlist>
1410       
1411        <para>
1412          The <classname docapi="net.sf.basedb.core.data">AnnotationData</classname> class is also
1413          just a placeholder. It connects the annotation set and
1414          annotation type with a <classname docapi="net.sf.basedb.core.data">ParameterValueData</classname>
1415          object. This is the object that holds the actual annotation
1416          values.
1417        </para>
1418       
1419      </sect3>
1420     
1421      <sect3 id="data_api.annotations.types">
1422        <title>Annotation types</title>
1423       
1424        <para>
1425        Instances of the <classname docapi="net.sf.basedb.core.data">AnnotationTypeData</classname> class
1426        defines the various annotations. It must have a <property>valueType</property> 
1427        property which cannot be changed. The value of this property controls
1428        which <classname docapi="net.sf.basedb.core.data">ParameterValueData</classname> subclass is used to store
1429        the annotation values, ie. <classname docapi="net.sf.basedb.core.data">IntegerParameterValueData</classname>,
1430        <classname docapi="net.sf.basedb.core.data">StringParameterValueData</classname>, etc.
1431        The <property>multiplicity</property> property holds the maximum allowed
1432        number of values for an annotation, or 0 if an unlimited number is
1433        allowed.
1434        </para>
1435       
1436        <para>
1437        The <property>itemTypes</property> collection holds the codes for
1438        the types of items the annotation type can be used on. This is
1439        checked when new annotations are created but already existing
1440        annotations are not affected if the collection is modified.
1441        </para>
1442       
1443        <para>
1444        Annotation types with the <property>protocolParameter</property> flag set
1445        are treated a bit differently. They will not show up as annotations
1446        to items with a type found in the <property>itemTypes</property> collection.
1447        Instead, a protocol parameter should be attached to a protocol. Then, when an item
1448        is using that protocol it becomes possible to add annotation values for
1449        the annotation types specified as protocol parameters. It doesn't matter
1450        if the item's type is found in the <property>itemTypes</property> 
1451        collection or not.
1452        </para>
1453       
1454        <para>
1455        The <property>options</property> collection is used to store additional
1456        options required by some of the value types, for example a max string
1457        length for string annotations or the max and min allowed value for
1458        integer annotations.
1459        </para>
1460       
1461        <para>
1462        The <property>enumeration</property> property is a boolean flag
1463        indicating if the allowed values are predefined as an enumeration.
1464        In that case those values are found in the <property>enumerationValues</property>
1465        property. The actual subclass is determined by the <property>valueType</property>
1466        property.
1467        </para>
1468       
1469        <para>
1470        Most of the other properties are hints to client applications how
1471        to render the input field for the annotation.
1472        </para>
1473       
1474      </sect3>
1475     
1476      <sect3 id="data_api.annotations.units">
1477        <title>Units</title>
1478        <para>
1479        Numerical annotation values can have units. A unit is described by
1480        a <classname docapi="net.sf.basedb.core.data">UnitData</classname> object.
1481        Each unit belongs to a <classname docapi="net.sf.basedb.core.data">QuantityData</classname> 
1482        object which defines the class of units. For example, if the quantity is
1483        <emphasis>weight</emphasis>, we can have units, <emphasis>kg</emphasis>,
1484        <emphasis>mg</emphasis>, <emphasis>µg</emphasis>, etc. The <classname>UnitData</classname>
1485        contains a factor and offset that relates all units to a common reference
1486        defined by the <classname>QuantityData</classname> class. For example,
1487        <emphasis>1 meter</emphasis> is the reference unit for distance, and we
1488        have <code>1 meter * 0.001 = 1 millimeter</code>. In this case, the factor is
1489        <emphasis>0.001</emphasis> and the offset 0. Another example is the relationship between
1490        kelvin and Celsius, which is <code>1 kelvin + 273.15 = 1 °Celsius</code>.
1491        Here, the factor is 1 and the offset is <emphasis>+273.15</emphasis>.
1492        The <classname
1493        docapi="net.sf.basedb.core.data">UnitSymbolData</classname>
1494        is used to make it possible to assign alternative symbols to a single unit.
1495        This is needed to simplify input where it may be hard to know what to
1496        type to get <emphasis></emphasis> or <emphasis>°C</emphasis>. Instead,
1497        <emphasis>m2</emphasis> and <emphasis>C</emphasis> can be used as
1498        alternative symbols.
1499        </para>
1500       
1501        <para>
1502        The creator of an annotation type may select a
1503        <classname>QuantityData</classname>, which can't be changed later, and
1504        a default <classname>UnitData</classname>. When entering annotation values
1505        a user may select any unit for the selected quantity (unless annotation type
1506        owner has limited this by selecting <varname>usableUnits</varname>). Before
1507        the values are stored in the database, they are converted to the default
1508        unit. This makes it possible to compare and filter on annotation values
1509        using different units. For example, filtering with <emphasis>&gt;5mg</emphasis> 
1510        also finds items that are annotated with <emphasis>2g</emphasis>.
1511        </para>
1512       
1513        <para>
1514        The core should automatically update the stored annotation values if
1515        the default unit is changed for an annotation type, or if the reference
1516        factor for a unit is changed.
1517        </para>
1518      </sect3>
1519     
1520      <sect3 id="data_api.annotations.categories">
1521        <title>Categories</title>
1522       
1523        <para>
1524        The <classname docapi="net.sf.basedb.core.data">AnnotationTypeCategoryData</classname> class defines
1525        categories that are used to group annotation types that are related to
1526        each other. This information is mainly useful for client applications
1527        when displaying forms for annotating items, that wish to provide a
1528        clearer interface when there are many (say 50+) annotations type for
1529        an item. An annotation type can belong to more than one category.
1530        </para>
1531       
1532      </sect3>
1533     
1534    </sect2>
1535
1536    <sect2 id="data_api.protocols">
1537      <title>Protocols, hardware and software</title>
1538
1539      <para>
1540        This section gives an overview of how protocols that describe various
1541        processes, such as sampling, extraction and scanning, are used in BASE.
1542      </para>
1543     
1544        <figure id="data_api.figures.protocols">
1545          <title>Protocols, hardware and software</title>
1546          <screenshot>
1547            <mediaobject>
1548              <imageobject>
1549                <imagedata 
1550                  align="center"
1551                  fileref="figures/uml/datalayer.protocols.png" format="PNG" />
1552              </imageobject>
1553            </mediaobject>
1554          </screenshot>
1555        </figure>
1556     
1557      <sect3 id="data_api.protocols.description">
1558        <title>Protocols</title>
1559       
1560        <para>
1561        A protocol is something that defines a procedure or recipe for some
1562        kind of action, such as sampling, extraction and scanning. The subtype
1563        of the protocol is used to determine what the protocol is used for.
1564        In BASE we only store a short name and description. It is possible to
1565        attach a file that provides a longer description of the procedure.
1566        </para>
1567     
1568      </sect3>
1569     
1570      <sect3 id="data_api.protocols.parameters">
1571        <title>Parameters</title>
1572       
1573        <para>
1574        The procedure described by the protocol may have parameters
1575        that are set indepentently each time the protocol is used. It
1576        could for example be a temperature, a time or something else.
1577        The definition of parameters is done by creating annotation
1578        types and attaching them to the protocol. It is only possible
1579        to attach annotation types which has the <property>protocolParameter</property>
1580        property set to <constant>true</constant>. The same annotation type
1581        can be used for more than one protocol, but only do this if the
1582        parameters actually has the same meaning.
1583        </para>
1584     
1585      </sect3>
1586
1587      <sect3 id="data_api.wares.description">
1588        <title>Hardware and software</title>
1589        <para>
1590          BASE is pre-installed with a set of subtypes for hardware and software.
1591          They are typically used to filter the registered hardware and software
1592          depending on what a user is doing. For example, when adding raw data
1593          to BASE a user can select a scanner. The GUI will display the hardware
1594          that has been registered as <emphasis>scanner</emphasis> subtype.
1595          Other subtypes are <emphasis>hybridization station</emphasis>
1596          and <emphasis>print robot</emphasis>. An administrator may register more
1597          subtypes.
1598        </para>
1599      </sect3>
1600     
1601    </sect2>
1602
1603    <sect2 id="data_api.plugins">
1604      <title>Plug-ins, jobs and job agents</title>
1605     
1606      <para>
1607         This section gives an overview of plug-ins, jobs and job agents.
1608      </para>
1609     
1610      <itemizedlist>
1611        <title>See also</title>
1612        <listitem><xref linkend="plugins.installation" /></listitem>
1613        <listitem><xref linkend="installation.jobagents" /></listitem>
1614      </itemizedlist>
1615     
1616        <figure id="data_api.figures.plugins">
1617          <title>Plug-ins, jobs and job agents</title>
1618          <screenshot>
1619            <mediaobject>
1620              <imageobject>
1621                <imagedata 
1622                  align="center"
1623                  scalefit="1" width="100%"
1624                  fileref="figures/uml/datalayer.plugins.png" format="PNG" />
1625              </imageobject>
1626            </mediaobject>
1627          </screenshot>
1628        </figure>
1629
1630      <sect3 id="data_api.plugins.plugins">
1631        <title>Plug-ins</title>
1632       
1633        <para>
1634          The <classname docapi="net.sf.basedb.core.data">PluginDefinitionData</classname> holds information of the
1635          installed plugin classes. Much of the information is copied from the
1636          plug-in itself from the <classname docapi="net.sf.basedb.core.plugin">About</classname> object and by checking
1637          which interfaces it implements.
1638        </para>
1639       
1640        <para>
1641          There are five main types of plug-ins:
1642        </para>
1643       
1644        <itemizedlist>
1645        <listitem>
1646          <para>
1647          IMPORT (mainType = 1): A plug-in that imports data to BASE.
1648          </para>
1649        </listitem>
1650        <listitem>
1651          <para>
1652          EXPORT (mainType = 2): A plug-in that exports data from BASE.
1653          </para>
1654        </listitem>
1655        <listitem>
1656          <para>
1657          INTENSITY (mainType = 3): A plug-in that calculates intensity values
1658          from raw data.
1659          </para>
1660        </listitem>
1661        <listitem>
1662          <para>
1663          ANALYZE (mainType = 4): A plug-in that analyses data.
1664          </para>
1665        </listitem>
1666        <listitem>
1667          <para>
1668          OTHER (mainType = 5): Any other plug-in.
1669          </para>
1670        </listitem>
1671        </itemizedlist>
1672       
1673        <para>
1674          A plug-in may have different configurations. The flags <property>supportsConfigurations</property>
1675          and <property>requiresConfiguration</property> are used to specify if a plug-in
1676          must have or can't have any configurations. Configuration parameter values are
1677          versioned. Each time anyone updates a configuration the version number
1678          is increased and the parameter values are stored as a new entity.
1679          This is required because we want to be able to know exactly which
1680          parameters a job were using when it was executed. When a job is
1681          created we also store the parameter version number
1682          (<property>JobData.parameterVersion</property>). This means that even if
1683          someone changes the configuration later we will always know which
1684          parameters the job used.
1685        </para>
1686       
1687        <para>
1688          The <classname docapi="net.sf.basedb.core.data">PluginTypeData</classname> class is ued to group
1689          plug-ins that share some common functionality, by implementing
1690          additional (optional) interfaces. For example, the
1691          <interfacename docapi="net.sf.basedb.core.plugin">AutoDetectingImporter</interfacename> should be implemented
1692          by import plug-ins that supports automatic detection of file formats.
1693          Another example is the <interfacename docapi="net.sf.basedb.core.plugin">AnalysisFilterPlugin</interfacename>
1694          interface which should be implemented by all analysis plug-ins that
1695          only filters data.
1696        </para>
1697
1698      </sect3>
1699     
1700      <sect3 id="data_api.plugins.jobs">
1701        <title>Jobs</title>
1702       
1703        <para>
1704          A job represents a single invokation of a plug-in to do some work.
1705          The <classname docapi="net.sf.basedb.core.data">JobData</classname> class holds information about this.
1706          A job is usuallu executed by a plug-in, but doesn't have to be. The
1707          <property>status</property> property holds the current state of a job.
1708        </para>
1709       
1710        <itemizedlist>
1711        <listitem>
1712          <para>
1713            UNCONFIGURED (status = 0): The job is not yet ready to be executed.
1714          </para>
1715        </listitem>
1716        <listitem>
1717          <para>
1718            WAITING (status = 1): The job is waiting to be executed.
1719          </para>
1720        </listitem>
1721        <listitem>
1722          <para>
1723            PREPARING (status = 5): The job is about to be executed but hasn't started yet.
1724          </para>
1725        </listitem>
1726        <listitem>
1727          <para>
1728            EXECUTING (status = 2): The job is currently executing.
1729          </para>
1730        </listitem>
1731        <listitem>
1732          <para>
1733            ABORTING (status = 6): The job is executing but an ABORT signal has been sent
1734            requesting it to abort and finish.
1735          </para>
1736        </listitem>
1737        <listitem>
1738          <para>
1739            DONE (status = 3): The job finished successfully.
1740          </para>
1741        </listitem>
1742        <listitem>
1743          <para>
1744            ERROR (status = 4): The job finished with an error.
1745          </para>
1746        </listitem>
1747        </itemizedlist>
1748      </sect3>
1749
1750      <sect3 id="data_api.plugins.agents">
1751        <title>Job agents</title>
1752       
1753        <para>
1754          A job agent is a program running on the same or a different server that
1755          is regularly checking for jobs that are waiting to be executed. The
1756          <classname docapi="net.sf.basedb.core.data">JobAgentData</classname> holds information about a job agent
1757          and the <classname docapi="net.sf.basedb.core.data">JobAgentSettingsData</classname> links the agent
1758          with the plug-ins the agent is able to execute. The job agent will only
1759          execute jobs that are owned by users or projects that the job agent has
1760          been shared to with at least use permission. The <property>priorityBoost</property>
1761          property can be used to give specific plug-ins higher priority.
1762          Thus, for a job agent it is possible to:
1763        </para>
1764       
1765        <itemizedlist>
1766        <listitem>
1767          <para>
1768          Specify exactly which plug-ins it will execute. For example, it is possible
1769          to dedicate one agent to only run one plug-in.
1770          </para>
1771        </listitem>
1772        <listitem>
1773          <para>
1774          Give some plug-ins higher priority. For example a job agent that is mainly
1775          used for importing data should give higher priority to all import plug-ins.
1776          Other types of jobs will have to wait until there are no more data to be
1777          imported.
1778          </para>
1779        </listitem>
1780        <listitem>
1781          <para>
1782          Specify exactly which users/groups/projects that may use the agent. For
1783          example, it is possible to dedicate one agent to only run jobs for a certain
1784          project.
1785          </para>
1786        </listitem>
1787        </itemizedlist>
1788       
1789      </sect3>
1790
1791
1792    </sect2>
1793   
1794    <sect2 id="data_api.biomaterials">
1795      <title>Biomaterial LIMS</title>
1796     
1797        <figure id="data_api.figures.biomaterials">
1798          <title>Biomaterial LIMS</title>
1799          <screenshot>
1800            <mediaobject>
1801              <imageobject>
1802                <imagedata 
1803                  align="center"
1804                  fileref="figures/uml/datalayer.biomaterials.png" format="PNG" />
1805              </imageobject>
1806            </mediaobject>
1807          </screenshot>
1808        </figure>
1809     
1810      <sect3 id="data_api.biomaterials.description">
1811        <title>Biomaterials</title>
1812       
1813        <para>
1814          There are three main types of biomaterials: <classname docapi="net.sf.basedb.core.data">BioSourceData</classname>,
1815          <classname docapi="net.sf.basedb.core.data">SampleData</classname> and
1816          <classname docapi="net.sf.basedb.core.data">ExtractData</classname>.
1817          All types of are derived from the base class <classname docapi="net.sf.basedb.core.data">BioMaterialData</classname>.
1818          The reason for this is that they all share common functionality such as pooling
1819          and events. By using a common base class we do not have to create duplicate
1820          classes for keeping track of events and parents.
1821        </para>
1822       
1823        <para>
1824          The <classname docapi="net.sf.basedb.core.data">BioSourceData</classname> is the simplest of the biomaterials.
1825          It cannot have parents and can't participate in events. It's only used as a
1826          (non-required) parent for samples.
1827        </para>
1828       
1829        <para>
1830          The <classname docapi="net.sf.basedb.core.data">MeasuredBioMaterialData</classname> class is used as a base
1831          class for the other biomaterial types. It introduces quantity
1832          measurements and can store original and remaining quantities. They are
1833          both optional. If an original quantity has been specified the core
1834          automatically calculates the remaining quantity based on the events a
1835          biomaterial participates in.
1836        </para>
1837       
1838        <para>
1839          All measured biomaterial have at least one event associated with them,
1840          the <emphasis>creation event</emphasis>, which holds information about the creation of the
1841          biomaterial. A measured biomaterial can be created in three ways:
1842        </para>
1843       
1844        <itemizedlist>
1845        <listitem>
1846          <para>
1847          From a single item of the same type or the parent type. Biosource is the parent type of
1848          samples and sample is the parent type of extracts. The <property>parentType</property> 
1849          property must be set to the correct parent type and the <property>parent</property> property
1850          is set to point to the parent item. The parent information
1851          is also always duplicated in the <property>sources</property> collection of the <classname docapi="net.sf.basedb.core.data">BioMaterialEventData</classname>
1852          object representing the creation event. It is the responsibility of the
1853          core to make sure that everything is properly synchronized and that
1854          remaining quantities are calculated.
1855          </para>
1856        </listitem>
1857       
1858        <listitem>
1859          <para>
1860          From multiple items of the same type, i.e pooling.
1861          In this case the <property>parentType</property> property is set, but
1862          the <property>parent</property> property is null. All source
1863          biomaterials are contained in the <property>sources</property> collection.
1864          The core is still responsible for keeping everything synchronized and to
1865          update remaining quantities.
1866          </para>
1867        </listitem>
1868       
1869        <listitem>
1870          <para>
1871          As a standalone biomaterial without parents. The <property>parentType</property>
1872          property should be null, as should the <property>parent</property> property
1873          and the <property>sources</property> collection.
1874          </para>
1875        </listitem>
1876        </itemizedlist>
1877
1878      </sect3>
1879     
1880      <sect3 id="data_api.biomaterials.plates">
1881        <title>Bioplates and plate types</title>
1882       
1883        <para>
1884          Biomaterial (except biosource) may optionally be placed on <classname 
1885          docapi="net.sf.basedb.core.data">BioPlateData</classname>:s. A bioplate is something
1886          that collects multiple biomaterial as a unit. A bioplate typically has a
1887          <classname docapi="net.sf.basedb.core.data">PlateGeometryData</classname> that
1888          determines the number of locations on the plate (<classname docapi="net.sf.basedb.core.data">BioWellData</classname>).
1889          A single well can hold a single biomaterial at a time.
1890        </para>
1891       
1892        <para>
1893          The bioplate must be of a specific <classname docapi="net.sf.basedb.core.data">BioPlateTypeData</classname>.
1894          The type can be used to put limitations on how the plate can be used. For example,
1895          it can be limited to a single type of biomaterial. It is also possible to lock wells
1896          so that the biomaterial in them can't be changed. Supported lock modes are:
1897        </para>
1898       
1899        <itemizedlist>
1900        <listitem>
1901          <para>
1902          <emphasis>Unlocked</emphasis>: Wells are unlocked and the biomaterial may be changed
1903          any number of times.
1904          </para>
1905        </listitem>
1906        <listitem>
1907          <para>
1908          <emphasis>Locked-after-move</emphasis>: The well is locked after it has been used one
1909          time and the biomaterial that was put in it has been moved to another plate.
1910          </para>
1911        </listitem>
1912        <listitem>
1913          <para>
1914          <emphasis>Locked-after-add</emphasis>: The well is locked after biomaterial has been
1915          put into it. It is not possible to remove the biomaterial.
1916          </para>
1917        </listitem>
1918        <listitem>
1919          <para>
1920          <emphasis>Locked-after-create</emphasis>: The well is locked once it has been created.
1921          Biomaterial must be put into wells before the plate is saved to the database.
1922          </para>
1923        </listitem>
1924        </itemizedlist>
1925       
1926      </sect3>
1927 
1928      <sect3 id="data_api.biomaterials.events">
1929        <title>Biomaterial and plate events</title>
1930       
1931        <para>
1932          An event represents something that happened to one or more biomaterials, for example
1933          the creation of another biomaterial. The <classname docapi="net.sf.basedb.core.data">BioMaterialEventData</classname>
1934          holds information about entry and event dates, protocols used, the user who is
1935          responsible, etc. There are three types of events represented by the <property>eventType</property>
1936          property.
1937        </para>
1938       
1939        <orderedlist>
1940        <listitem>
1941          <para>
1942          <emphasis>Creation event</emphasis>: This event represents the creation of a (measured)
1943          biomaterial. The <property>sources</property> collection contains
1944          information about the biomaterials that were used to create the new
1945          biomaterial. All sources must be of the same type. There can only be one
1946          source of the parent type. These rules are maintained by the core.
1947          </para>
1948        </listitem>
1949       
1950        <listitem>
1951          <para>
1952          <emphasis>Bioassay event</emphasis>: This event represents the creation
1953          of a bioassay. This event type is needed because we want to keep track
1954          of quantities for extracts. This event has a <classname docapi="net.sf.basedb.core.data">PhysicalBioAssayData</classname> 
1955          as a product instead of a biomaterial. The sources collection can only contain
1956          extracts. If the bioassay can hold extracts in multiple positions the
1957          <property>position</property> property in <classname docapi="net.sf.basedb.core.data">BioMaterialEventSourceData</classname> 
1958          can be used to track which extract that was put in each position. It is allowed
1959          to put multiple extracts in the same position, but then the usually need
1960          to use different <classname docapi="net.sf.basedb.core.data">TagData</classname> 
1961          items. However, this is not enforced by the core.
1962          </para>
1963        </listitem>
1964
1965        <listitem>
1966          <para>
1967          <emphasis>Other event</emphasis>: This event represents some other important
1968          information about a single biomaterial that affected the remaining quantity.
1969          This event type doesn't have any sources.
1970          </para>
1971        </listitem>
1972        </orderedlist>
1973       
1974        <para>
1975          It is also possible to register events that applies to one or more
1976          bioplates using the <classname docapi="net.sf.basedb.core.data">BioPlateEventData</classname>
1977          class. The <classname docapi="net.sf.basedb.core.data">BioPlateEventParticipantData</classname>
1978          class holds information about each plate that is part of the event. The <property>role</property>
1979          property is a textual description of what happened to the plate. Eg. a move event, may have one
1980          <emphasis>source</emphasis> plate and one <emphasis>destination</emphasis> plate. It is
1981          recommended (but not required) that all biomaterial that are affected by the plate event
1982          are linked via a <code>BioMaterialEventData</code> to a <code>BioPlateEventParticipantData</code>.
1983          This will make it easier to keep track of the history of individual biomaterial items.
1984          Biomaterial events that are linked in this way are also automatically updated if the
1985          bioplate event is modified (eg. selecting a protocol, event date, etc.).
1986        </para>
1987       
1988      </sect3> 
1989    </sect2>
1990
1991    <sect2 id="data_api.plates">
1992      <title>Array LIMS - plates</title>
1993
1994        <figure id="data_api.figures.plates">
1995          <title>Array LIMS - plates</title>
1996          <screenshot>
1997            <mediaobject>
1998              <imageobject>
1999                <imagedata 
2000                  align="center"
2001                  scalefit="1" width="100%"
2002                  fileref="figures/uml/datalayer.plates.png" format="PNG" />
2003              </imageobject>
2004            </mediaobject>
2005          </screenshot>
2006        </figure>
2007
2008      <sect3 id="data_api.plates.description">
2009        <title>Plates</title>
2010       
2011        <para>
2012          The <classname docapi="net.sf.basedb.core.data">PlateData</classname> is the main class holding information
2013          about a single plate. The associated <classname docapi="net.sf.basedb.core.data">PlateGeometryData</classname>
2014          defines how many rows and columns there are on a plate. Since this
2015          information is used to create wells, and for various other checks it is
2016          not possible to change the number of rows or columns once a geometry has
2017          been created.
2018        </para>
2019         
2020        <para>
2021          All plates must have a <classname docapi="net.sf.basedb.core.data">PlateTypeData</classname> which defines
2022          the geometry and a set of event types (see below).
2023        </para>
2024       
2025        <para>
2026          If the destroyed flag of a plate is set it is not allowed to use the
2027          plate for a plate mapping or to create array designs. However, it
2028          is possible to change the flag to not destroyed.
2029        </para>
2030
2031        <para>
2032          The barcode is intended to be used as an external identifier of the plate.
2033          But, the core doesn't care about the value or if it is unique or not.
2034        </para>
2035      </sect3>
2036     
2037      <sect3 id="data_api.plates.events">
2038        <title>Plate events</title>
2039
2040        <para>
2041          The plate type defines a set of <classname docapi="net.sf.basedb.core.data">PlateEventTypeData</classname>
2042          objects, each one represening a particular event a plate of this type
2043          usually goes trough. For a plate of a certain type, it is possible to
2044          attach exactly one event of each event type. The event type defines an
2045          optional protocol type, which can be used by client applications to
2046          filter a list of protocols for the event. The core doesn't check that
2047          the selected protocol for an event is of the same protocol type as
2048          defined by the event type.
2049        </para>
2050
2051        <para>
2052          The ordinal value can be used as a hint to client applications in
2053          which order the events actually are performed in the lab. The core doesn't
2054          care about this value or if several event types have the same value.
2055        </para>
2056      </sect3>
2057
2058      <sect3 id="data_api.plates.mappings">
2059        <title>Plate mappings</title>
2060       
2061        <para>
2062          A plate can be created either from scratch, with the help of the information
2063          in a <classname docapi="net.sf.basedb.core.data">PlateMappingData</classname>, from a set of parent plates.
2064          In the first case it is possible to specify a reporter for each well on the
2065          plate. In the second case the mapping code creates all the wells and links
2066          them to the parent wells on the parent plates. Once the plate has been saved
2067          to the database, the wells cannot be modified (because they are used
2068          downstream for various validation, etc.)
2069        </para>
2070       
2071        <para>
2072          The details in a plate mapping are simply coordinates that for each
2073          destination plate, row and column define a source plate, row and column.
2074          It is possible for a single source well to be mapped to multiple destination
2075          wells, but for each destination well only a single source well can be
2076          used.
2077        </para>
2078       
2079      </sect3>
2080
2081    </sect2>
2082
2083    <sect2 id="data_api.arrays">
2084      <title>Array LIMS - arrays</title>
2085     
2086        <figure id="data_api.figures.arrays">
2087          <title>Array LIMS - arrays</title>
2088          <screenshot>
2089            <mediaobject>
2090              <imageobject>
2091                <imagedata 
2092                  align="center"
2093                  fileref="figures/uml/datalayer.arrays.png" format="PNG" />
2094              </imageobject>
2095            </mediaobject>
2096          </screenshot>
2097        </figure>
2098     
2099      <sect3 id="data_api.arrays.designs">
2100        <title>Array designs</title>
2101       
2102        <para>
2103          Array designs are stored in <classname docapi="net.sf.basedb.core.data">ArrayDesignData</classname> objects
2104          and can be created either as standalone designs or
2105          from plates. In the first case the features on an array design
2106          are described by a reporter map. A reporter map is a file
2107          that maps a coordinate (block, meta-grid, row, column),
2108          position or an external ID on an array design to a
2109          reporter. Which method to use is given by the
2110          <property>ArrayDesign.featureIdentificationMethod</property> property.
2111          The coordinate system on an array design is divided into blocks.
2112          Each block can be identified either by a <property>blockNumber</property>
2113          or by meta coordinates. This information is stored in
2114          <classname docapi="net.sf.basedb.core.data">ArrayDesignBlockData</classname> items. Each block
2115          contains several <classname docapi="net.sf.basedb.core.data">FeatureData</classname> items, each
2116          one identified by a row and column coordinate. Platforms that doesn't
2117          divide the array design into blocks or doesn't use the coordinate system at all
2118          must still create a single super-block that holds all features.
2119        </para>
2120       
2121        <para>
2122          Array designs that are created from plates use a print map file
2123          instead of a reporter map. A print map is similar to a plate mapping
2124          but maps features (instead of wells) to wells. The file should
2125          specifify which plate and well a feature is created from. Reporter
2126          information will automatically be copied by BASE from the well.
2127        </para>
2128       
2129        <para>
2130          It is also possible to skip the importing of features into the
2131          database and just keep the data in the orginal files instead.
2132          This is typically done for Affymetrix CDF files.
2133        </para>
2134       
2135      </sect3>
2136     
2137      <sect3 id="data_api.arrays.slides">
2138        <title>Array slides</title>
2139       
2140        <para>
2141          The <classname docapi="net.sf.basedb.core.data">ArraySlideData</classname> represents a single
2142          array. Arrays are usually printed several hundreds in a batch,
2143          represented by a <classname docapi="net.sf.basedb.core.data">ArrayBatchData</classname> item.
2144          The <property>batchIndex</property> is the ordinal number of the
2145          array in the batch. The <property>barcode</property> can be used
2146          as a means for external programs to identify the array. BASE doesn't
2147          care if a value is given or if they are unique or not. If the
2148          <property>destroyed</property> flag is set it prevents a slide from
2149          beeing used by a hybridization.
2150        </para>
2151
2152      </sect3>
2153    </sect2>
2154
2155    <sect2 id="data_api.bioassays">
2156      <title>Bioassays and raw data</title>
2157     
2158        <figure id="data_api.figures.rawdata">
2159          <title>Bioassays and raw data</title>
2160          <screenshot>
2161            <mediaobject>
2162              <imageobject>
2163                <imagedata 
2164                  align="center"
2165                  scalefit="1" width="100%"
2166                  fileref="figures/uml/datalayer.bioassays.png" format="PNG" />
2167              </imageobject>
2168            </mediaobject>
2169          </screenshot>
2170        </figure>
2171     
2172      <sect3 id="data_api.bioassays.physical">
2173        <title>Physical bioassays</title>
2174       
2175        <para>
2176        A <classname docapi="net.sf.basedb.core.data">PhysicalBioAssayData</classname>
2177        item connect the array slides from the Array LIMS part
2178        with extracts from the biomaterials part. The <property>creationEvent</property>
2179        is used to register which extracts that were used on the bioassay.
2180        The relation to slides is a one-to-one relation. A slide can only be used on
2181        a single physical bioassay and a bioassay can only use a single slide. The relation
2182        is optional from both sides.
2183        </para>
2184
2185        <para>
2186        Further processing of the bioassay is registered as a series
2187        of <classname docapi="net.sf.basedb.core.data">DerivedBioAssayData</classname>
2188        items. For microarray experiments the first step is typically a scanning
2189        of the hybridization. Information about the software/hardware and protocol
2190        used can be registered. Any data files generated by the process can be
2191        registered with the <classname docapi="net.sf.basedb.core.data">FileSetData</classname>
2192        item. If more than one processsing step is required child derived
2193        bioassays can be created that descrive each additional step. 
2194        </para>
2195       
2196        <para>
2197        If the root physical bioassay has multiple extracts in multiple positions, the
2198        <property>extract</property> property of a derived bioassay is used to link
2199        with the extract that the specific derived bioassay represents. If the
2200        link is null the derived bioassay represents all extracts on the
2201        physical bioassay.
2202        </para>
2203       
2204      </sect3>
2205     
2206      <sect3 id="data_api.bioassays.rawdata">
2207        <title>Raw data</title>
2208       
2209        <para>
2210        A <classname docapi="net.sf.basedb.core.data">RawBioAssayData</classname> object
2211        represents the raw data that is produced by analysing the data from the physical
2212        bioassay. You may register which software that was used, the
2213        protocol and any parameters (through the annotation system).
2214        </para>
2215
2216        <para>
2217        Files with the analysed data values can be attached to the
2218        associated <classname docapi="net.sf.basedb.core.data">FileSetData</classname> object.
2219        The platform and, optionally, the variant has information about the file types
2220        that can be used for that platform. If the platform file types support
2221        metadata extraction, headers, the number of spots, and other
2222        information may be automatically extracted from the raw data file(s).
2223        </para>
2224       
2225        <para>
2226        If the platform support it, raw data can also be imported into the database.
2227        This is handled by batchers and <classname docapi="net.sf.basedb.core.data">RawData</classname> objects.
2228        Which table to store the data in depends on the <property>rawDataType</property>
2229        property. The properties shown for the <classname>RawData</classname> class
2230        in the diagram are the mandatory properties. Each raw data type defines additional
2231        properties that are specific to that raw data type.
2232        </para>
2233       
2234      </sect3>
2235       
2236    </sect2>
2237
2238    <sect2 id="data_api.experiments">
2239      <title>Experiments and analysis</title>
2240     
2241     
2242        <figure id="data_api.figures.experiments">
2243          <title>Experiments</title>
2244          <screenshot>
2245            <mediaobject>
2246              <imageobject>
2247                <imagedata 
2248                  align="center"
2249                  scalefit="1" width="75%"
2250                  fileref="figures/uml/datalayer.experiments.png" format="PNG" />
2251              </imageobject>
2252            </mediaobject>
2253          </screenshot>
2254        </figure>
2255     
2256      <sect3 id="data_api.experiments.description">
2257        <title>Experiments</title>
2258       
2259        <para>
2260          The <classname docapi="net.sf.basedb.core.data">ExperimentData</classname> 
2261          class is used to collect information about a single experiment. It
2262          links to any number of <classname docapi="net.sf.basedb.core.data">RawBioAssayData</classname>
2263          items, which must all be of the same <classname 
2264          docapi="net.sf.basedb.core">RawDataType</classname>.
2265        </para>
2266       
2267        <para>
2268          Annotation types that are needed in the analysis must connected to
2269          the experiment as experimental factors and the annotation values should
2270          be set on or inherited by each raw bioassay that is part of the
2271          experiment.
2272        </para>
2273       
2274        <para>
2275          The directory connected to the experiment is the default directory
2276          where plugins that generate files should store them.
2277        </para>
2278      </sect3>
2279           
2280      <sect3 id="data_api.experiments.bioassays">
2281        <title>Bioassay sets, bioassays and transformations</title>
2282       
2283        <para>
2284          Each line of analysis starts with the creation of a <emphasis>root</emphasis>
2285          <classname docapi="net.sf.basedb.core.data">BioAssaySetData</classname>,
2286          which holds the intensities calculated from the raw data.
2287          A bioassayset can hold one intensity for each channel. The number of
2288          channels is defined by the raw data type. For each raw bioassay used a
2289          <classname docapi="net.sf.basedb.core.data">BioAssayData</classname>
2290          is created.
2291        </para>
2292       
2293        <para>
2294          Information about the process that calculated the intensities are
2295          stored in a <classname docapi="net.sf.basedb.core.data">TransformationData</classname>
2296          object. The root transformation links with the raw bioassays that are used
2297          in this line of analysis and to a <classname 
2298          docapi="net.sf.basedb.core.data">JobData</classname> which has information
2299          about which plug-in and parameters that was used in the calculation.
2300        </para>
2301     
2302        <para>
2303          Once the root bioassayset has been created it is possible to
2304          again apply a transformation to it. This time the transformation
2305          links to a single source bioassayset instead of the raw bioassays.
2306          As before, it still links to a job with information about the plug-in and
2307          parameters that does the actual work. The transformation must make sure
2308          that new bioassays are created and linked to the bioassays in the
2309          source bioassayset. This above process may be repeated as many times
2310          as needed.
2311        </para>
2312       
2313        <para>
2314          Data to a bioassay set can only be added to it before it has been
2315          committed to the database. Once the transaction has been committed
2316          it is no longed possible to add more data or to modify existing
2317          data.
2318        </para>
2319     
2320      </sect3>
2321
2322      <sect3 id="data_api.experiments.virtualdb">
2323        <title>Virtual databases, datacubes, etc.</title>
2324       
2325        <para>
2326          The above processes requires a flexible storage solution for the data.
2327          Each experiment is related to a <classname docapi="net.sf.basedb.core.data">VirtualDb</classname>
2328          object. This object represents the set of tables that are needed to store
2329          data for the experiment. All tables are created in a special part of the
2330          BASE database that we call the <emphasis>dynamic database</emphasis>.
2331          In MySQL the dynamic database is a separate database, in Postgres it is
2332          a separate schema.
2333        </para>
2334       
2335        <para>
2336          A virual database is divided into data cubes. A data cube can be seen
2337          as a three-dimensional object where each point can hold data that in
2338          most cases can be interpreted as data for a single spot from an
2339          array. The coordinates to a point is given by <emphasis>layer</emphasis>,
2340          <emphasis>column</emphasis> and <emphasis>position</emphasis>. The
2341          layer and column coordinates are represented by the
2342          <classname docapi="net.sf.basedb.core.data">DataCubeLayerData</classname>
2343          and <classname docapi="net.sf.basedb.core.data">DataCubeColumnData</classname>
2344          objects. The position coordinate has no separate object associated with
2345          it.
2346        </para>
2347       
2348        <para>
2349          Data for a single bioassay set is always stored in a single layer. It
2350          is possible for more than one bioassay set to use the same layer. This
2351          usually happens for filtering transformations that doesn't modify the
2352          data.  The filtered bioassay set is then linked to a
2353          <classname docapi="net.sf.basedb.core.data">DataCubeFilterData</classname>
2354          object, which has information about which data points that
2355          passed the filter.
2356        </para>
2357       
2358        <para>
2359          All data for a bioassay is stored in a single column.
2360          Two bioassays in different bioassaysets (layers) can only have the same
2361          column if one is the parent of the other.
2362        </para>
2363       
2364        <para>
2365          The position coordinate is tied to a reporter.
2366        </para>
2367       
2368        <para>
2369          A child bioassay set may use the same data cube as it's parent
2370          bioassay set if all of the following conditions are true:
2371        </para>
2372       
2373        <itemizedlist>
2374        <listitem>
2375          <para>
2376          All positions are linked to the same reporter as the positions
2377          in the parent bioassay set.
2378          </para>
2379        </listitem>
2380       
2381        <listitem>
2382          <para>
2383          All data points are linked to the same (possible many) raw data
2384          spots as the corresponding data points in the parent bioassay set.
2385          </para>
2386        </listitem>
2387       
2388        <listitem>
2389          <para>
2390          The bioassays in the child bioassay set each have exactly one
2391          parent in the parent bioassay set. No parent bioassay may be the
2392          parent of more than one child bioassay.
2393          </para>
2394        </listitem>
2395        </itemizedlist>
2396       
2397        <para>
2398          If any of the above conditions are not true, a new data cube
2399          must be created for the child bioassay set.
2400        </para>
2401      </sect3>
2402     
2403      <sect3 id="data_api.dynamic.description">
2404        <title>The dynamic database</title>
2405
2406        <figure id="data_api.figures.dynamic">
2407          <title>The dynamic database</title>
2408          <screenshot>
2409            <mediaobject>
2410              <imageobject>
2411                <imagedata 
2412                  align="center"
2413                  fileref="figures/uml/datalayer.dynamic.png" format="PNG" />
2414              </imageobject>
2415            </mediaobject>
2416          </screenshot>
2417        </figure>
2418       
2419        <para>
2420          Each virtual database consists of several tables. The tables
2421          are dynamically created when needed. For each table shown in the diagram
2422          the # sign is replaced by the id of the virtual database object at run
2423          time.
2424        </para>
2425       
2426        <para>
2427          There are no classes in the data layer for these tables and they
2428          are not mapped with Hibernate. When we work with these tables we
2429          are always using batcher classes and queries that works with integer,
2430          floats and strings.
2431        </para>
2432       
2433        <bridgehead>The D#Spot table</bridgehead>
2434        <para>
2435          This is the main table which keeps the intensities for a single spot
2436          in the data cube. Extra values attached to the spot are kept in separate
2437          tables, one for each type of value (D#SpotInt, D#SpotFloat and D#SpotString).
2438        </para>
2439       
2440        <bridgehead>The D#Pos table</bridgehead>
2441        <para>
2442          This table stores the reporter id for each position in a cube.
2443          Extra values attached to the position are kept in separate tables,
2444          one for each type of value (D#PosInt, D#PosFloat and D#PosString).
2445        </para>
2446       
2447        <bridgehead>The D#Filter table</bridgehead>
2448        <para>
2449          This table stores the coordinates for the spots that remain after
2450          filtering. Note that each filter is related to a bioassayset which
2451          gives the cube and layer values. Each row in the filter table then
2452          adds the column and position allowing us to find the spots in the
2453          D#Spot table.
2454        </para>
2455       
2456        <bridgehead>The D#RawParents table</bridgehead>
2457        <para>
2458          This table holds mappings for a spot to the raw data it is calculated
2459          from. We don't need the layer coordinate since all layers in a cube
2460          must have the same mapping to raw data.
2461        </para>
2462       
2463      </sect3>     
2464
2465     
2466    </sect2>
2467   
2468    <sect2 id="data_api.misc">
2469      <title>Other classes</title>
2470     
2471        <figure id="data_api.figures.misc">
2472          <title>Other classes</title>
2473          <screenshot>
2474            <mediaobject>
2475              <imageobject>
2476                <imagedata 
2477                  align="center"
2478                  fileref="figures/uml/datalayer.misc.png" format="PNG" />
2479              </imageobject>
2480            </mediaobject>
2481          </screenshot>
2482        </figure>
2483     
2484    </sect2>
2485
2486  </sect1>
2487 
2488  <sect1 id="base_api.core" chunked="1">
2489    <?dbhtml filename="core_api.html" ?>
2490    <title>The Core API</title>
2491   
2492    <para>
2493      This section gives an overview of various parts of the core API.
2494    </para>
2495   
2496    <sect2 id="core_api.authentication">
2497      <title>Authentication and sessions</title>
2498      <para>
2499        This documentation is only available in the old format which may not be up-to-date with
2500        the current implementation.
2501        See <ulink url="http://base.thep.lu.se/chrome/site/doc/historical/development/overview/core/authentication.html"
2502          >http://base.thep.lu.se/chrome/site/doc/historical/development/overview/core/authentication.html</ulink>
2503      </para>
2504    </sect2>   
2505   
2506    <sect2 id="core_api.accesspermissions">
2507      <title>Access permissions</title>
2508      <para>
2509        This documentation is only available in the old format which may not be up-to-date with
2510        the current implementation.
2511        See <ulink url="http://base.thep.lu.se/chrome/site/doc/historical/development/overview/core/accesspermissions.html"
2512          >http://base.thep.lu.se/chrome/site/doc/historical/development/overview/core/accesspermissions.html</ulink>
2513      </para>
2514    </sect2>
2515   
2516    <sect2 id="core_api.datavalidation">
2517      <title>Data validation</title>
2518      <para>
2519        TODO
2520      </para>
2521    </sect2>   
2522    <sect2 id="core_api.transactions">
2523      <title>Transaction handling</title>
2524      <para>
2525        TODO
2526      </para>
2527    </sect2>   
2528    <sect2 id="core_api.crwd">
2529      <title>Create/read/write/delete operations</title>
2530      <para>
2531        This documentation is only available in the old format which may not be up-to-date with
2532        the current implementation.
2533        See <ulink url="http://base.thep.lu.se/chrome/site/doc/historical/development/overview/core/itemhandling.html"
2534          >http://base.thep.lu.se/chrome/site/doc/historical/development/overview/core/itemhandling.html</ulink>
2535      </para>
2536    </sect2>   
2537    <sect2 id="core_api.batch">
2538      <title>Batch operations</title>
2539      <para>
2540        This documentation is only available in the old format which may not be up-to-date with
2541        the current implementation.
2542        See <ulink url="http://base.thep.lu.se/chrome/site/doc/historical/development/overview/core/batchprocessing.html"
2543          >http://base.thep.lu.se/chrome/site/doc/historical/development/overview/core/batchprocessing.html</ulink>
2544      </para>
2545    </sect2>   
2546   
2547    <sect2 id="core_api.quota">
2548      <title>Quota</title>
2549      <para>
2550        TODO
2551      </para>
2552    </sect2>   
2553    <sect2 id="core_api.pluginexecution">
2554      <title>Plugin execution / job queue</title>
2555      <para>
2556        This documentation is only available in the old format which may not be up-to-date with
2557        the current implementation.
2558        See <ulink url="http://base.thep.lu.se/chrome/site/doc/historical/development/overview/core/plugins.html"
2559          >http://base.thep.lu.se/chrome/site/doc/historical/development/overview/core/plugins.html</ulink>
2560      </para>
2561    </sect2>   
2562   
2563    <sect2 id="core_api.data_in_files">
2564      <title>Using files to store data</title>
2565     
2566      <para>
2567        BASE has support for storing data in files instead of importing it into the database.
2568        Files can be attached to any item that implements the <interfacename 
2569        docapi="net.sf.basedb.core">FileStoreEnabled</interfacename>
2570        interface. For example, <classname docapi="net.sf.basedb.core">RawBioAssay</classname>,
2571        and <classname docapi="net.sf.basedb.core">ArrayDesign</classname> and a few other
2572        classes. The ability to store data in files is not a replacement for storing data in the
2573        database. It is possible (for some platforms/raw data types) to have data in
2574        files and in the database at the same time. There are three cases:
2575      </para>
2576     
2577      <itemizedlist>
2578      <listitem>
2579        <para>
2580        Data in files only
2581        </para>
2582      </listitem>
2583      <listitem>
2584        <para>
2585        Data in the database only
2586        </para>
2587      </listitem>
2588      <listitem>
2589        <para>
2590        Data in both files and in the database
2591        </para>
2592      </listitem>
2593      </itemizedlist>
2594     
2595      <para>
2596        Not all three cases are supported for all types of data. This is controlled
2597        by the <classname docapi="net.sf.basedb.core">Platform</classname> class, which may disallow
2598        that data is stored in the database. To check this call
2599        <methodname>Platform.isFileOnly()</methodname> and/or
2600        <methodname>Platform.getRawDataType()</methodname>. If the <methodname>isFileOnly()</methodname>
2601        method returns <constant>true</constant>, the platform can't store data in
2602        the database. If the value is <constant>false</constant> more information
2603        can be obtained by calling <methodname>getRawDataType()</methodname>,
2604        which may return:
2605      </para>
2606     
2607      <itemizedlist>
2608      <listitem>
2609        <para>
2610          <constant>null</constant>: The platform can store data with any
2611          raw data type in the database.
2612        </para>
2613      </listitem>
2614      <listitem>
2615        <para>
2616        A <classname docapi="net.sf.basedb.core">RawDataType</classname> that has <code>isStoredInDb() == true</code>:
2617        The platform can store data in the database but only data with the specified raw
2618        data type.
2619        </para>
2620      </listitem>
2621      <listitem>
2622        <para>
2623        A <classname docapi="net.sf.basedb.core">RawDataType</classname> that has <code>isStoredInDb() == false</code>:
2624        The platform can't store data in the database.
2625        </para>
2626      </listitem>
2627      </itemizedlist>
2628
2629      <para>
2630        Some <classname docapi="net.sf.basedb.core">FileStoreEnabled</classname> items doesn't
2631        have a platform (for example, <classname docapi="net.sf.basedb.core">DerivedBioAssay</classname>).
2632        In this case, the file storage ability is controlled by the subtype of the item.
2633        See <methodname>getDataFileTypes()</methodname> method in the
2634        <classname docapi="net.sf.basedb.core">ItemSubtype</classname> class.
2635      </para>
2636
2637      <para>
2638        For backwards compatibility reasons, each <classname docapi="net.sf.basedb.core">Platform</classname>
2639        that can only store data in files will create "virtual" raw data type
2640        objects internally. These raw data types all return <constant>false</constant>
2641        from the <methodname>RawDataType.isStoredInDb()</methodname>
2642        method. They also have a back-link to the platform/variant that
2643        created it: <methodname>RawDataType.getPlatform()</methodname>
2644        and <methodname>RawDataType.getVariant()</methodname>. These two methods
2645        will always return <constant>null</constant> when called on a raw data type
2646        that can be stored in the database.
2647      </para>
2648     
2649      <itemizedlist>
2650        <title>See also</title>
2651        <listitem><xref linkend="data_api.platforms" /></listitem>
2652        <listitem><xref linkend="extensions_developer.fileset_validator" /></listitem>
2653        <listitem><xref linkend="appendix.rawdatatypes.platforms" /></listitem>
2654      </itemizedlist>
2655     
2656      <sect3 id="core_api.data_in_files.diagram">
2657        <title>Diagram of classes and methods</title>
2658        <figure id="core_api.figures.data_in_files">
2659          <title>Store data in files</title>
2660          <screenshot>
2661            <mediaobject>
2662              <imageobject>
2663                <imagedata 
2664                  align="center"
2665                  scalefit="1" width="100%"
2666                  fileref="figures/uml/corelayer.datainfiles.png" format="PNG" />
2667              </imageobject>
2668            </mediaobject>
2669          </screenshot>
2670        </figure>
2671       
2672        <para>
2673          This is rather large set of classes and methods. The ultimate goal
2674          is to be able to create links between a <classname docapi="net.sf.basedb.core">FileStoreEnabled</classname>
2675          item and <classname docapi="net.sf.basedb.core">File</classname>
2676          items and to provide some metadata about the files.
2677          The <classname docapi="net.sf.basedb.core">FileStoreUtil</classname> class is one of the most
2678          important ones. It is intended to make it easy for plug-in (and other)
2679          developers to access the files without having to mess with platform
2680          or file type objects. The API is best described
2681          by a set of use-case examples.
2682        </para>
2683       
2684      </sect3>
2685     
2686      <sect3 id="core_api.data_in_files.ask">
2687        <title>Use case: Asking the user for files for a given item</title>
2688
2689        <para>
2690          A client application must know what types of files it makes sense
2691          to ask the user for. In some cases, data may be split into more than
2692          one file so we need a generic way to select files.
2693        </para>
2694       
2695        <para>
2696          Given that we have a <interfacename docapi="net.sf.basedb.core">FileStoreEnabled</interfacename>
2697          item we want to find out which <classname docapi="net.sf.basedb.core">DataFileType</classname>
2698          items that can be used for that item. The
2699          <methodname>Base.getDataFileTypes()</methodname>
2700          can be used for this. You'll need to supply information about the platform,
2701          variant and subtype of the item. The method will create a query that returns
2702          a list of <classname>DataFileType</classname> items, each one representing a
2703          specific file type that we should ask the user about. Examples:
2704        </para>
2705
2706        <orderedlist>
2707        <listitem>
2708          <para>
2709          The <constant>Affymetrix</constant> platform defines <constant>CEL</constant>
2710          as a raw data file and <constant>CDF</constant> as an array design (reporter map)
2711          file. If we have a <classname docapi="net.sf.basedb.core">RawBioAssay</classname> 
2712          the query will only return
2713          the CEL file type and the client can ask the user for a CEL file.
2714          </para>
2715        </listitem>
2716        <listitem>
2717          <para>
2718          The <constant>Generic</constant> platform defines <constant>PRINT_MAP</constant>
2719          and <constant>REPORTER_MAP</constant> for array designs. If we have
2720          an <classname docapi="net.sf.basedb.core">ArrayDesign</classname> the query will return those two
2721          items.
2722          </para>
2723        </listitem>
2724        <listitem>
2725          <para>
2726          The <constant>Scan</constant> subtype defines <constant>MICROARRAY_IMAGE</constant>
2727          for derived bioassays.
2728          </para>
2729        </listitem>
2730        </orderedlist>
2731     
2732        <para>
2733          It might also be interesting to know the currently selected file
2734          for each file type and if the file is <varname>required</varname>
2735          and if <varname>multiple</varname> files are allowed.
2736          Here is a simple code example
2737          that may be useful to start from:
2738        </para>
2739     
2740        <programlisting language="java">
2741DbControl dc = ...
2742FileStoreEnabled item = ...
2743Platform platform = item.getPlatform();
2744PlatformVariant variant = item.getVariant();
2745Itemsubtype subtype = item instanceof Subtypable ?
2746   ((Subtypable)item).getItemSubtype() : null;
2747
2748// Get list of DataFileTypes used by the platform
2749ItemQuery&lt;DataFileType&gt; query =
2750   Base.getDataFileTypes(item.getType(), item, platform, variant, subtype);
2751List&lt;DataFileType&gt; types = query.list(dc);
2752
2753// Always check hasFileSet() method first to avoid
2754// creating the file set if it doesn't exists
2755FileSet fileSet = item.hasFileSet() ?
2756   null : item.getFileSet();
2757   
2758for (DataFileType type : types)
2759{
2760   // Get the current file, if any
2761   FileSetMember member = fileSet == null || !fileSet.hasMember(type) ?
2762      null : fileSet.getMember(type);
2763   File current = member == null ?
2764      null : member.getFile();
2765   
2766   // Check if a file is required by the platform/subtype
2767   PlatformFileType pft = platform == null ?
2768      null : platform.getFileType(type, variant, false);
2769   ItemSubtypeFileType ift = subtype == null ?
2770      null : subtype.getAssociatedDataFileType(type, false);
2771   boolean isRequired = pft == null ?
2772      false : pft.isRequired();
2773   isRequired |= ift == null ?
2774      false : ift.isRequired();
2775     
2776   // Now we can do something with this information to
2777   // let the user select a file ...
2778}
2779</programlisting>
2780     
2781        <note>
2782          <title>Also remember to catch PermissionDeniedException</title>
2783          <para>
2784            The above code may look complicated, but this is mostly because
2785            of all checks for <constant>null</constant> values. Remember
2786            that many things are optional and may return <constant>null</constant>.
2787            Another thing to look out for is
2788            <exceptionname>PermissionDeniedException</exceptionname>:s. The logged in
2789            user may not have access to all items. The above example doesn't include
2790            any code for this since it would have made it too complex.
2791          </para>
2792        </note>
2793      </sect3>
2794     
2795      <sect3 id="core_api.data_in_files.link">
2796        <title>Use case: Link, validate and extract metadata from the selected files</title>
2797        <para>
2798          When the user has selected the file(s) we must store the links
2799          to them in the database. This is done with a <classname docapi="net.sf.basedb.core">FileSet</classname>
2800          object. A file set can contain any number of files.
2801          Call <methodname>FileSet.setMember()</methodname> or <methodname>FileSet.addMember()</methodname>
2802          to store a file in the file set. If a file already exists for the given file type
2803          it is replaced if the <methodname>setMember</methodname> method is called.
2804          The following
2805          program example assumes that we have a map where <classname docapi="net.sf.basedb.core">File</classname>:s
2806          are related to <classname docapi="net.sf.basedb.core">DataFileType</classname>:s. When all files
2807          have been added we call <methodname>FileSet.validate()</methodname>
2808          to validate the files and extract metadata.
2809        </para>
2810       
2811        <programlisting language="java">
2812DbControl dc = ...
2813FileStoreEnabled item = ...
2814Map&lt;DataFileType, File&gt; files = ...
2815
2816// Store the selected files in the fileset
2817FileSet fileSet = item.getFileSet();
2818for (Map.Entry&lt;DataFileType, File&gt; entry : files)
2819{
2820   DataFileType type = entry.getKey();
2821   File file = entry.getValue();
2822   fileSet.setMember(type, file);
2823}
2824
2825// Validate the files and extract metadata
2826fileSet.validate(dc);
2827</programlisting>
2828
2829        <para>
2830          Validation and extraction of metadata is important since we want
2831          data in files to be equivalent to data in the database. The validation
2832          and metadata extraction is initiated by the core when the
2833          <methodname>FileSet.validate()</methodname> is called.
2834          The validation and metadata extraction is handled by extensions
2835          so the actual outcome depends on what has been installed on the
2836          BASE server.
2837        </para>
2838
2839        <note>
2840          <para>
2841          The <methodname>FileSet.validate()</methodname> method doesn't
2842          throw any exceptions. Instead, all validation errors
2843          are returned a list of <classname>Throwable</classname>:s. The
2844          validation result is also stored for each file and can be access
2845          with <methodname>FileSetMember.isValid()</methodname> and
2846          <methodname>FileSetMember.getErrorMessage()</methodname>.
2847          </para>
2848        </note>
2849
2850        <para>
2851          Here is the general outline of what is going on in the core:
2852        </para>
2853
2854        <orderedlist>
2855        <listitem>
2856          <para>
2857          The core calls the main <classname 
2858          docapi="net.sf.basedb.util.extensions.manager">ExtensionsManager</classname>
2859          and initiates the action factory for all file set validator extensions.
2860          </para>
2861        </listitem>
2862       
2863        <listitem>
2864          <para>
2865          After inspecting the current item and file set, the factories create
2866          one or more <interfacename 
2867          docapi="net.sf.basedb.util.fileset">ValidationAction</interfacename>:s.
2868          </para>
2869        </listitem>
2870       
2871        <listitem>
2872          <para>
2873          For each file in the file set, the <methodname>ValidationAction.acceptFile()</methodname> 
2874          method is called on each action, which is supposed to either accept or deny
2875          validation of the file.
2876          </para>
2877        </listitem>
2878       
2879        <listitem>
2880          <para>
2881          If the file is accepted the <methodname>ValidationAction.validateAndExtractMetadata()</methodname>
2882          method is called.
2883          </para>
2884        </listitem>
2885        </orderedlist>
2886
2887        <note>
2888          <title>Only one instance of each validator class is created</title>
2889          <para>
2890          The validation is not done until all files have been
2891          added to the fileset. If the same validator is
2892          used for more than one file, the same instance is reused. Eg.
2893          the <methodname>acceptFile()</methodname> is called one time
2894          for each file. Depending on the return value, the
2895          <methodname>validateAndExtractMetadata()</methodname> may be called either
2896          immediately or not until all files have been processed.
2897          </para>
2898        </note>
2899       
2900      </sect3>
2901     
2902      <sect3 id="core_api.data_in_files.import">
2903        <title>Use case: Import data into the database</title>
2904       
2905        <para>
2906          This should be done by existing plug-ins in the same way as before.
2907          A slight modification is needed since it is good if the importers
2908          are made aware of already selected files in the <classname docapi="net.sf.basedb.core">FileSet</classname>
2909          to provide good default values. The <classname docapi="net.sf.basedb.core">FileStoreUtil</classname>
2910          class is very useful in cases like this:
2911        </para>
2912       
2913        <programlisting language="java">
2914RawBioAssay rba = ...
2915DbControl dc = ...
2916
2917// Get the current raw data file, if any
2918List&lt;File&gt; rawDataFiles =
2919   FileStoreUtil.getGenericDataFiles(dc, rba, FileType.RAW_DATA);
2920File defaultFile = rawDataFiles.size() > 0 ?
2921   rawDataFiles.get(0) : null;
2922   
2923// Create parameter asking for input file - use current as default
2924PluginParameter&lt;File&gt; fileParameter = new PluginParameter&lt;File&gt;(
2925   "file",
2926   "Raw data file",
2927   "The file that contains the raw data that you want to import",
2928   new FileParameterType(defaultFile, true, 1)
2929);
2930</programlisting>
2931
2932      <para>
2933        An import plug-in should also save the file that was used to the file set:
2934      </para>
2935     
2936      <programlisting language="java">
2937RawBioassay rba = ...
2938// The file the user selected to import from
2939File rawDataFile = (File)job.getValue("file");
2940
2941// Save the file to the fileset. The method will check which file
2942// type the platform uses as the raw data type. As a fallback the
2943// GENERIC_RAW_DATA type is used
2944FileStoreUtil.setGenericDataFile(dc, rba, FileType.RAW_DATA,
2945   DataFileType.GENERIC_RAW_DATA, rawDataFile);
2946</programlisting>
2947
2948      </sect3>
2949     
2950      <sect3 id="core_api.data_in_files.experiments">
2951        <title>Use case: Using raw data from files in an experiment</title>
2952       
2953        <para>
2954          Just as before, an experiment is still locked to a single
2955          <classname docapi="net.sf.basedb.core">RawDataType</classname>. This is a design issue that
2956          would break too many things if changed. If data is stored in files
2957          the experiment is also locked to a single <classname docapi="net.sf.basedb.core">Platform</classname>.
2958          This has been designed to have as little impact on existing
2959          plug-ins as possible. In most cases, the plug-ins will continue
2960          to work as before.
2961        </para>
2962       
2963        <para>
2964          A plug-in (using data from the database that needs to check if it can
2965          be used within an experiment can still do:
2966        </para>
2967       
2968        <programlisting language="java">
2969Experiment e = ...
2970RawDataType rdt = e.getRawDataType();
2971if (rdt.isStoredInDb())
2972{
2973   // Check number of channels, etc...
2974   // ... run plug-in code ...
2975}
2976</programlisting>
2977       
2978        <para>
2979          A newer plug-in which uses data from files should do:
2980        </para>
2981       
2982        <programlisting language="java">
2983Experiment e = ...
2984DbControl dc = ...
2985RawDataType rdt = e.getRawDataType();
2986if (!rdt.isStoredInDb())
2987{
2988   // Check that platform/variant is supported
2989   Platform p = rdt.getPlatform(dc);
2990   PlatformVariant v = rdt.getVariant(dc);
2991   // ...
2992
2993   // Get data files
2994   File aFile = FileStoreUtil.getDataFile(dc, ...);
2995   
2996   // ... run plug-in code ...
2997}
2998</programlisting>
2999       
3000      </sect3>
3001     
3002    </sect2>
3003   
3004    <sect2 id="core_api.signals">
3005      <title>Sending signals (to plug-ins)</title>
3006   
3007      <para>
3008        BASE has a simple system for sending signals between different parts of
3009        a system. This signalling system was initially developed to be able to
3010        kill plug-ins that a user for some reason wanted to abort. The signalling
3011        system as such is not limited to this and it can be used for other purposes
3012        as well. Signals can of course be handled internally in a single JVM but
3013        also sent externally to other JVM:s running on the same or a different
3014        computer. The transport mechanism for signals is decoupled from the actual
3015        handling of them. If you want to, you could implement a signal transporter
3016        that sends signal as emails and the target plug-in would never know.
3017      </para>
3018     
3019      <para>
3020        The remainder of this section will focus mainly on the sending and
3021        transportation of signals. For more information about handling signals
3022        on the receiving end, see <xref linkend="plugin_developer.signals" />.
3023      </para>
3024     
3025      <sect3 id="core_api.signals.diagram">
3026        <title>Diagram of classes and methods</title>
3027        <figure id="core_api.figures.signals">
3028          <title>The signalling system</title>
3029          <screenshot>
3030            <mediaobject>
3031              <imageobject>
3032                <imagedata 
3033                  align="center"
3034                  scalefit="1" width="100%"
3035                  fileref="figures/uml/corelayer.signals.png" format="PNG" />
3036              </imageobject>
3037            </mediaobject>
3038          </screenshot>
3039        </figure>
3040     
3041        <para>
3042          The signalling system is rather simple. An object that wish
3043          to receieve signals must implement the
3044          <interfacename docapi="net.sf.basedb.core.signal"
3045          >SignalTarget</interfacename>. It's only method
3046          is <methodname>getSignalHandler()</methodname>. A
3047          <interfacename docapi="net.sf.basedb.core.signal"
3048          >SignalHandler</interfacename> is an object that
3049          knows what to do when a signal is delivered to it. The target object
3050          may implement the <interfacename>SignalHandler</interfacename> itself
3051          or use one of the existing handlers.
3052        </para>
3053       
3054        <para>
3055          The difficult part here is to be aware that a signal is usually
3056          delivered by a separate thread. The target object must be aware
3057          of this and know how to handle multiple threads. As an example we
3058          can use the <classname docapi="net.sf.basedb.core.signal"
3059          >ThreadSignalHandler</classname> which simply
3060          calls <code>Thread.interrupt()</code> to deliver a signal. The target
3061          object that uses this signal handler must know that it should check
3062          <code>Thread.interrupted()</code> at regular intervals from the worker
3063          thread. If that method returns true, it means that the <constant>ABORT</constant>
3064          signal has been delivered and the main thread should clean up and exit as
3065          soon as possible.
3066        </para>
3067       
3068        <para>
3069          Even if a signal handler could be given directly to the party
3070          that may be interested in sending a signal to the target this
3071          is not recommended. This would only work when sending signals
3072          within the same virtual machine. The signalling system includes
3073          <interfacename docapi="net.sf.basedb.core.signal"
3074          >SignalTransporter</interfacename> and
3075          <interfacename docapi="net.sf.basedb.core.signal"
3076          >SignalReceiver</interfacename> objects that are used
3077          to decouple the sending of signals with the handling of signals. The
3078          implementation usually comes in pairs, for example
3079          <classname docapi="net.sf.basedb.core.signal"
3080          >SocketSignalTransporters</classname> and <classname 
3081          docapi="net.sf.basedb.core.signal">SocketSignalReceiver</classname>.
3082        </para>
3083       
3084        <para>
3085          Setting up the transport mechanism is usually a system responsibility.
3086          Only the system know what kind of transport that is appropriate for it's current
3087          setup. Ie. should signals be delievered by TCP/IP sockets, only internally, or
3088          should a delivery mechanism based on web services be implemented?
3089          If a system wants to receive signals it must create an appropriate
3090          <interfacename>SignalReceiver</interfacename> object. Within BASE the
3091          internal job queue set up it's own signalling system that can be used to
3092          send signals (eg. kill) running jobs. The job agents do the same but uses
3093          a different implementation. See <xref linkend="appendix.base.config.jobqueue" />
3094          for more information about how to configure the internal job queue's
3095          signal receiver. In both cases, there is only one signal receiver instance
3096          active in the system.
3097        </para>
3098       
3099        <para>
3100          Let's take the internal job queue as an example. Here is how it works:
3101        </para>
3102       
3103        <itemizedlist>
3104        <listitem>
3105          <para>
3106          When the internal job queue is started, it will also create a signal
3107          receiver instance according to the settings in <filename>base.config</filename>.
3108          The default is to create <classname docapi="net.sf.basedb.core.signal"
3109          >LocalSignalReceiver</classname>
3110          which can only be used inside the same JVM. If needed, this can
3111          be changed to a <classname docapi="net.sf.basedb.core.signal"
3112          >SocketSignalReceiver</classname> or any other
3113          user-provided implementation.
3114          </para>
3115        </listitem>
3116       
3117        <listitem>
3118          <para>
3119          When the job queue has found a plug-in to execute it will check if
3120          it also implements the <interfacename docapi="net.sf.basedb.core.signal"
3121          >SignalTarget</interfacename>
3122          interface. If it does, a signal handler is created and registered
3123          with the signal receiver. This is actually done by the BASE core
3124          by calling <methodname>PluginExecutionRequest.registerSignalReceiver()</methodname>
3125          which also makes sure that the the ID returned from the registration is
3126          stored in the database together with the job item representing the
3127          plug-in to execute.
3128          </para>
3129        </listitem>
3130       
3131        <listitem>
3132          <para>
3133          Now, when the web client see's a running job which has a non-empty
3134          signal transporter property, the <guilabel>Abort</guilabel>
3135          button is activated. If the user clicks this button the BASE core
3136          uses the information in the database to create
3137          <interfacename docapi="net.sf.basedb.core.signal"
3138          >SignalTransporter</interfacename> object. This
3139          is simply done by calling <code>Job.getSignalTransporter()</code>.
3140          The created signal transporter knows how to send a signal
3141          to the signal receiver it was first registered with. When the
3142          signal arrives at the receiver it will find the handler for it
3143          and call <code>SignalHandler.handleSignal()</code>. This will in it's turn
3144          trigger some action in the signal target which soon will abort what
3145          it is doing and exit.
3146          </para>
3147        </listitem>
3148        </itemizedlist>
3149       
3150       
3151      </sect3>
3152   
3153    </sect2>
3154   
3155  </sect1>
3156
3157  <sect1 id="core_api.query">
3158    <?dbhtml filename="query_api.html" ?>
3159    <title>The Query API</title>
3160    <para>
3161      This documentation is only available in the old format which may not be up-to-date with
3162      the current implementation.
3163      See <ulink url="http://base.thep.lu.se/chrome/site/doc/historical/development/overview/query/index.html"
3164        >http://base.thep.lu.se/chrome/site/doc/historical/development/overview/query/index.html</ulink>
3165    </para>
3166   
3167  </sect1>
3168 
3169  <sect1 id="core_api.dynamic">
3170    <?dbhtml filename="dynamic_api.html" ?>
3171    <title>The Dynamic API</title>
3172    <para>
3173      This documentation is only available in the old format which may not be up-to-date with
3174      the current implementation.
3175      See <ulink url="http://base.thep.lu.se/chrome/site/doc/historical/development/overview/dynamic/index.html"
3176        >http://base.thep.lu.se/chrome/site/doc/historical/development/overview/dynamic/index.html</ulink>
3177    </para>
3178  </sect1>
3179
3180  <sect1 id="core_api.extensions">
3181    <?dbhtml filename="extensions_api.html" ?>
3182    <title>The Extensions API</title>
3183   
3184    <sect2 id="extensions_api.core">
3185      <title>The core part</title>
3186   
3187      <para>
3188        The <emphasis>Extensions API</emphasis> is divided into two parts. A core
3189        part and a web client specific part. The core part can be found in the
3190        <package>net.sf.basedb.util.extensions</package> package and it's sub-packages,
3191        and consists of two sub-parts:
3192      </para>
3193     
3194      <itemizedlist>
3195      <listitem>
3196        <para>
3197        An <classname docapi="net.sf.basedb.util.extensions.manager">ExtensionsManager</classname>
3198        that keeps track of the JAR files on the file system containing extensions code.
3199        The manager can detect new, updated and deleted files and is used to
3200        load metadata information about the extensions and register them in the <classname 
3201        docapi="net.sf.basedb.util.extensions">Registry</classname> so that they can be used.
3202        The manager is also used to install plug-ins.
3203        </para>
3204      </listitem>
3205
3206      <listitem>
3207        <para>
3208        A set of interface definitions which forms the core of the Extensions API.
3209        The interfaces defines, for example, what an <interfacename 
3210        docapi="net.sf.basedb.util.extensions">Extension</interfacename> is,
3211        what an <interfacename 
3212        docapi="net.sf.basedb.util.extensions">ActionFactory</interfacename> should do
3213        and a few other things.
3214        </para>
3215      </listitem>
3216      </itemizedlist>
3217     
3218      <para>
3219        Let's start by looking at the extensions manager and related classes.
3220      </para>
3221     
3222      <figure id="extensions_api.figures.manager">
3223        <title>The extensions manager</title>
3224        <screenshot>
3225          <mediaobject>
3226            <imageobject>
3227              <imagedata 
3228                align="center"
3229                fileref="figures/uml/corelayer.extensions_manager.png" format="PNG" />
3230            </imageobject>
3231          </mediaobject>
3232        </screenshot>
3233      </figure>
3234     
3235      <para>
3236      The BASE application is using a single manager and a single registry (handled by the
3237      <classname docapi="net.sf.basedb.core">Application</classname>) class.
3238      The manager has been configured to look for extensions and plug-ins in the directory
3239      specified by the <property>plugins.dir</property> setting in
3240      <filename>base.config</filename>. Theoretically, a single manager can handle
3241      multiple directories, but we do not use that feature. The BASE core also
3242      include some special files that are added with the <methodname>addURI()</methodname>
3243      method. They contain definitions for the core extensions and core plug-ins
3244      and are shipped as XML files that reside inside the BASE core JAR files.
3245      </para>
3246     
3247      <para>
3248      The <methodname>ExtensionsManager.scanForChanges()</methodname>
3249      method is called to initiate a check for new, updated and deleted files. The
3250      manager uses the <classname docapi="net.sf.basedb.util.extensions.xml">XmlLoader</classname>
3251      to find information about each JAR or XML file it find in the directory.
3252      After the scan, the <methodname>ExtensionsManager.getFiles()</methodname> 
3253      method can be used to find more information about each individual file, for example,
3254      if it is a new or modified file, if it contains valid extension definitions and
3255      information about the author, etc. This information is used by the installation
3256      wizard in the web client to display a dialog were the user can select which extensions
3257      to intall. See <xref linkend="plugins.figures.installwizard" />.
3258      Note that no installation or other actions take place at this stage.
3259      </para>
3260     
3261      <para>
3262      The <methodname>ExtensionsManager.processFiles()</methodname> method is called to actually do
3263      something. It needs an <interfacename docapi="net.sf.basedb.util.extensions.manager"
3264      >ExtensionsFileProcessor</interfacename> implementation as an argument. As you can see
3265      in the diagram above there are multiple implementations (all are not
3266      shown in the diagram), each with a very specific task.  A processor is usually
3267      also paired with a filter to target it at files that fulfil some criteria, for example,
3268      only at valid extension files that has been updated. Typically, the
3269      <methodname>ExtensionsManager.processFiles()</methodname> method need to be
3270      called multiple times with different processor implementations to perform a full
3271      installation of an extension or plug-in. Here is a list of the various processors
3272      currently in use in BASE.
3273      </para>
3274     
3275      <variablelist>
3276        <varlistentry>
3277          <term><classname docapi="net.sf.basedb.util.extensions.manager.processor"
3278            >RegisterExtensionsProcessor</classname></term>
3279          <listitem>
3280          <para>
3281            Is used to register
3282            extensions with the registry. Can be paired with different filters
3283            depending on when it is used. At BASE startup an <classname 
3284            docapi="net.sf.basedb.util.extensions.manager.filter">InstalledFilter</classname>
3285            is used so that only installed extensions are registered.
3286          </para>
3287          </listitem>
3288        </varlistentry>
3289
3290        <varlistentry>
3291          <term><classname docapi="net.sf.basedb.util.extensions.manager.processor"
3292            >UnregisterExtensionsProcessor</classname></term>
3293          <listitem>
3294          <para>
3295            Is used to unregister
3296            extensions when a file has been deleted. This should always be paired with
3297            for example, a <classname docapi="net.sf.basedb.util.extensions.manager.filter"
3298            >DeletedFilter</classname>.
3299          </para>
3300          </listitem>
3301        </varlistentry>
3302
3303        <varlistentry>
3304          <term><classname docapi="net.sf.basedb.util.extensions.manager.processor"
3305            >UnregisterExtensionsProcessor</classname></term>
3306          <listitem>
3307          <para>
3308            Is used to unregister
3309            extensions when a file has been deleted. This should always be paired with
3310            for example, a <classname docapi="net.sf.basedb.util.extensions.manager.filter"
3311            >DeletedFilter</classname>.
3312          </para>
3313          </listitem>
3314        </varlistentry>
3315
3316        <varlistentry>
3317          <term><classname docapi="net.sf.basedb.util.extensions.manager.processor"
3318            >ExtractResourcesProcessor</classname></term>
3319          <listitem>
3320          <para>
3321            Is used to extract
3322            files from the JAR file to a local directory. This is currently used
3323            by the web client to extract JSP files, images, etc. that are
3324            needed by web client extensions.
3325          </para>
3326          </listitem>
3327        </varlistentry>
3328
3329        <varlistentry>
3330          <term><classname docapi="net.sf.basedb.util.extensions.manager.processor"
3331            >DeleteResourcesProcessor</classname></term>
3332          <listitem>
3333          <para>
3334            The opposite of
3335            <classname>ExtractResourcesProcessor</classname>.
3336          </para>
3337          </listitem>
3338        </varlistentry>
3339
3340        <varlistentry>
3341          <term><classname docapi="net.sf.basedb.util.extensions.manager.processor"
3342            >PluginInstallationProcessor</classname></term>
3343          <listitem>
3344          <para>
3345            Is used to install and register plug-ins.
3346          </para>
3347          </listitem>
3348        </varlistentry>
3349       
3350        <varlistentry>
3351          <term><classname docapi="net.sf.basedb.util.extensions.manager.processor"
3352            >DisablePluginsProcessor</classname></term>
3353          <listitem>
3354          <para>
3355            Is used to disable plug-ins
3356            from extensions that have been removed.
3357          </para>
3358          </listitem>
3359        </varlistentry>
3360       
3361        <varlistentry>
3362          <term><classname docapi="net.sf.basedb.util.extensions.manager.processor"
3363            >MarkAsProcessedProcessor</classname></term>
3364       
3365          <listitem>
3366          <para>
3367            This is usually the final
3368            processor that is called and reset the timestamp on all processed files
3369            so that the next time <methodname>ExtensionsManger.scanForChanges()</methodname> 
3370            is called it will know what has been modified.
3371          </para>
3372          </listitem>
3373        </varlistentry>
3374      </variablelist>
3375     
3376      <note>
3377        <para>
3378        This list contains the core processors only. The web client part
3379        is using some additional processors to perform, for example, servlet
3380        registration.
3381        </para>
3382      </note>
3383     
3384      <para>
3385      The result of the processing can be collected with a <classname 
3386      docapi="net.sf.basedb.util.extensions.manager">ProcessResults</classname> object
3387      and is then displayed to the user as in <xref linkend="plugins.figures.installresults" />.
3388      All of the above is only used when BASE is starting up and initializing the extensions
3389      system or when the server administrator is performing a manual installation or update.
3390      The next diagram shows the part of the extensions system that is used when
3391      actually using the extensions for what they are intended to do (the web client adds some
3392      extra features to this as well, but that is discussed later).
3393      </para>
3394     
3395     
3396      <figure id="extensions_api.figures.core">
3397        <title>The main Extensions API</title>
3398        <screenshot>
3399          <mediaobject>
3400            <imageobject>
3401              <imagedata 
3402                align="center"
3403                fileref="figures/uml/corelayer.extensions_core.png" format="PNG" />
3404            </imageobject>
3405          </mediaobject>
3406        </screenshot>
3407      </figure>
3408     
3409      <para>
3410        The <classname docapi="net.sf.basedb.util.extensions">Registry</classname> 
3411        is one of the main classes in the extension system. All extension points and
3412        extensions must be registered before they can be used. Typically, you will
3413        first register extension points and then extensions, beacuse an extension
3414        can't be registered until the extension point it is extending has been
3415        registered.
3416      </para>
3417     
3418      <para>
3419        An <interfacename docapi="net.sf.basedb.util.extensions">ExtensionPoint</interfacename>
3420        is an ID and a definition of an <interfacename docapi="net.sf.basedb.util.extensions">Action</interfacename>
3421        class. The other options (name, description, renderer factory, etc.) are optional.
3422        An <interfacename docapi="net.sf.basedb.util.extensions">Extension</interfacename>
3423        that extends a specific extension point must provide an
3424        <interfacename docapi="net.sf.basedb.util.extensions">ActionFactory</interfacename>
3425        instance that can create actions of the type the extension point requires.
3426      </para>
3427     
3428      <example id="extensions_api.example.menu">
3429        <title>The menu extensions point</title>
3430        <para>
3431        The <code>net.sf.basedb.clients.web.menu.extensions</code> extension point
3432        requires <interfacename 
3433        docapi="net.sf.basedb.clients.web.extensions.menu">MenuItemAction</interfacename>
3434        objects. An extension for this extension point must provide a factory that
3435        can create <classname>MenuItemAction</classname>:s. BASE ships with default
3436        factory implementations, for example the <classname 
3437        docapi="net.sf.basedb.clients.web.extensions.menu">FixedMenuItemFactory</classname>
3438        class, but an extension may provide it's own factory implementation if it wants to.
3439        </para>
3440      </example>
3441     
3442      <para>
3443        Call the <methodname>Registry.useExtensions()</methodname> method
3444        to use extensions from one or several extension points. This method will
3445        find all extensions for the given extension points. If a filter is given,
3446        it checks if any of the extensions or extension points has been disabled.
3447        It will then call <methodname>ActionFactory.prepareContext()</methodname>
3448        for all remaining extensions. This gives the action factory a chance to
3449        also disable the extension, for example, if the logged in user doesn't
3450        have a required permission. The action factory may also set attributes
3451        on the context. The attributes can be anything that the extension point
3452        may make use of. Check the documentation for the specific extension point
3453        for information about which attributes it supports. If there are
3454        any renderer factories, their <methodname>RendererFactory.prepareContext()</methodname>
3455        is also called. They have the same possibility of setting attributes
3456        on the context, but can't disable an extension.
3457      </para>
3458     
3459      <para>
3460        After this, an <classname 
3461        docapi="net.sf.basedb.util.extensions">ExtensionsInvoker</classname>
3462        object is created and returned to the extension point. Note that
3463        the <methodname>ActionFactory.getActions()</methodname> has not been
3464        called yet, so we don't know if the extensions are actually
3465        going to generate any actions. The <methodname>ActionFactory.getActions()</methodname>
3466        is not called until we have got ourselves an
3467        <classname docapi="net.sf.basedb.util.extensions">ActionIterator</classname>
3468        from the <methodname>ExtensionsInvoker.iterate()</methodname> method and
3469        starts to iterate. The call to <methodname>ActionIterator.hasNext()</methodname>
3470        will propagate down to <methodname>ActionFactory.getActions()</methodname>
3471        and the generated actions are then available with the
3472        <methodname>ActionIterator.next()</methodname> method.
3473      </para>
3474     
3475      <para>
3476        The <methodname>ExtensionsInvoker.renderDefault()</methodname>
3477        and <methodname>ExtensionsInvoker.render()</methodname> are
3478        just convenience methods that will make it easer to render
3479        the actions. The first method will of course only work if the
3480        extension point is providing a renderer factory, that can
3481        create the default renderer.
3482      </para>
3483     
3484      <note>
3485        <title>Be aware of multi-threading issues</title>
3486        <para>
3487          When you are creating extensions you must be aware that
3488          multiple threads may access the same objects at the same time.
3489          In particular, any action factory or renderer factory has to be
3490          thread-safe, since only one exists for each extension.
3491          Action and renderer objects should be thread-safe if the
3492          factories re-use the same objects.
3493        </para>
3494      </note>
3495   
3496      <para>
3497        Any errors that happen during usage of an extension is handled by an
3498        <interfacename docapi="net.sf.basedb.util.extensions">ErrorHandler</interfacename>.
3499        The core provides two implementations. We usually don't want the
3500        errors to show up in the gui so the <classname 
3501        docapi="net.sf.basedb.util.extensions">LoggingErrorHandlerFactory</classname> 
3502        is the default implementation that only writes to the log file. The
3503        <classname 
3504        docapi="net.sf.basedb.util.extensions">RethrowErrorHandlerFactory</classname>
3505        error handler can be used to re-throw exceptions which usually means that
3506        they trickle up to the gui and are shown to the user. It is also
3507        possible for an extension point to provide its own implementation of
3508        an <interfacename docapi="net.sf.basedb.util.extensions">ErrorHandlerFactory</interfacename>.
3509      </para>
3510   
3511    </sect2>
3512   
3513    <sect2 id="extensions_api.web">
3514      <title>The web client part</title>
3515   
3516      <para>
3517        The web client specific parts of the Extensions API can be found
3518        in the <package>net.sf.basedb.client.web.extensions</package> package
3519        and it's subpackages. The top-level package contains the
3520        <classname docapi="net.sf.basedb.client.web.extensions">ExtensionsControl</classname>
3521        class which is used to administrate the extension system. It is more or
3522        less a wrapper around the <classname 
3523        docapi="net.sf.basedb.util.extensions.manager">ExtensionsManager</classname>
3524        provided by the core, but adds permission control and a few other things.
3525      </para>
3526     
3527      <para>
3528        In the top-level package there are also some abstract classes that may
3529        be useful to extend for developers creating their own extensions.
3530        For example, we recommend that all action factories extend the <classname 
3531        docapi="net.sf.basedb.client.web.extensions">AbstractJspActionFactory</classname>
3532        class. All web client extension points use the <classname 
3533        docapi="net.sf.basedb.client.web.extensions">JspContext</classname> class instead of
3534        <classname docapi="net.sf.basedb.util.extensions">ClientContext</classname>.
3535        The JSP context provides some extra information about the current request.
3536      </para>
3537     
3538      <para>
3539        The sub-packages to <package>net.sf.basedb.client.web.extensions</package>
3540        are mostly specific to a single extension point or to a specific type of
3541        extension point. The <package>net.sf.basedb.client.web.extensions.menu</package>
3542        package, for example, contains classes that are/can be used for extensions
3543        adding menu items to the <menuchoice><guimenu>Extensions</guimenu></menuchoice>
3544        menu. See <xref linkend="extensions_developer.base_extension_points" />
3545        for more information about the extension points defined by BASE.
3546      </para>
3547     
3548      <figure id="extensions_api.figures.web">
3549        <title>The web client part of the Extensions API</title>
3550        <screenshot>
3551          <mediaobject>
3552            <imageobject>
3553              <imagedata 
3554                align="center"
3555                fileref="figures/uml/corelayer.extensions_web.png" format="PNG" />
3556            </imageobject>
3557          </mediaobject>
3558        </screenshot>
3559      </figure>
3560   
3561      <para>
3562        When the Tomcat web server is starting up, the <classname 
3563        docapi="net.sf.basedb.client.web.extensions">ExtensionsServlet</classname>
3564        is automatically loaded. This servlet has as two purposes:
3565      </para>
3566     
3567      <itemizedlist>
3568      <listitem>
3569        <para>
3570        Initialise the extensions system by calling
3571        <methodname>ExtensionsControl.init()</methodname>. This will result in
3572        an initial scan for installed extensions. This means that
3573        the extension system is up an running as soon as the first user log's
3574        in to BASE. The processing scheme is slightly different from what is done
3575        when the core is setting up the initial manager. The major additions are:
3576        </para>
3577       
3578        <itemizedlist>
3579          <listitem>
3580            <para>
3581            The <classname docapi="net.sf.basedb.client.web.extensions"
3582            >LoadServletsProcessor</classname> is used to load servlets that
3583            has been defined in <filename>META-INF/servlets.xml</filename>.
3584            </para>
3585          </listitem>
3586          <listitem>
3587            <para>
3588            The <classname docapi="net.sf.basedb.util.extensions.manager.processor"
3589            >ExtractResourcesProcessor</classname> is used to extract all files
3590            from <filename>resources/*</filename> in the JAR file to
3591            <filename>www/extensions/jar-name.jar/</filename>
3592            in Tomcat's web application directory.
3593            </para>
3594          </listitem>
3595        </itemizedlist>
3596       
3597      </listitem>
3598     
3599      <listitem>
3600        <para>
3601        Act as a proxy for custom servlets defined by the extensions. URL:s
3602        ending with <code>.servlet</code> has been mapped to the
3603        <classname>ExtensionsServlet</classname>. When a request is made it
3604        will extract the name of the extension's JAR file from the
3605        URL, get the corresponding <classname 
3606        docapi="net.sf.basedb.client.web.extensions">ServletWrapper</classname>
3607        and then invoke the custom servlet. More information can be found in
3608        <xref linkend="extensions_developer.servlets" />.
3609        </para>
3610      </listitem>
3611     
3612      </itemizedlist>
3613     
3614      <para>
3615        Using extensions only involves calling the
3616        <methodname>ExtensionsControl.createContext()</methodname> and
3617        <methodname>ExtensionsControl.useExtensions()</methodname> methods. This
3618        returns an <classname docapi="net.sf.basedb.util.extensions">ExtensionsInvoker</classname> 
3619        object as described in the previous section.
3620      </para>
3621     
3622      <para>
3623        To render the actions it is possible to either use the
3624        <methodname>ExtensionsInvoker.iterate()</methodname> method
3625        and generate HTML from the information in each action. Or
3626        (the better way) is to use a renderer together with the
3627        <classname docapi="net.sf.basedb.clients.web.taglib.extensions">Render</classname>
3628        taglib.
3629      </para>
3630     
3631      <para>
3632        To get information about the installed extensions, 
3633        change settings, enabled/disable extensions, performing a manual
3634        installation, etc. use the <methodname>ExtensionsControl.get()</methodname>
3635        method. This will create a permission-controlled object. All
3636        users has read permission, administrators has write permission.
3637      </para>
3638     
3639      <note>
3640        <para>
3641          The permission we check for is WRITE permission on the
3642          web client item. This means it is possible to give a user
3643          permissions to manage the extension system by assigning
3644          WRITE permission to the web client entry in the database.
3645          Do this from <menuchoice>
3646            <guimenu>Administrate</guimenu>
3647            <guimenuitem>Clients</guimenuitem>
3648          </menuchoice>.
3649        </para>
3650      </note>
3651   
3652      <para>
3653        The <classname docapi="net.sf.basedb.clients.web.extensions">XJspCompiler</classname>
3654        is mapped to handle the compilation <code>.xjsp</code> files
3655        which are regular JSP files with a different extension. The difference is that
3656        the XJSP compiler include the extension's JAR file on the class path, which
3657        means that the JSP file can use classes that would otherwise be impossible.
3658        This feature is experimental and requires installing an extra JAR into Tomcat's lib
3659        directory. See <xref linkend="plugins.installation.xjspcompiler" /> for
3660        more information.
3661      </para>
3662   
3663    </sect2>
3664   
3665  </sect1>
3666
3667  <sect1 id="base_api.other">
3668    <title>Other useful classes and methods</title>
3669    <para>
3670      TODO
3671    </para>
3672  </sect1>
3673 
3674</chapter>
Note: See TracBrowser for help on using the repository browser.