source: trunk/doc/src/docbook/appendix/raw_data_types.xml @ 4005

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE appendix PUBLIC 
    "-//Dawid Weiss//DTD DocBook V3.1-Based Extension for XML and graphics inclusion//EN" 
    "../../../../lib/docbook/preprocess/dweiss-docbook-extensions.dtd">
<!--
  $Id: raw_data_types.xml 4005 2007-11-26 15:00:38Z martin $
  
  Copyright (C) 2007 Nicklas Nordborg
  
  This file is part of BASE - BioArray Software Environment.
  Available at http://base.thep.lu.se/
  
  BASE is free software; you can redistribute it and/or
  modify it under the terms of the GNU General Public License
  as published by the Free Software Foundation; either version 2
  of the License, or (at your option) any later version.
  
  BASE is distributed in the hope that it will be useful,
  but WITHOUT ANY WARRANTY; without even the implied warranty of
  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
  GNU General Public License for more details.
  
  You should have received a copy of the GNU General Public License
  along with this program; if not, write to the Free Software
  Foundation, Inc., 59 Temple Place - Suite 330,
  Boston, MA  02111-1307, USA.
-->

<appendix id="appendix.rawdatatypes">
  <title>Platforms and raw-data-types.xml reference</title>
  
  <para>
    Raw data can be stored either as files attached to items and/or in
    the database. The <classname docapi="net.sf.basedb.core">Platform</classname> 
    item has information  about this.  For more information see 
    <xref linkend="core_api.data_in_files" />.
  </para>
  
  <sect1 id="appendix.rawdatatypes.platforms">
    <title>Default platforms/variants installed with BASE</title>
    
    <informaltable>
      <tgroup cols="7">
        <colspec colname="platform.name" />
        <colspec colname="platform.id" />
        <colspec colname="variant.name" />
        <colspec colname="variant.id" />
        <colspec colname="filetype.item" />
        <colspec colname="filetype.name" />
        <colspec colname="filetype.id" />
        <thead>
          <row>
            <entry namest="platform.name" nameend="platform.id">Platform</entry>
            <entry namest="variant.name" nameend="variant.id">Variants</entry>
            <entry namest="filetype.item" nameend="filetype.id">Data file types</entry>
          </row>
          <row>
            <entry>Name</entry>
            <entry>ID</entry>
            <entry>Name</entry>
            <entry>ID</entry>
            <entry>Item</entry>
            <entry>Name</entry>
            <entry>ID</entry>
          </row>
        </thead>
        <tbody>
          <row>
            <entry morerows="2">Generic</entry>
            <entry morerows="2">generic</entry>
            <entry morerows="2">-</entry>
            <entry morerows="2">-</entry>

            <entry morerows="1">Array design</entry>
            <entry>Reporter map</entry>
            <entry>generic.reportermap</entry>
          </row>
          <row>
            <entry>Print map</entry>
            <entry>generic.printmap</entry>
          </row>
          <row>
            <entry>Raw bioassay</entry>
            <entry>Generic raw data</entry>
            <entry>generic.rawdata</entry>
          </row>
          <row>
            <entry morerows="1">Affymetrix</entry>
            <entry morerows="1">affymetrix</entry>
            <entry morerows="1">-</entry>
            <entry morerows="1">-</entry>
            <entry>Array design</entry>
            <entry>CDF file</entry>
            <entry>affymetrix.cdf</entry>
          </row>
          <row>
            <entry>Raw bioassay</entry>
            <entry>CEL file</entry>
            <entry>affymetrix.cel</entry>
          </row>
        </tbody>
      </tgroup>
    </informaltable>
    
  
  </sect1>
  
  <sect1 id="appendix.rawdatatypes.ref">
    <title>raw-data-types.xml reference</title>
    
    <para>
      A given platform either supports importing data to the database or it
      doesn't. If it supports import, it may be locked to specific raw data type
      or it may use any raw data type. Among the default platforms installed with
      BASE, the Affymetrix platform doesn't support importing data while the Generic 
      platform supports importing to any raw data type.
    </para>
    
    <para>
      Raw data types are defined in the <filename>raw-data-types.xml</filename>
      file. This file is located in the <filename>&lt;basedir&gt;/www/WEB-INF/classes</filename>
      directory and contains information about the database tables and columns to
      use for storing raw data. BASE ships with default raw data types for many
      different microarray platforms, including Genepix, Agilent and Illumina.
    </para>
    
    <para>
      If you want your BASE installation to be configured differently we recommend that 
      you do it before the first initialisation of the database.
      It is possible to change the configuration of an existing BASE installation but it
      requires manual updates to the database. Following procedure covers how to update:
    </para>

  <orderedlist>
  <listitem>
    <para>
    Shut down the BASE web server. If you have installed job agents you should shut
    down them as well.
    </para>
  </listitem>
  
  <listitem>
    <para>
    Modify the <filename>raw-data-types.xml</filename> file. If you have installed
    job agents, make sure they all have the same version as the web server.
    </para>
  </listitem>
  
  <listitem>
    <para>
    Run the <filename>updatedb.sh</filename> script. Tables for new raw data types
    and new columns for existing raw data types automatically be created, but the script 
    can't delete tables or columns that have been removed, or modify columns that have 
    changed datatype. You will have to do these kind of changes by manually executing
    SQL against your database. Check your database documentation for information about SQL syntax.
    </para>
    
    <tip>
      <title>Create a parallel installation</title>
      <para>
      You can always create a new temporary parallel installation to check 
      what the table generated by installation script looks like. Compare the 
      new table to the existing one and make sure they match.
      </para>
    </tip>
  </listitem>
  
  <listitem>
    <para>
    Start up the BASE web server and job agents, if any, again.
    </para>
  </listitem>
  </orderedlist>

  <tip>
    <title>Start with few columns</title>
    <para>
    It is better to start with too few columns, since it is easier to add
    more columns than it is to remove columns that are not needed.
    </para>
  </tip>

  <bridgehead>Format of the raw-data-types.xml file</bridgehead>
  <para>
    The following example will serve as a description of the format used in
    <filename>raw-data-types.xml</filename>:
  </para>


  <programlisting language="xml">
<![CDATA[
<?xml version="1.0" ?>
<?xml-stylesheet type="text/xsl" href="raw-data-types.xsl"?>
<!DOCTYPE raw-data-types SYSTEM "raw-data-types.dtd" >
<raw-data-types>
   <raw-data-type
      id="genepix"
      name="GenePix"
      channels="2"
      table="RawDataGenePix"
      >
      <property
         name="diameter"
         title="Spot diameter"
         description="The diameter of the spot in µm"
         column="diameter"
         type="float"
      />
      <property
         name="ch1FgMedian"
         title="Channel 1 foreground median"
         description="The median of the foreground intensity in channel 1"
         column="ch1_fg_median"
         type="float"
         channel="1"
      />
      <!-- skipped a lot of properties -->
      <intensity-formula
         name="mean"
         title="Mean FG - Mean BG"
         description="Subtract mean background from mean foreground"
         >
         <formula 
            channel="1"
            expression="raw('ch1FgMean') - raw('ch1BgMean')"
         />
         <formula 
            channel="2"
            expression="raw('ch2FgMean') - raw('ch2BgMean')"
         />
      </intensity-formula>
      <!-- and a few more... --->
   </raw-data-type>
</raw-data-types>
]]> 
</programlisting>
  
  <para>
    Each raw data type is represented by a <sgmltag class="starttag">raw-data-type</sgmltag> 
    tag. The following attributes can be used:
  </para>
  
    <table frame="all" id="appendix.rawdatatypes.tag">
    <title>Attributes for the <sgmltag class="starttag">raw-data-type</sgmltag> tag</title>
    <tgroup cols="3" align="left">
      <colspec colname="attribute" align="left" />
      <colspec colname="required" />
      <colspec colname="comment" />
      <thead>
        <row>
          <entry>Attribute</entry>
          <entry>Required</entry>
          <entry>Comment</entry>
        </row>
      </thead>
      <tbody>
        <row>
          <entry>id</entry>
          <entry>yes</entry>
          <entry>
            A unique ID of the raw data type. It should contain only letters,
            numbers and underscores and the first character must be a letter.
          </entry>
        </row>
        <row>
          <entry>name</entry>
          <entry>yes</entry>
          <entry>
            A unique name of the raw data type. The name is usually used by client
            applications for display.
          </entry>
        </row>
        <row>
          <entry>table</entry>
          <entry>yes</entry>
          <entry>
            The name of the database table to store data in. The table name
            must be unique and can only contain letters,
            numbers and underscores. The first character must be a letter.
          </entry>
        </row>
        <row>
          <entry>channels</entry>
          <entry>yes</entry>
          <entry>
            The number of channels used by this raw data type. It must be 
            a number &gt; 0.
          </entry>
        </row>
        <row>
          <entry>description</entry>
          <entry>no</entry>
          <entry>
            An optional (longer) description of the raw data type.
          </entry>
        </row>
      </tbody>
    </tgroup>
    </table>
    
    <para>
      Following the <sgmltag class="starttag">raw-data-type</sgmltag> tag
      is one or more  <sgmltag class="starttag">property</sgmltag> tags.
      Each one defines a column in the database that is designed to hold 
      data values of a particular type. The following attributes can be used
      on this tag:
    </para>
  
    <table frame="all" id="appendix.rawdatatypes.property">
    <title>Attributes for the <sgmltag class="starttag">property</sgmltag> tag</title>
    <tgroup cols="3" align="left">
      <colspec colname="attribute" align="left" />
      <colspec colname="required" />
      <colspec colname="comment" />
      <thead>
        <row>
          <entry>Attribute</entry>
          <entry>Required</entry>
          <entry>Comment</entry>
        </row>
      </thead>
      <tbody>
        <row>
          <entry>*</entry>
          <entry></entry>
          <entry>
            All attributes defined by the 
            <sgmltag class="starttag">property</sgmltag> tag in
            <filename>extended-properties.xml</filename>. See 
            <xref linkend="appendix.extendedproperties.property" />.
          </entry>
        </row>
        <row>
          <entry>channels</entry>
          <entry>no</entry>
          <entry>
            The channel number the property belongs to. Allowed values are 0 to
            the number of channels specified for the raw data type. If the property
            doesn't belong to any channels set the value to 0 or leave it
            unspecified.
          </entry>
        </row>
      </tbody>
    </tgroup>
    </table>
    
    <para>
      Following the <sgmltag class="starttag">property</sgmltag> tags comes 0
      or more <sgmltag class="starttag">intensity-formula</sgmltag> tags.
      Each one defines mathematical formulas that can be used to
      calculate the intensity values from the raw data. In the Genepix case,
      there are several formulas which differs in the way background is 
      subtracted from foreground intensity values. For other raw data
      types, the intensity formula may just copy one of the raw data values.
    </para>
    
    <para>
      The intensity formulas are installed as <classname 
      docapi="net.sf.basedb.core">Formula</classname> items in the database. This
      means that you can manually add, change or remove intensity formulas directly 
      from the web interface. The intensity formulas in the <filename>raw-data-types.xml</filename>
      file are only used at installation time.
    </para>
    
    <para>
      The <sgmltag class="starttag">intensity-formula</sgmltag> tag has the following
      attributes:
    </para>
    
    <table frame="all" id="appendix.rawdatatypes.intensity-formula">
    <title>Attributes for the <sgmltag class="starttag">intensity-formula</sgmltag> tag</title>
    <tgroup cols="3" align="left">
      <colspec colname="attribute" align="left" />
      <colspec colname="required" />
      <colspec colname="comment" />
      <thead>
        <row>
          <entry>Attribute</entry>
          <entry>Required</entry>
          <entry>Comment</entry>
        </row>
      </thead>
      <tbody>
        <row>
          <entry>name</entry>
          <entry>yes</entry>
          <entry>
            A unique name for the formula. This is only used during installation.
          </entry>
        </row>
        <row>
          <entry>title</entry>
          <entry>yes</entry>
          <entry>
            The title of the formula. This is used by client applications for
            display.
          </entry>
        </row>
        <row>
          <entry>description</entry>
          <entry>no</entry>
          <entry>
            An optional, longer, description of the formula.
          </entry>
        </row>
      </tbody>
    </tgroup>
    </table>
    
    <para>
      The <sgmltag class="starttag">intensity-formula</sgmltag> must contain
      one <sgmltag class="starttag">formula</sgmltag> tag for each channel
      of the raw data type. The attributes of this tag are:
    </para>
    
    <table frame="all" id="appendix.rawdatatypes.formula">
    <title>Attributes for the <sgmltag class="starttag">formula</sgmltag> tag</title>
    <tgroup cols="3" align="left">
      <colspec colname="attribute" align="left" />
      <colspec colname="required" />
      <colspec colname="comment" />
      <thead>
        <row>
          <entry>Attribute</entry>
          <entry>Required</entry>
          <entry>Comment</entry>
        </row>
      </thead>
      <tbody>
        <row>
          <entry>channel</entry>
          <entry>yes</entry>
          <entry>
            The channel number. One tag for each channel must be specified. No
            duplicates are allowed.
          </entry>
        </row>
        <row>
          <entry>expression</entry>
          <entry>yes</entry>
          <entry>
            The mathematical expression used to calculate the intensities.
            The expression is parsed with the <classname docapi="net.sf.basedb.util.jep">Jep</classname>
            parser. It supports the common mathematical operations such as +, -, *, /,
            some mathematical function like, log2(), ln(), sqrt(), etc. See the API
            documentation for Jep for more information. You can also use two special
            function developed specifically for this case:
            <itemizedlist>
            <listitem>
              <para>
              raw(name): Get the value from the raw data property with the given name,
              for example: <code>raw('ch1FgMedian')</code>.
              </para>
            </listitem>
            <listitem>
              <para>
              mean(name): Get the mean value of the raw data property with the given name,
              for example: <code>mean('ch1BgMean')</code>. The mean is calculated from
              all raw data spots in the raw bioassay.
              </para>
            </listitem>
            </itemizedlist>
          </entry>
        </row>
      </tbody>
    </tgroup>
    </table>
    
  </sect1>

</appendix>