Changeset 4002

Timestamp:

Nov 26, 2007, 1:36:38 PM (16 years ago)

Author:

Nicklas Nordborg

Message:

References #596

File:

• trunk/doc/src/docbook/appendix/raw_data_types.xml (modified) (2 diffs)

trunk/doc/src/docbook/appendix/raw_data_types.xml

@@ -31 +31 @@
   
   <para>
-    Raw data can be stored either as files attached to items or in
-    the database. 
-    The <classname docapi="net.sf.basedb.core">Platform</classname> item has information
-    about this. Configuration information for the database tables
-    and columns used to store raw data in the database is found in the
-    <filename>raw-data-types.xml</filename> file. For detailed information
-    see <xref linkend="core_api.data_in_files" />.
+    Raw data can be stored either as files attached to items and/or in
+    the database. The <classname docapi="net.sf.basedb.core">Platform</classname> 
+    item has information  about this.  For more information see 
+    <xref linkend="core_api.data_in_files" />.
   </para>
   
@@ -113 +110 @@
     
     <para>
-      TODO
-    </para>
+      A given platform either supports importing data to the database or it
+      doesn't. If it supports import, it may be locked to specific raw data type
+      or it may use any raw data type. Among the default platforms installed with
+      BASE, the Affymetrix platform doesn't support importing data. The Generic platform
+      supports importing to any raw data type.
+    </para>
+    
+    <para>
+      Raw data types are defined in the <filename>raw-data-types.xml</filename>
+      file. This file is located in the <filename>&lt;basedir&gt;/www/WEB-INF/classes</filename>
+      directory and contains information about the database tables and columns to
+      use for storing raw data. BASE ships with default raw data types for many
+      different microarray platforms, including Genepix, Agilent and Illumina.
+    </para>
+    
+    <para>
+      If you want your BASE installation to be configured differently we recommend that 
+      you do it before the first initialisation of the database.
+      It is possible to change the configuration of an existing BASE installation but it
+      requires manual updates to the database. Follow this procedure:
+    </para>
+
+  <orderedlist>
+  <listitem>
+    <para>
+    Shut down the BASE web server. If you have installed job agents you should shut
+    down them as well.
+    </para>
+  </listitem>
+  
+  <listitem>
+    <para>
+    Modify the <filename>raw-data-types.xml</filename> file. If you have installed
+    job agents, make sure they all have the same version as the web server.
+    </para>
+  </listitem>
+  
+  <listitem>
+    <para>
+    Run the <filename>updatedb.sh</filename> script. Tables for new raw data types
+    and new columns for existing raw data types automatically be created, but the script 
+    can't delete tables or columns that have been removed, or modify columns that have 
+    changed datatype. You will have to do these kind of changes by manually executing
+    SQL against your database. Check your database documentation for information about SQL syntax.
+    </para>
+    
+    <tip>
+      <title>Create a parallell installation</title>
+      <para>
+      You can always create a new temporary parallell installation to check 
+      what the table generated by installation script looks like. Compare the 
+      new table to the existing one and make sure they match.
+      </para>
+    </tip>
+  </listitem>
+  
+  <listitem>
+    <para>
+    Start up the BASE web server and job agents, if any, again.
+    </para>
+  </listitem>
+  </orderedlist>
+
+  <tip>
+    <title>Start with few columns</title>
+    <para>
+    It is better to start with too few columns, since it is easier to add
+    more columns than it is to remove columns that are not needed.
+    </para>
+  </tip>
+
+  <bridgehead>Format of the raw-data-types.xml file</bridgehead>
+  <para>
+    The <filename>raw-data-types.xml</filename> is an XML file.
+    The following example will serve as a description of the format:
+  </para>
+  
+  
+  <programlisting language="xml">
+<![CDATA[
+<?xml version="1.0" ?>
+<?xml-stylesheet type="text/xsl" href="raw-data-types.xsl"?>
+<!DOCTYPE raw-data-types SYSTEM "raw-data-types.dtd" >
+<raw-data-types>
+   <raw-data-type
+      id="genepix"
+      name="GenePix"
+      channels="2"
+      table="RawDataGenePix"
+      >
+      <property
+         name="diameter"
+         title="Spot diameter"
+         description="The diameter of the spot in µm"
+         column="diameter"
+         type="float"
+      />
+      <property
+         name="ch1FgMedian"
+         title="Channel 1 foreground median"
+         description="The median of the foreground intensity in channel 1"
+         column="ch1_fg_median"
+         type="float"
+         channel="1"
+      />
+      <!-- skipped a lot of properties -->
+      <intensity-formula
+         name="mean"
+         title="Mean FG - Mean BG"
+         description="Subtract mean background from mean foreground"
+         >
+         <formula 
+            channel="1"
+            expression="raw('ch1FgMean') - raw('ch1BgMean')"
+         />
+         <formula 
+            channel="2"
+            expression="raw('ch2FgMean') - raw('ch2BgMean')"
+         />
+      </intensity-formula>
+      <!-- and a few more... --->
+   </raw-data-type>
+</raw-data-types>
+]]> 
+</programlisting>
+  
+  <para>
+    Each raw data type is represented by a <sgmltag class="starttag">raw-data-type</sgmltag> 
+    tag. The following attributes can be used:
+  </para>
+  
+    <table frame="all" id="appendix.rawdatatypes.tag">
+    <title>Attributes for the <sgmltag class="starttag">raw-data-type</sgmltag> tag</title>
+    <tgroup cols="3" align="left">
+      <colspec colname="attribute" align="left" />
+      <colspec colname="required" />
+      <colspec colname="comment" />
+      <thead>
+        <row>
+          <entry>Attribute</entry>
+          <entry>Required</entry>
+          <entry>Comment</entry>
+        </row>
+      </thead>
+      <tbody>
+        <row>
+          <entry>id</entry>
+          <entry>yes</entry>
+          <entry>
+            A unique ID of the raw data type. It should contain only letters,
+            numbers and underscores and the first character must be a letter.
+          </entry>
+        </row>
+        <row>
+          <entry>name</entry>
+          <entry>yes</entry>
+          <entry>
+            A unique name of the raw data type. The name is usually used by client
+            applications for disaplay.
+          </entry>
+        </row>
+        <row>
+          <entry>table</entry>
+          <entry>yes</entry>
+          <entry>
+            The name of the database table to store data in. The table name
+            must be unique and can only contain letters,
+            numbers and underscores. The first character must be a letter.
+          </entry>
+        </row>
+        <row>
+          <entry>channels</entry>
+          <entry>yes</entry>
+          <entry>
+            The number of channels used by this raw data type. It must be 
+            a number &gt; 0.
+          </entry>
+        </row>
+        <row>
+          <entry>description</entry>
+          <entry>no</entry>
+          <entry>
+            An optional (longer) description of the raw data type.
+          </entry>
+        </row>
+      </tbody>
+    </tgroup>
+    </table>
+    
+    <para>
+      Following the <sgmltag class="starttag">raw-data-type</sgmltag> tag
+      is one or more  <sgmltag class="starttag">property</sgmltag> tags.
+      Each one defines a column in the database that is designed to hold 
+      data values of a particular type. The following attributes can be used
+      on this tag:
+    </para>
+  
+    <table frame="all" id="appendix.rawdatatypes.property">
+    <title>Attributes for the <sgmltag class="starttag">property</sgmltag> tag</title>
+    <tgroup cols="3" align="left">
+      <colspec colname="attribute" align="left" />
+      <colspec colname="required" />
+      <colspec colname="comment" />
+      <thead>
+        <row>
+          <entry>Attribute</entry>
+          <entry>Required</entry>
+          <entry>Comment</entry>
+        </row>
+      </thead>
+      <tbody>
+        <row>
+          <entry>*</entry>
+          <entry></entry>
+          <entry>
+            All attributes defined by the 
+            <sgmltag class="starttag">property</sgmltag> tag in
+            <filename>extended-properties.xml</filename>. See 
+            <xref linkend="appendix.extendedproperties.property" />.
+          </entry>
+        </row>
+        <row>
+          <entry>channels</entry>
+          <entry>no</entry>
+          <entry>
+            The channel number the property belongs to. Allowed values are 0 to
+            the number of channels specified for the raw data type. If the property
+            doesn't belong to any channels set the value to 0 or leave it
+            unspecified.
+          </entry>
+        </row>
+      </tbody>
+    </tgroup>
+    </table>
+    
+    <para>
+      Following the <sgmltag class="starttag">property</sgmltag> tags comes 0
+      or more <sgmltag class="starttag">intensity-formula</sgmltag> tags.
+      Each one defines mathematical formulas that can be used to create
+      calculate the intensity values from the raw data. In the Genepix, case
+      there are several formulas which differs in the way background is 
+      subtracted from foregorund intensity values. For other raw data
+      types, the intensity formula may just copy one of the raw data values.
+    </para>
+    
+    <para>
+      The intensity formulas are installed as <classname 
+      docapi="net.sf.basedb.core">Formula</classname> items in the database. This
+      means that you can manually add, change or remove intensity formulas directly 
+      from the web interface. The intensity formulas in the <filename>raw-data-types.xml</filename>
+      file are only used at installation time.
+    </para>
+    
+    <para>
+      The <sgmltag class="starttag">intensity-formula</sgmltag> tag has the following
+      attributes:
+    </para>
+    
+    <table frame="all" id="appendix.rawdatatypes.intensity-formula">
+    <title>Attributes for the <sgmltag class="starttag">intensity-formula</sgmltag> tag</title>
+    <tgroup cols="3" align="left">
+      <colspec colname="attribute" align="left" />
+      <colspec colname="required" />
+      <colspec colname="comment" />
+      <thead>
+        <row>
+          <entry>Attribute</entry>
+          <entry>Required</entry>
+          <entry>Comment</entry>
+        </row>
+      </thead>
+      <tbody>
+        <row>
+          <entry>name</entry>
+          <entry>yes</entry>
+          <entry>
+            A unique name for the formula. This is only used during installation.
+          </entry>
+        </row>
+        <row>
+          <entry>title</entry>
+          <entry>yes</entry>
+          <entry>
+            The title of the formula. This is used by client applications for
+            display.
+          </entry>
+        </row>
+        <row>
+          <entry>description</entry>
+          <entry>no</entry>
+          <entry>
+            An optional, longer, description of the formula.
+          </entry>
+        </row>
+      </tbody>
+    </tgroup>
+    </table>
+    
+    <para>
+      The <sgmltag class="starttag">intensity-formula</sgmltag> must contain
+      one <sgmltag class="starttag">formula</sgmltag> tag for each channel
+      of the raw data type. The attributes of this tag are:
+    </para>
+    
+    <table frame="all" id="appendix.rawdatatypes.formula">
+    <title>Attributes for the <sgmltag class="starttag">formula</sgmltag> tag</title>
+    <tgroup cols="3" align="left">
+      <colspec colname="attribute" align="left" />
+      <colspec colname="required" />
+      <colspec colname="comment" />
+      <thead>
+        <row>
+          <entry>Attribute</entry>
+          <entry>Required</entry>
+          <entry>Comment</entry>
+        </row>
+      </thead>
+      <tbody>
+        <row>
+          <entry>channel</entry>
+          <entry>yes</entry>
+          <entry>
+            The channel number. One tag for each channel must be specified. No
+            duplicates are allowed.
+          </entry>
+        </row>
+        <row>
+          <entry>expression</entry>
+          <entry>yes</entry>
+          <entry>
+            The mathematical expression used to calculate the intensities.
+            The expression is parsed with the <classname docapi="net.sf.basedb.util.jep">Jep</classname>
+            parser. It supports the common mathematical operations such as +, -, *, /,
+            some mathematical function like, log2(), ln(), sqrt(), etc. See the API
+            documentation for Jep for more information. You can also use two special
+            function developed specifically for this case:
+            <itemizedlist>
+            <listitem>
+              <para>
+              raw(name): Get the value from the raw data property with the given name,
+              for example: <code>raw('ch1FgMedian')</code>.
+              </para>
+            </listitem>
+            <listitem>
+              <para>
+              mean(name): Get the mean value of the raw data property with the given name,
+              for example: <code>mean('ch1BgMean')</code>. The mean is calculated from
+              all raw data spots in the raw bioassay.
+              </para>
+            </listitem>
+            </itemizedlist>
+          </entry>
+        </row>
+      </tbody>
+    </tgroup>
+    </table>
     
   </sect1>