Changeset 4633


Ignore:
Timestamp:
Nov 10, 2008, 4:35:27 PM (15 years ago)
Author:
Martin Svensson
Message:

Fixes #1151 User role-test can now test batch importers. Documentation is updated(still in old format). Also updated build.xml to copy the test/role-directory when creating dist, links from wiki will work after next release.

Location:
trunk
Files:
3 edited

Legend:

Unmodified
Added
Removed

  • trunk/build.xml

    r4537 r4633  
    14081408      <fileset dir="doc" includes="historical/development/**/*" />
    14091409      <fileset dir="doc" includes="historical/specifications/**/*" />
     1410      <fileset dir="doc" includes="test/roles/**/*" />
    14101411      <!--fileset dir="doc" includes="user/**/*" /-->
    14111412      <fileset dir="doc" includes="*.*" />

  • trunk/doc/test/roles/index.html

    r4626 r4633  
    2424  <head>
    2525    <title>BASE - Test procedures for predefined roles</title>
    26   <link rel=stylesheet type="text/css" href="../../styles.css">
     26  <link rel=stylesheet type="text/css" href="../../historical/styles.css">
    2727  </head>
    2828<body>
    … …  
    158158    <li>Change to the <code>build/test/</code> directory.</li>
    159159
    160     <li>Run test programs: <code>./test.sh roles &lt;cmds&gt;</code> where
     160    <li>Run test programs: <code>./test.sh roles [OPTION] &lt;cmds&gt;</code> where
    161161      <code>&lt;cmds&gt;</code> is one or more of the following:
    162162      <ul>
    … …  
    167167      <li><code>user</code>: run the regular user tests</li>
    168168      <li><code>guest</code>: run the guest user tests</li>
     169      </ul>
     170      and <code>OPTION</code> can be none or more of:
     171      <ul>
     172      <li><code>-b</code>: if the batch importers should be tested</li>
     173      <li><code>-w</code>: if the test program should wait for ENTER before emptying the database.</li>
    169174      </ul>
    170175    </li>
    … …  
    10761081    </table>
    10771082    <p>
     1083    Or
     1084    <p>
     1085    Import array designs with the ArrayDesignImporter plug-in.
     1086    <ol>
     1087      <li>Make sure the data-files, mentioned in table above, are located in <code>/home/power/</code>, upload them if not</li>
     1088      <li>Click on the <code>Import</code> button on the array design list page.</li>
     1089      <li>Choose ArrayDesignImporter in the plug-in drop-down list.</li>
     1090      <li><b>Test with file:</b> <code>arraydesign_out.txt</code> and set
     1091        the parsing parameters with help of the <code>Auto generate</code> button on the <code>Column mapping</code>
     1092        tab.</li>
     1093      <li>Start the import-job by clicking on the <code>Finish</code> button on the third wizard-page.</li>     
     1094    </ol>
     1095    <p>
    10781096  </li>
    10791097 
    … …  
    11871205    </table>
    11881206    <p>
     1207    Or
     1208    <p>
     1209    Import array batches with the ArrayBatchImporter plug-in.
     1210    <ol>
     1211      <li>Click on the <code>Import</code> button on the array batch list page.</li>
     1212      <li>Select ArrayBatchImporter in the plug-in drop-down list and click <code>Next</code></li>
     1213      <li><b>Test with file:</b> <code>arraybatch_out.txt</code> and set  the parsing parameters
     1214        with help of the <code>Auto generate</code> button on the <code>Column mapping</code>
     1215        tab.</li>
     1216      <li>Start the import-job by clicking on the <code>Finish</code> button on the third wizzard-page.</li>     
     1217    </ol>
     1218    <p>   
    11891219  </li>
    11901220
    … …  
    12101240    </tr>
    12111241    </table>
     1242    <p>
     1243    Or
     1244    <p>
     1245    Import array slides with the ArraySlideImporter plug-in.
     1246    <ol>
     1247      <li>Click on the <code>Import</code> button on the array slide list page.</li>
     1248      <li>Select ArraySlideImporter in the plug-in drop-down list and click <code>Next</code></li>
     1249      <li><b>Test with file:</b> <code>arrayslide_out.txt</code> and set  the parsing parameters
     1250        with help of the <code>Auto generate</code> button on the <code>Column mapping</code>
     1251        tab.</li>
     1252      <li>Start the import-job by clicking on the <code>Finish</code> button on the third wizzard-page.</li>     
     1253    </ol>
    12121254    <p>
    12131255  </li>
    … …  
    12191261    The user is a typical worker in the project. The user does the actual experimentation in the
    12201262    lab, which includes collecting samples, doing extraction, labeling and hybridizations.
    1221     The user also scans and analyses the raw data resulting from the images.
     1263    The user also scans and analyses the raw data resulting from the images. Inserting items can be
     1264    done in two different ways, .
    12221265  </p>
    12231266 
    1224   <ol>
    1225   <li>
    1226     Activate the <code>Project A</code> project<p></li>
    1227    
     1267  <p>
     1268    First activate the <code>Project A</code> project and then do one of following proceedures.
     1269  </p>
     1270  <p>
     1271    Use the batch importers for each item type.
     1272    <ol>
     1273      <li>Click on the <code>Import</code> button on the list page.</li>
     1274      <li>Select &lt;itemtype&gt;Importer in the plug-in drop-down list and click <code>Next</code></li>
     1275      <li><b>Test with file:</b> using the right file(listed below) and set the parsing parameters
     1276        by using the <code>Auto generate</code> button on the <code>Column mapping</code> tab.
     1277        <table class="listing" cellspacing="0" cellpadding="2" border="0">
     1278        <tr>
     1279          <th>Itemtype</th>
     1280          <th>File</th>
     1281        </tr>
     1282        <tr>
     1283          <td>Biosource</td>
     1284          <td>biosource_out.txt</td>       
     1285        </tr>
     1286        <tr>
     1287          <td>Samples</td>
     1288          <td>sample_out.txt</td>
     1289        </tr>
     1290        <tr>
     1291          <td>Extracts</td>
     1292          <td>extract_out.txt</td>
     1293        </tr>
     1294        <tr>
     1295          <td>Labeled extracts</td>
     1296          <td>labeledextract_out.txt</td>
     1297        </tr>
     1298        <tr>
     1299          <td>Hybridizations</td>
     1300          <td>hybridization_out.txt</td>
     1301        </tr>
     1302        <tr>
     1303          <td>Scans</td>
     1304          <td>scan_out.txt</td>
     1305        </tr>
     1306        <tr>
     1307          <td>Raw bioassays</td>
     1308          <td>rawbioassay_out.txt</td>
     1309        </tr>
     1310        </table>
     1311        The files listed for biosource and samples also contains annotations
     1312        for the items and these files should also be used with the annotation importer.
     1313        The procedure is the same as for batch importers except that only <code>\\Name\\</code>
     1314        is needed in the column mapping. The annotation column should be selected by default in
     1315        the second wizzard-step.       
     1316      </li>
     1317      <li>Start the import-job by clicking on the <code>Finish</code> button on the third wizzard-page.</li>
     1318      <li>Then <a href="#inheritannotations">go to the step</a> in the next list where inherited annotations are set to the raw data
     1319        and proceed from there.</li>     
     1320    </ol>
     1321  </p>
     1322  <p>
     1323  Inserting each item manually
     1324  <ol> 
    12281325  <li>
    12291326    Create a biosource:
    … …  
    15931690  </li>
    15941691 
    1595   <li>
     1692  <li><a name="inheritannotations"></a>
    15961693    Inherit the annotations from the samples and biosource for each raw bioassay.
    15971694    <p>
    … …  
    17041801  </li>
    17051802 
     1803  </ol>
     1804  </li>
    17061805  </ol>
    17071806

  • trunk/src/test/net/sf/basedb/test/roles/UserTest.java

    r4628 r4633  
    3737import net.sf.basedb.core.FileType;
    3838import net.sf.basedb.core.Hybridization;
     39import net.sf.basedb.core.Include;
    3940import net.sf.basedb.core.Item;
    4041import net.sf.basedb.core.ItemParameterType;
     42import net.sf.basedb.core.ItemQuery;
    4143import net.sf.basedb.core.Job;
    4244import net.sf.basedb.core.Label;
    … …  
    5557import net.sf.basedb.core.SystemItems;
    5658import net.sf.basedb.core.Unit;
     59import net.sf.basedb.core.query.Hql;
     60import net.sf.basedb.core.query.Orders;
    5761import net.sf.basedb.test.FileUtil;
    5862import net.sf.basedb.test.PluginUtil;
    5963import net.sf.basedb.test.TestUtil;
     64
     65import java.util.ArrayList;
     66import java.util.List;
     67import java.util.Map;
     68import java.util.TreeMap;
     69
     70import edu.emory.mathcs.backport.java.util.Arrays;
    6071
    6172/**
    … …  
    90101      PluginDefinition scanBatchImporter = null;
    91102      PluginDefinition rawBioAssayBatchImporter = null;
     103      PluginDefinition annotationImporter = null;
    92104     
    93105      File batchImportFile_BioSource = null;
    … …  
    117129        scanBatchImporter = PluginDefinition.getByClassName(dc, "net.sf.basedb.plugins.batchimport.ScanImporter");
    118130        rawBioAssayBatchImporter = PluginDefinition.getByClassName(dc, "net.sf.basedb.plugins.batchimport.RawBioAssayImporter");
     131        annotationImporter = PluginDefinition.getByClassName(dc, "net.sf.basedb.plugins.AnnotationFlatFileImporter");
    119132       
    120133        batchImportFile_BioSource = FileUtil.uploadFile(dc, "/batchimport", "biosource_out.txt", null);
    … …  
    159172      RawBioAssay affyRaw2 = null;
    160173      RawBioAssay affyRaw3 = null;
     174     
    161175      if (!useBatchImporters)
    162176      {
    … …  
    229243      {
    230244        // Import biomaterials
     245        dc = TestUtil.getDbControl();
    231246        importBioSources(bioSourceBatchImporter, batchImportFile_BioSource);
     247        importAnnotations(annotationImporter, batchImportFile_BioSource, Item.BIOSOURCE, getBioSourceColumnHeaders(), "\\t");
     248       
    232249        importSamples(sampleBatchImporter, batchImportFile_Sample);
     250        importAnnotations(annotationImporter, batchImportFile_Sample, Item.SAMPLE, getSampleColumnHeaders(), "\\t");
     251       
    233252        importExtracts(extractBatchImporter, batchImportFile_Extract);
    234253        importLabelExtracts(labeledExtractBatchImporter, batchImportFile_LabeledExtract);
    … …  
    238257        importScans(scanBatchImporter, batchImportFile_Scan);
    239258        importRawBioAssays(rawBioAssayBatchImporter, batchImportFile_RawBioAssay);
    240         dc = TestUtil.getDbControl();
    241259        bioSource = Util.findBioSource(dc, "Biosource A");
    242260        s1 = Util.findSample(dc, "Sample A.00h");
    243261        s2 = Util.findSample(dc, "Sample A.24h");
    244262        sRef = Util.findSample(dc, "Sample A.ref");
    245         rba1 = Util.findRawBioAssay(dc, "Raw bioassay A.00h");
    246         rba2 = Util.findRawBioAssay(dc, "Raw bioassay A.24h");
    247         rba1DyeSwap = Util.findRawBioAssay(dc, "Raw bioassay A.24h");
    248         rba2DyeSwap = Util.findRawBioAssay(dc, "Raw bioassay A.24h (dye-swap)");
    249         affyRaw1 = Util.findRawBioAssay(dc, "Affymetrix raw A.1");
    250         affyRaw2 = Util.findRawBioAssay(dc, "Affymetrix raw A.2");
    251         affyRaw3 = Util.findRawBioAssay(dc, "Affymetrix raw A.3");
     263       
    252264        dc.close();
    253265      }
    254      
    255       dc = TestUtil.getDbControl();
     266      dc = TestUtil.getDbControl();
     267      rba1 = Util.findRawBioAssay(dc, "Raw bioassay A.00h");
     268      rba2 = Util.findRawBioAssay(dc, "Raw bioassay A.24h");
     269      rba1DyeSwap = Util.findRawBioAssay(dc, "Raw bioassay A.00h (dye-swap)");
     270      rba2DyeSwap = Util.findRawBioAssay(dc, "Raw bioassay A.24h (dye-swap)");
     271      affyRaw1 = Util.findRawBioAssay(dc, "Affymetrix raw A.1");
     272      affyRaw2 = Util.findRawBioAssay(dc, "Affymetrix raw A.2");
     273      affyRaw3 = Util.findRawBioAssay(dc, "Affymetrix raw A.3");
    256274      inheritAnnotations(dc, rba1, s1, bioSource);
    257275      inheritAnnotations(dc, rba2, s2, bioSource);
    … …  
    584602      job.setParameterValue("dataSplitterRegexp", new StringParameterType(), "\\t");
    585603      job.setParameterValue("dataHeaderRegexp", new StringParameterType(),
    586           "Name\\tArray slide\\tDescription\\tHardware\\tArrays/slide\\tProtocol\\tParent\\tParent used quantity");
     604          "Name\\tArray slide\\tDescription\\tHardware\\tArrays/slide\\tProtocol\\tParent\\tParent used quantity\\tCreated");
    587605     
    588606      job.setParameterValue("nameColumnMapping", new StringParameterType(), "\\Name\\");
    … …  
    594612      job.setParameterValue("arraySlideColumnMapping", new StringParameterType(), "\\Array slide\\");
    595613      job.setParameterValue("numArraysColumnMapping", new StringParameterType(), "\\Arrays/slide\\");
     614      job.setParameterValue("createdColumnMapping", new StringParameterType(), "\\Created\\");
    596615     
    597616      dc.saveItem(job);
    … …  
    692711      job.setParameterValue("dataSplitterRegexp", new StringParameterType(), "\\t");
    693712      job.setParameterValue("dataHeaderRegexp", new StringParameterType(),
    694           "Name\\tOriginal quantity\\tDescription\\tExternal id\\tCreated\\tPooled\\tBiosource\\tProtocol");
     713          "Name\\tOriginal quantity\\tDescription\\tExternal id\\tCreated\\tPooled\\tBiosource\\tProtocol\\tTime");
    695714     
    696715      job.setParameterValue("nameColumnMapping", new StringParameterType(), "\\Name\\");
    … …  
    726745      job.setParameterValue("file", new FileParameterType(), file);
    727746      job.setParameterValue("dataSplitterRegexp", new StringParameterType(), "\\t");
    728       job.setParameterValue("dataHeaderRegexp", new StringParameterType(), "Name\\tDescription\\tExternal id");
     747      job.setParameterValue("dataHeaderRegexp", new StringParameterType(), "Name\\tDescription\\tExternal id\\tDrug resistance");
    729748     
    730749      job.setParameterValue("nameColumnMapping", new StringParameterType(), "\\Name\\");
    … …  
    741760    }   
    742761  }
     762 
     763  public static void importAnnotations(PluginDefinition annotationImporter, File file, Item item, String[] columnHeaders, String dataSplitter)
     764  {
     765    TestUtil.write("--Importing annotations for " + item.name() + " from file: " + file + "\n");
     766    DbControl dc = null;
     767    StringBuilder headerString = new StringBuilder();
     768    try
     769    {
     770      dc = TestUtil.getDbControl();
     771      PluginDefinition plugin = PluginDefinition.getById(dc, annotationImporter.getId());
     772      Job job = Job.getNew(dc, plugin, null, null);
     773      job.setName("Batch import of annotations for " + item.name());
     774     
     775      // Get annotation types for the current item
     776      ItemQuery<AnnotationType> query = AnnotationType.getQuery(item);
     777      query.include(Include.MINE, Include.SHARED, Include.IN_PROJECT, Include.OTHERS);
     778      query.order(Orders.asc(Hql.property("name")));
     779      List<AnnotationType> annotationTypes;
     780      Map<String, AnnotationType> names =
     781        new TreeMap<String, AnnotationType>(String.CASE_INSENSITIVE_ORDER);
     782     
     783      annotationTypes = new ArrayList<AnnotationType>(query.list(dc));
     784      for (AnnotationType at : annotationTypes)
     785      {
     786        names.put(at.getName(), at);
     787      }
     788      for (int i=0; i<columnHeaders.length; i++)
     789      {
     790        String header = columnHeaders[i];
     791        headerString.append(i>0 ? "\\t" : "");
     792        headerString.append(header);
     793        if (names.get(header) != null)
     794        {
     795          job.setParameterValue("columnMapping." + i,
     796              new ItemParameterType<AnnotationType>(AnnotationType.class, annotationTypes),
     797              names.get(header));
     798        }
     799      }
     800      job.setParameterValue("file", new FileParameterType(), file);
     801      job.setParameterValue("dataSplitterRegexp", new StringParameterType(), dataSplitter);
     802      job.setParameterValue("dataHeaderRegexp", new StringParameterType(), headerString.toString());
     803      job.setParameterValue("nameColumnMapping", new StringParameterType(), "\\Name\\");
     804      job.setParameterValue("itemType", new StringParameterType(), item.name());
     805     
     806      dc.saveItem(job);
     807      dc.commit();
     808      PluginUtil.executeJob(job);
     809    }
     810    finally
     811    {
     812      if (dc != null ) dc.close();
     813    }
     814  }
     815 
     816  private static String[] getBioSourceColumnHeaders()
     817  {
     818    return new String[] {"Name", "Description", "External id", "Drug resistance"};
     819  }
     820 
     821  private static String[] getSampleColumnHeaders()
     822  {
     823    return new String[] {"Name", "Original quantity", "Description", "External id", "Created", "Pooled", "Biosource", "Protocol", "Time"};
     824  }
    743825}
Note: See TracChangeset for help on using the changeset viewer.