Changeset 4083

Timestamp:

Jan 17, 2008, 11:08:21 AM (14 years ago)

Author:

Nicklas Nordborg

Message:

References #894: Support for array designs with other feature identification methods than coordinates

Everything should now be fixed. May need some more testing before closing this ticket.

Location:

trunk

Files:

• doc/src/docbook/appendix/incompatible.xml (modified) (2 diffs)
• doc/src/docbook/userdoc/rawbioassays.xml (modified) (1 diff)
• src/clients/migrate/net/sf/basedb/clients/migrate/RawBioAssayDataTransfer.java (modified) (1 diff)
• src/core/net/sf/basedb/core/FeatureBatcher.java (modified) (2 diffs)
• src/core/net/sf/basedb/core/FeatureIdentificationMethod.java (modified) (1 diff)
• src/core/net/sf/basedb/core/RawBioAssay.java (modified) (15 diffs)
• src/core/net/sf/basedb/core/RawDataBatcher.java (modified) (5 diffs)
• src/core/net/sf/basedb/core/data/ArrayDesignData.java (modified) (1 diff)
• src/plugins/core/net/sf/basedb/plugins/IlluminaRawDataImporter.java (modified) (11 diffs)
• src/plugins/core/net/sf/basedb/plugins/RawDataFlatFileImporter.java (modified) (8 diffs)
• src/plugins/core/net/sf/basedb/plugins/ReporterMapFlatFileImporter.java (modified) (5 diffs)
• src/test/TestAnalyzePluginUtil.java (modified) (1 diff)
• src/test/TestArrayDesign.java (modified) (6 diffs)
• src/test/TestBioAsssaySetExporter.java (modified) (1 diff)
• src/test/TestRawBioAssay.java (modified) (2 diffs)
• src/test/net/sf/basedb/test/merge/MergeTest.java (modified) (1 diff)
• www/lims/arraydesigns/features/index.jsp (modified) (1 diff)
• www/lims/arraydesigns/features/list_features.jsp (modified) (5 diffs)
• www/lims/arraydesigns/features/view_feature.jsp (modified) (3 diffs)
• www/lims/arraydesigns/list_designs.jsp (modified) (4 diffs)
• www/lims/arraydesigns/view_design.jsp (modified) (1 diff)
• www/views/rawbioassays/edit_rawbioassay.jsp (modified) (3 diffs)
• www/views/rawbioassays/index.jsp (modified) (9 diffs)
• www/views/rawbioassays/rawdata/list_rawdata.jsp (modified) (4 diffs)

trunk/doc/src/docbook/appendix/incompatible.xml

@@ -52 +52 @@
     <para>
       Array design features can now be identified by three different methods:
-      COORDINATES, POSITION and EXTERNAL_ID. The coordinates method was the
+      COORDINATES, POSITION and FEATURE_ID. The coordinates method was the
       only one supported earlier. 
     </para>
@@ -64 +64 @@
       <classname docapi="net.sf.basedb.core">FeatureBatcher</classname> or 
       <classname docapi="net.sf.basedb.core">RawDataBatcher</classname> are 
-      probably affected by this. For example, includes raw data importers which 
-      doesn't know how to set the position or the feature ID when those identification
-      methods are used.
+      probably affected by this. For example, a raw data importer which 
+      doesn't know how to set the position or the feature ID can't import data to
+      an array design that is using one of the new identification methods.
       </para>
     </listitem>

trunk/doc/src/docbook/userdoc/rawbioassays.xml

@@ -138 +138 @@
             The array design can be changed after raw data has been 
             imported, but this triggers a new validation. If the raw data
-            is stored in the database, the position, coordinates and reporter of all features
-            on the new array design must exactly match the position,
-            coordinates and reporter of the raw data. For Affymetrix data, the
+            is stored in the database, the features on the new array design must 
+            match the the raw data. The verification can use three different methods:
+          </para>
+          
+          <itemizedlist>
+          <listitem>
+            <para>
+            Coordinates: Verify block, meta-grid, row and column coordinates.
+            </para>
+          </listitem>
+          <listitem>
+            <para>Position: Verify the position number.</para>
+          </listitem>
+          <listitem>
+            <para>
+            Feature ID: Verify the feature ID. This option can only be used
+            if the raw bioassay is currently connected to an array design that
+            has feature ID values already.
+            </para>
+          </listitem>
+          </itemizedlist>
+          <para>
+            In all three cases it is also verified that the reporter of the raw
+            data matches the reporter of the features.
+          </para>
+
+          <para>
+            For Affymetrix data, the
             CEL file is validated against the CDF file of the new array design.
             If the validation fails, the array design is not changed.

trunk/src/clients/migrate/net/sf/basedb/clients/migrate/RawBioAssayDataTransfer.java

@@ -168 +168 @@
      */
     cleanUpMemory();
-    RawDataBatcher batcher = rawBioAssay.getRawDataBatcher();
+    RawDataBatcher batcher = rawBioAssay.getRawDataBatcher(null);
     batcher.setBatchSize(getBatchSize());
     runRawBioAssay(rawBioAssay, batcher, hasArrayDesign, rawBioAssayId);

trunk/src/core/net/sf/basedb/core/FeatureBatcher.java

@@ -54 +54 @@
   <li>COORDINATES: {@link FeatureData#setRow(int)} and {@link FeatureData#setColumn(int)}
   <li>POSITION: {@link FeatureData#setPosition(int)}
-  <li>EXTERNAL_ID: {@link FeatureData#setExternalId(String)}
+  <li>FEATURE_ID: {@link FeatureData#setExternalId(String)}
   </ul>
   
   For all methods, each feature that is inserted must have a unqiue value. For example,
-  there can't be two features with the same external ID if the EXTERNAL_ID method is 
+  there can't be two features with the same external ID if the FEATURE_ID method is 
   used. Values for the methods that are not used can still be set and saved to the
   database, but the uniqueness is not checked. The position number will be auto-generated 
@@ -79 +79 @@
       add features to
     @throws BaseException If there is an error
-    @see ArrayDesign#getFeatureBatcher()
+    @see ArrayDesign#getFeatureBatcher(FeatureIdentificationMethod)
   */
   static FeatureBatcher getNew(DbControl dc, ArrayDesign arrayDesign)

trunk/src/core/net/sf/basedb/core/FeatureIdentificationMethod.java

@@ -111 +111 @@
     Use the external feature ID.
   */
-  EXTERNAL_ID(3, true, "External ID")
+  FEATURE_ID(3, true, "Feature ID")
   {
     /**

trunk/src/core/net/sf/basedb/core/RawBioAssay.java

@@ -58 +58 @@
   <p>
   Raw data is added by a {@link RawDataBatcher} object. You can get a 
-  batcher for this raw bioassay by calling the {@link #getRawDataBatcher()}
-  method.
+  batcher for this raw bioassay by calling the {@link 
+  #getRawDataBatcher(FeatureIdentificationMethod)} method.
 
   @author Nicklas
@@ -418 +418 @@
       if (getRawDataType().isStoredInDb() && getSpots() > 0)
       {
-        RawDataBatcher rdb = RawDataBatcher.getNew(getDbControl(), this);
+        RawDataBatcher rdb = RawDataBatcher.getNew(getDbControl(), this, null);
         rdb.deleteAll();
       }
@@ -731 +731 @@
     getData().setArrayDesign(arrayDesign == null ? null : arrayDesign.getData());
   }
-  
+
+  /**
+    @deprecated Use {@link #updateArrayDesign(ArrayDesign, FeatureIdentificationMethod, ProgressReporter)}
+    instead
+  */
+  public void updateArrayDesign(ArrayDesign arrayDesign, ProgressReporter progress)
+    throws PermissionDeniedException, InvalidDataException
+  {
+    updateArrayDesign(arrayDesign, null, progress);
+  }
+
   /**
     Set the {@link ArrayDesign} this raw data uses. If data has already been
@@ -739 +749 @@
   
     @param arrayDesign The <code>ArrayDesign</code> item or null if not known
+    @param fiMethod The method to use for identifying features on the
+      new array design. Use null to use the method choosen by the new array design.
+      The {@link FeatureIdentificationMethod#FEATURE_ID} will not work unless the
+      raw bioassay is already connected to an array design which has features with
+      external ID:s
     @param progress An optional progress reporter
     @throws PermissionDeniedException If the logged in user doesn't have 
@@ -745 +760 @@
       match the raw data
     @throws BaseException If there is another error
-    @since 2.4
-  */
-  public void updateArrayDesign(ArrayDesign arrayDesign, ProgressReporter progress)
+    @since 2.6
+  */
+  public void updateArrayDesign(ArrayDesign arrayDesign, FeatureIdentificationMethod fiMethod, ProgressReporter progress)
     throws PermissionDeniedException, InvalidDataException
   {
@@ -760 +775 @@
     if (getData().getSpots() > 0 && arrayDesign != null) 
     {
-      validateFeatures(getDbControl(), arrayDesign, true, progress);
+      validateFeatures(getDbControl(), arrayDesign, fiMethod, true, progress);
     }
     getData().setArrayDesign(arrayDesign == null ? null : arrayDesign.getData());
@@ -956 +971 @@
     return query;
   }
+
+  /**
+    @deprecated Use {@link #getRawDataBatcher(FeatureIdentificationMethod)} instead
+  */
+  public RawDataBatcher getRawDataBatcher()
+    throws PermissionDeniedException, BaseException
+  {
+    return getRawDataBatcher(null);
+  }
   
   /**
     Get a raw data batcher object which can be used to add raw data to
-    the raw bioassay.
+    the raw bioassay. 
+    @param fiMethod The method to use for identifying features, or null to
+      use the method choosen by the array design. This parameter has no meaning
+      if the raw bioassay is not connected to an array design or if the array
+      design has no features.
+    @return A <code>RawDataBatcher</code> object. If this method is called multiple
+      times the fiMethod parameter is ignored and the same batcher as before 
+      is returned
     @throws PermissionDeniedException If raw data has already been added
       or the logged in user doesn't have write permission
     @throws BaseException If there is another error
-  */
-  public RawDataBatcher getRawDataBatcher()
+    @since 2.6
+  */
+  public RawDataBatcher getRawDataBatcher(FeatureIdentificationMethod fiMethod)
     throws PermissionDeniedException, BaseException
   {
     checkPermission(Permission.WRITE);
-    if (hasData()) 
+    if (getNumDbSpots() > 0) 
     {
       throw new PermissionDeniedException("Raw data has already been added to "+this);
@@ -974 +1006 @@
     if (rawDataBatcher == null)
     {
-      rawDataBatcher = RawDataBatcher.getNew(getDbControl(), this);
+      rawDataBatcher = RawDataBatcher.getNew(getDbControl(), this, fiMethod);
     }
     return rawDataBatcher;
@@ -1004 +1036 @@
     if (getNumDbSpots() > 0)
     {
-      RawDataBatcher rdb = RawDataBatcher.getNew(getDbControl(), this);
+      RawDataBatcher rdb = RawDataBatcher.getNew(getDbControl(), this, null);
       rdb.deleteAll();
     }
@@ -1073 +1105 @@
     features/reporter on the array design.
     @param dc The DbControl to use for database access
-    @param design The array design to validate agains
+    @param design The array design to validate against
+    @param fiMethod The method to use for identifying features, or null
+      to use the method choosen by the array design
     @param update If the raw data should be linked to the new feature or not
     @param progress An optional progress reporter
-    @since 2.4
-  */
-  private void validateFeatures(DbControl dc, ArrayDesign design, boolean update, ProgressReporter progress)
+    @since 2.6
+  */
+  private void validateFeatures(DbControl dc, ArrayDesign design, FeatureIdentificationMethod fiMethod, 
+    boolean update, ProgressReporter progress)
   {
     if (design.hasFeatures())
     {
+      if (fiMethod == null) fiMethod = design.getFeatureIdentificationMethod();
+      if (fiMethod == FeatureIdentificationMethod.NONE) fiMethod = FeatureIdentificationMethod.COORDINATES;
+      
       if (progress != null) progress.display(5, "Loading features from array design...");
       
@@ -1089 +1127 @@
       String entityName = getRawDataType().getEntityName();
       
-      // Preload features from array design
-      org.hibernate.Query query = HibernateUtil.getPredefinedQuery(dc.getStatelessSession(), 
-        "COUNT_FEATURES_FOR_ARRAYDESIGN");
-      /*
-        SELECT count(*)
-        FROM FeatureData f
-        WHERE f.arrayDesignBlock.arrayDesign = :arrayDesign
-      */
-      query.setInteger("arrayDesign", design.getId());
-      int numFeatures = HibernateUtil.loadData(Long.class, query).intValue();
-      Map<Integer, FeatureData> features = new HashMap<Integer, FeatureData>(numFeatures);
-      query = HibernateUtil.getPredefinedQuery(dc.getStatelessSession(), "PRELOAD_FEATURES");
+      int numFeatures = design.getNumDbFeatures();
+      Map<Object, FeatureData> features = new HashMap<Object, FeatureData>(numFeatures);
+      org.hibernate.Query query = HibernateUtil.getPredefinedQuery(dc.getStatelessSession(), "PRELOAD_FEATURES");
       /*
         SELECT f 
@@ -1114 +1143 @@
       {
         FeatureData f = si.next();
-        features.put(f.getPosition(), f);
+        Object featureId = fiMethod.getIdentifier(f);
+        if (features.put(featureId, f) != null)
+        {
+          throw new InvalidDataException("Can't identify features with " + fiMethod + 
+            ". Found duplicate feature: " + featureId);
+        }
       }
       si.close();
@@ -1121 +1155 @@
       if (progress != null) progress.display(10, "Loading raw data from raw bioassay...");
       DataQuery<RawData> rawQuery = getRawData();
+      // NOTE!! Due to an issue in Hibernate we must join all three
+      // tables or we will get an AssertionFailure: possible non-threadsafe access to the session
+      // The query also works if don't join at all, but that will generate a lot
+      // of extra queries for each iteration
+      rawQuery.join(Hql.leftJoin(null, "reporter", "rep", null, true));
+      rawQuery.join(Hql.leftJoin(null, "feature", "feat", null, true));
+      rawQuery.join(Hql.leftJoin("feat", "arrayDesignBlock", "bl", null, true));
+
       DataResultIterator<RawData> result = rawQuery.iterate(dc);
       int numValidated = 0;
@@ -1127 +1169 @@
       {
         RawData raw = result.next();
-        int position = raw.getPosition();
-        FeatureData feature = features.get(position);
-        if (feature == null)
+
+        // Get the feature identifier for the current raw data
+        FeatureData oldFeature = raw.getFeature();
+        String externalFeatureId = oldFeature == null ? null : oldFeature.getExternalId();;
+        Object rawFeatureId = fiMethod.getIdentifier(raw, externalFeatureId);
+        
+        // Find the new feature with the same identifier
+        FeatureData newFeature = features.get(rawFeatureId);
+        if (newFeature == null)
         {
           throw new InvalidDataException("Array design '" + design.getName() +
-            "' is missing a feature at position " + raw.getPosition());
+            "' is missing a feature for " + fiMethod + "=" + rawFeatureId);
         }
         
         // Check that reporters are the same
         ReporterData rawReporter = raw.getReporter();
-        ReporterData featureReporter = feature.getReporter();
+        ReporterData featureReporter = newFeature.getReporter();
         if (rawReporter == null)
         {
           if (featureReporter != null) 
           {
-            throw new InvalidDataException("Reporter mismatch at position " + position +
+            throw new InvalidDataException("Reporter mismatch at " +  fiMethod + "=" + rawFeatureId + 
               "; raw reporter=none, feature reporter=" + featureReporter.getExternalId());
           }
@@ -1150 +1198 @@
           if (featureReporter == null)
           {
-            throw new InvalidDataException("Reporter mismatch at position " + position +
+            throw new InvalidDataException("Reporter mismatch at " + fiMethod + "=" + rawFeatureId +
               "; raw reporter=" + rawReporter.getExternalId() + ", feature reporter=none");
           }
           else if (rawReporter.getId() != featureReporter.getId())
           {
-            throw new InvalidDataException("Reporter mismatch at position " + position +
+            throw new InvalidDataException("Reporter mismatch at " + fiMethod + "=" + rawFeatureId +
               "; raw reporter=" + rawReporter.getExternalId() + 
               ", feature reporter=" + featureReporter.getExternalId());
           }
         }
-          
-        // Check that coordinates are the same
-        FeatureCoordinate featureCoordinate = Feature.getFeatureCoordinate(feature);
-        FeatureCoordinate rawCoordinate = new FeatureCoordinate(raw.getBlock(), 
-          raw.getMetaGridX(), raw.getMetaGridY(), raw.getRow(), raw.getColumn());
-        
-        if (!featureCoordinate.equals(rawCoordinate))
-        {
-          throw new InvalidDataException("Coordinate mismatch at position " + position +
-            "; raw=" + rawCoordinate +", feature=" + featureCoordinate);
-        }
-        
+    
         // All is ok, re-assign raw data to the new feature
         if (update)
         {
-          metaData.setPropertyValue(raw, "feature", feature, EntityMode.POJO);
+          metaData.setPropertyValue(raw, "feature", newFeature, EntityMode.POJO);
+          raw.setPosition(newFeature.getPosition());
           session.update(entityName, raw);
         }

trunk/src/core/net/sf/basedb/core/RawDataBatcher.java

@@ -92 +92 @@
     @param rawBioAssay The raw bioassay this batcher should 
       add raw data to
+    @param fiMethod The method to use for identifying features, use null
+      to use the method specified by the array design
     @throws BaseException If there is an error
-    @see RawBioAssay#getRawDataBatcher()
-  */
-  static RawDataBatcher getNew(DbControl dc, RawBioAssay rawBioAssay)
-    throws BaseException
-  {
-    RawDataBatcher rdb = new RawDataBatcher(dc, rawBioAssay);
+    @see RawBioAssay#getRawDataBatcher(FeatureIdentificationMethod)
+  */
+  static RawDataBatcher getNew(DbControl dc, RawBioAssay rawBioAssay, FeatureIdentificationMethod fiMethod)
+    throws BaseException
+  {
+    RawDataBatcher rdb = new RawDataBatcher(dc, rawBioAssay, fiMethod);
     rdb.initPermissions(0, 0);
     return rdb;
@@ -164 +166 @@
   private boolean useNullIfReporterNotFound = true;
   
-  RawDataBatcher(DbControl dc, RawBioAssay rawBioAssay)
+  RawDataBatcher(DbControl dc, RawBioAssay rawBioAssay, FeatureIdentificationMethod fim)
     throws BaseException
   {
@@ -178 +180 @@
     }
 
-    this.fiMethod = arrayDesign != null && arrayDesign.getHasFeatures() ?
-        FeatureIdentificationMethod.fromValue(arrayDesign.getFeatureIdentificationMethod()) :
-        FeatureIdentificationMethod.NONE;
-
-    if (fiMethod != FeatureIdentificationMethod.NONE)
+    // Use the feature identification method set by the array design if none was given
+    this.fiMethod = fim;
+    if (arrayDesign != null && arrayDesign.getHasFeatures())
+    {
+      if (fiMethod == null)
+      {
+        fiMethod = FeatureIdentificationMethod.fromValue(arrayDesign.getFeatureIdentificationMethod());
+      }
+      // As a last resort fallback to the coordinates method
+      if (fiMethod == null || fiMethod == FeatureIdentificationMethod.NONE)
+      {
+        fiMethod = FeatureIdentificationMethod.COORDINATES;
+      }
+    }
+    else
+    {
+      fiMethod = FeatureIdentificationMethod.NONE;
+    }
+    
+    // Preload features 
+    if (fiMethod != FeatureIdentificationMethod.NONE && rawBioAssay.getNumDbSpots() == 0)
     {
       org.hibernate.Query query = HibernateUtil.getPredefinedQuery(dc.getStatelessSession(), "COUNT_FEATURES_FOR_ARRAYDESIGN");
@@ -208 +226 @@
         FeatureData feature = si.next();
         Object featureId = fiMethod.getIdentifier(feature);
-        preloaded.put(featureId, feature);
+        if (preloaded.put(featureId, feature) != null)
+        {
+          throw new InvalidDataException("Can't identify features with " + fiMethod + 
+            ". Found duplicate feature: " + featureId);
+        }
       }
       si.close();
@@ -309 +331 @@
     {
       granted |= Permission.grant(Permission.WRITE, Permission.DELETE, Permission.CREATE);
+      if (rawBioAssay.getNumDbSpots() > 0)
+      {
+        denied |= Permission.deny(Permission.CREATE);
+      }
     }
     super.initPermissions(granted, denied);

trunk/src/core/net/sf/basedb/core/data/ArrayDesignData.java

@@ -159 +159 @@
   /**
     The method to use for identiying features.
-    1 = COORDINATES, 2 = POSITION, 3 = EXTERNAL_ID
+    1 = COORDINATES, 2 = POSITION, 3 = FEATURE_ID
     @hibernate.property column="`identification_method`" type="int" not-null="true"
     @since 2.6

trunk/src/plugins/core/net/sf/basedb/plugins/IlluminaRawDataImporter.java

@@ -38 +38 @@
 
 
+import net.sf.basedb.core.ArrayDesign;
 import net.sf.basedb.core.BaseException;
 import net.sf.basedb.core.BasicItem;
@@ -109 +110 @@
     
   <li>
-    If the array design uses the {@link FeatureIdentificationMethod#EXTERNAL_ID}
+    If the array design uses the {@link FeatureIdentificationMethod#FEATURE_ID}
     method for identifying features. The plug-in assumes that the feature ID is the
     same as the reporter ID.
@@ -151 +152 @@
     ));
   
+  private static final PluginParameter<String> featureIdentificationParameter = 
+    new PluginParameter<String>(
+      "featureIdentification",
+      "Identify features by",
+      "Choose which method to use for identifying features. If no value " +
+      "is selected the identification method used on the array design is used: \n\n" +
+      "* COORDINATES: Use auto-generated coordinates (block = 1, column = 1, row = line number in file)\n" +
+      "* POSITION: Use auto-generated position = line number in file\n" +
+      "* FEATURE_ID: Use TargetID (requires that each Target ID only appears once in the file)\n\n" +
+      "NOTE! This parameter has no meaning unless an array design is selected",
+      new StringParameterType(255, null, false, 1, 0, 0, 
+        Arrays.asList(new String[] {"COORDINATE", "POSITION", "FEATURE_ID" } ))
+    );
+
+  
   private static final PluginParameter<String> invalidColumnsErrorParameter = new PluginParameter<String>(
       "invalidColumnsError",
@@ -277 +293 @@
         // Associations
         storeValue(job, request, ri.getParameter("experiment"));
+        storeValue(job, request, ri.getParameter("arrayDesign"));
+        storeValue(job, request, featureIdentificationParameter);
         storeValue(job, request, ri.getParameter("scan"));
         storeValue(job, request, ri.getParameter("protocol"));
@@ -306 +324 @@
   private DbControl dc;
   private Experiment experiment;
+  private ArrayDesign design;
   private Scan scan;
   private Software software;
   private Protocol protocol;
+  private FeatureIdentificationMethod fiMethod;
   private List<RawBioAssay> rawBioAssays;
   private List<BatchAndMapHolder> holders;
@@ -369 +389 @@
     this.headerLines = new LinkedList<Line>();
     this.experiment = (Experiment)job.getValue("experiment");
+    this.design = (ArrayDesign)job.getValue("arrayDesign");
     this.scan = (Scan)job.getValue("scan");
     this.protocol = (Protocol)job.getValue("protocol");
     this.software = (Software)job.getValue("software");
     
+    // Feature identification
+    try
+    {
+      String fiTemp = (String)job.getValue("featureIdentification");
+      fiMethod = FeatureIdentificationMethod.valueOf(fiTemp);
+    }
+    catch (Exception ex)
+    {
+      fiMethod = null;
+    }
     // Setup error handling
     this.nullIfException = "null".equals(getErrorOption("numberFormatError"));
@@ -429 +460 @@
       // Need to reload raw bioassay with current DbControl
       rba = RawBioAssay.getById(dc, rba.getId());
-      RawDataBatcher batcher = rba.getRawDataBatcher();
+      RawDataBatcher batcher = rba.getRawDataBatcher(fiMethod);
       batcher.useNullIfReporterNotFound(nullIfMissingReporter);
       holders.add(new BatchAndMapHolder(batcher, illumina, ffp));
@@ -566 +597 @@
             RawBioAssay rba = RawBioAssay.getNew(dc, generic, illumina);
             rba.setName(arrayName);
+            if (design != null) rba.setArrayDesign(design);
             if (scan != null) rba.setScan(scan);
             if (protocol != null) rba.setProtocol(protocol);
@@ -618 +650 @@
         parameters.add(getCharsetParameter(null, null, null));
   
-        // parameters for scan, protocol and software
+        // parameters for scan, protocol, software and array design
         dc = sc.newDbControl();
         List<Scan> scans = getItems(dc, Scan.getQuery());
@@ -633 +665 @@
             )
           );
+        List<ArrayDesign> designs = getItems(dc, ArrayDesign.getQuery(),
+          Restrictions.gt(
+              Hql.property("numDbFeatures"), 
+              Expressions.integer(0)
+            )
+          );
         
         boolean hasAssociations = 
           context.getItem() == Item.EXPERIMENT || scans.size() > 0 || 
-          protocols.size() > 0 || software.size() > 0;
+          protocols.size() > 0 || software.size() > 0 || designs.size() > 0;
         
         if (hasAssociations)
@@ -650 +688 @@
                 new ItemParameterType<Experiment>(Experiment.class, null, true, 1, null)
               ));
+          }
+          if (!designs.isEmpty())
+          {
+            parameters.add(new PluginParameter<ArrayDesign>(
+              "arrayDesign",
+              "Array design",
+              "The imported raw bioassays will be linked to the selected array design.",
+              new ItemParameterType<ArrayDesign>(ArrayDesign.class, null, false, 1, designs)
+            ));
+            parameters.add(featureIdentificationParameter);
           }
           if (!scans.isEmpty())

trunk/src/plugins/core/net/sf/basedb/plugins/RawDataFlatFileImporter.java

@@ -26 +26 @@
 
 import net.sf.basedb.core.AnnotationType;
+import net.sf.basedb.core.ArrayDesign;
 import net.sf.basedb.core.BaseException;
 import net.sf.basedb.core.DataFileType;
 import net.sf.basedb.core.DbControl;
+import net.sf.basedb.core.FeatureIdentificationMethod;
 import net.sf.basedb.core.File;
 import net.sf.basedb.core.FileParameterType;
@@ -208 +210 @@
       "Mapping that picks the spot's feature ID from the data columns. " +
       "This column is only used when the raw data is connected to an array design " +
-      "which uses the EXTERNAL_ID method for identifying features. The value is not saved " +
+      "which uses the FEATURE_ID method for identifying features. The value is not saved " +
       "to the database." +
       "For example: \\Feature ID\\",
@@ -530 +532 @@
     }
     
-    batcher = rawBioAssay.getRawDataBatcher();
+    FeatureIdentificationMethod fiMethod = null;
+    if (job.getValue("featureIdentification") != null)
+    {
+      try
+      {
+        fiMethod = FeatureIdentificationMethod.valueOf((String)job.getValue("featureIdentification"));
+      }
+      catch (Exception ex)
+      {}
+    }
+    
+    batcher = rawBioAssay.getRawDataBatcher(fiMethod);
     numInserted = 0;
     this.ffp = ffp;
@@ -678 +691 @@
     {
       allColumnMappings = new ArrayList<PluginParameter<String>>();
+      allColumnMappings.add(reporterIdColumnMapping);
+      allColumnMappings.add(featureIdColumnMapping);
       allColumnMappings.add(positionColumnMapping);
       allColumnMappings.add(blockColumnMapping);
@@ -686 +701 @@
       allColumnMappings.add(xColumnMapping);
       allColumnMappings.add(yColumnMapping);
-      allColumnMappings.add(reporterIdColumnMapping);
-      allColumnMappings.add(featureIdColumnMapping);
       for(RawDataProperty rdp : rdt.getProperties())
       {
@@ -726 +739 @@
       dc = sc.newDbControl();
       boolean configIsAnnotated = false;
+      ArrayDesign design = null;
       try
       {
@@ -731 +745 @@
         rawDataFiles = FileStoreUtil.getGenericDataFiles(dc, rba, FileType.RAW_DATA);
         configIsAnnotated = getCurrentConfiguration(dc).isAnnotated();
+        design = rba.getArrayDesign();
       }
       catch (Throwable t)
@@ -748 +763 @@
           null : rawDataFiles.get(0), true, 1)
       );
-      parameters.add(fileParameter);      
+      parameters.add(fileParameter);
+      
+      if (design != null && design.getNumDbFeatures() > 0)
+      {
+        // Parameter for asking which feature identification method to use
+        // Only used if the connected array design has features
+        
+        FeatureIdentificationMethod fiMethod = design.getFeatureIdentificationMethod();
+        List<String> methods = new ArrayList<String>();
+        String description = "Choose which method to use for identifying features: \n";
+        if (configuration.getValue("rowColumnMapping") != null && configuration.getValue("columnColumnMapping") != null)
+        {
+          methods.add("COORDINATES");
+          description += "COORDINATES: Use block, meta-grid, row and column coordinates\n";
+        }
+        if (configuration.getValue("positionColumnMapping") != null)
+        {
+          methods.add("POSITION");
+          description += "POSITION: Use a position number only\n";
+        }
+        if (configuration.getValue("featureIdColumnMapping") != null)
+        {
+          methods.add("FEATURE_ID");
+          description += "FEATURE_ID: Use the feature ID value (string)\n";
+        }
+        
+        if (methods.size() > 0)
+        {
+          PluginParameter<String> featureIdentificationParameter = 
+            new PluginParameter<String>(
+              "featureIdentification",
+              "Identify features by",
+              description,
+              new StringParameterType(255, methods.contains(fiMethod.name()) ? fiMethod.name() : null, 
+                true, 1, 0, 0, methods)
+            );
+          parameters.add(featureIdentificationParameter);
+        }
+      }
+      
       parameters.add(getCharsetParameter(null, null, (String)configuration.getValue(CHARSET)));
       parameters.add(getDecimalSeparatorParameter(null, null, (String)configuration.getValue(DECIMAL_SEPARATOR)));

trunk/src/plugins/core/net/sf/basedb/plugins/ReporterMapFlatFileImporter.java

@@ -118 +118 @@
       "COORDINATES: Use block, meta-grid, row and column coordinates\n" +
       "POSITION: Use a position number only\n" +
-      "EXTERNAL_ID: Use the feature ID value (string)\n" +
+      "FEATURE_ID: Use the feature ID value (string)\n" +
       "In all cases, the identifier value must be unique.",
       new StringParameterType(255, "COORDINATES", true, 1, 0, 0, 
-        Arrays.asList(new String[] {"COORDINATES", "POSITION", "EXTERNAL_ID"}))
+        Arrays.asList(new String[] {"COORDINATES", "POSITION", "FEATURE_ID"}))
     );
   
@@ -136 +136 @@
     "Feature ID",
     "Mapping that picks the feature's ID from the data columns. This column is " +
-    "only used when the array design uses the EXTERNAL_ID method for " +
+    "only used when the array design uses the FEATURE_ID method for " +
     "identifying features. In the other cases, the value is just stored as it is." +
     "For example: \\Feature ID\\",
@@ -407 +407 @@
           if (positionMapping == null) mapErrors.add(new InvalidUseOfNullException("Position"));
         }
-        else if (fiMethod == FeatureIdentificationMethod.EXTERNAL_ID)
+        else if (fiMethod == FeatureIdentificationMethod.FEATURE_ID)
         {
           String positionMapping = (String)request.getParameterValue("featureIdColumnMapping");
@@ -421 +421 @@
           response.setError("Missing column mapping for feature identification method: " 
             + fiMethod, mapErrors);
+          return;
         }
         
@@ -571 +572 @@
     
     FeatureData feature = batcher.newFeature(adb, reporter);
+    feature.setExternalId(featureId);
     Integer position = positionMapper.getInt(data);
     Integer row = rowMapper.getInt(data);

trunk/src/test/TestAnalyzePluginUtil.java

@@ -72 +72 @@
         rba.setName("TestAnalyzePluginUtil"+i);
         rba.setDescription("RawBioAssay created by TestAnalyzePluginUtil. Number of spots should be "+spots);
-        RawDataBatcher rawDataBatcher = rba.getRawDataBatcher();
+        RawDataBatcher rawDataBatcher = rba.getRawDataBatcher(null);
         for (int j = 0; parser.hasMoreData() && j < spots; j++)
         {

trunk/src/test/TestArrayDesign.java

@@ -75 +75 @@
     int id3 = test_create(Platform.AFFYMETRIX, true);
     int id4 = test_create(Platform.AFFYMETRIX, false);
+    int id6 = test_create(Platform.GENERIC, false);
     test_load(id);
     test_list(-1);
@@ -82 +83 @@
     test_import_from_file(id2, "data/test.reportermap.import.txt", 
       "\"Block\"\\t\"Column\"\\t\"Row\"\\t\"ID\"\\t\"Name\".*", 0, 2, 1, 3);
-    test_add_features(id, 10);
+    test_add_features(id, FeatureIdentificationMethod.COORDINATES, 10);
+    test_add_features(id6, FeatureIdentificationMethod.FEATURE_ID, 10);
 
     write_feature_header();
@@ -115 +117 @@
     test_delete(id4);
     test_delete(id5);
+    test_delete(id6);
     
     TestPlate.test_delete_all();
@@ -310 +313 @@
   }
 
-  static void test_add_features(int arrayDesignId, int numFeatures)
+  static void test_add_features(int arrayDesignId, FeatureIdentificationMethod fiMethod, int numFeatures)
   {
     if (arrayDesignId == 0) return;
@@ -329 +332 @@
       }
       
-      FeatureBatcher batcher = ad.getFeatureBatcher(FeatureIdentificationMethod.COORDINATES);
+      FeatureBatcher batcher = ad.getFeatureBatcher(fiMethod);
       int block = 1;
       int row = 1;
       int column = 1;
+      int position = 1;
       ArrayDesignBlock adBlock = ad.addArrayDesignBlock(new BlockInfo(block, (int)Math.ceil(block/4F), ((block-1)%4)+1));
       time = System.currentTimeMillis();
@@ -345 +349 @@
         data.setRow(row);
         data.setColumn(column);
+        data.setPosition(position);
+        data.setExternalId(reporter.getExternalId() + "." + position);
 
         batcher.insert(data);
         row++;
+        position++;
         if (row == 11)
         {

trunk/src/test/TestBioAsssaySetExporter.java

@@ -171 +171 @@
       ReporterData r2 = Reporter.getById(dc, r2Id);
       ReporterData r3 = Reporter.getById(dc, r3Id);
-      RawDataBatcher rdb = rba.getRawDataBatcher();
+      RawDataBatcher rdb = rba.getRawDataBatcher(null);
       for (int i=0; i < 4; ++i)
       {

trunk/src/test/TestRawBioAssay.java

@@ -315 +315 @@
       RawDataType rdt = rba.getRawDataType();
       List<RawDataProperty> extraProperties = rdt.getProperties();
-      RawDataBatcher rdb = rba.getRawDataBatcher();
+      RawDataBatcher rdb = rba.getRawDataBatcher(null);
       int block = 1;
       int row = 1;
@@ -459 +459 @@
       dc = TestUtil.getDbControl();
       RawBioAssay rba = RawBioAssay.getById(dc, rawBioAssayId);
-      RawDataBatcher rawDataBatcher = rba.getRawDataBatcher();
+      RawDataBatcher rawDataBatcher = rba.getRawDataBatcher(null);
       ReporterBatcher reporterBatcher = ReporterBatcher.getNew(dc);
     

trunk/src/test/net/sf/basedb/test/merge/MergeTest.java

@@ -238 +238 @@
     TestUtil.write("--Creating data for raw bioassay: " + rba.getName() + "\n");
     dc.reattachItem(rba);
-    RawDataBatcher batcher = rba.getRawDataBatcher();
+    RawDataBatcher batcher = rba.getRawDataBatcher(null);
     for (int block = 1; block <= 10; block++)
     {

trunk/www/lims/arraydesigns/features/index.jsp

@@ -46 +46 @@
 <%@ taglib prefix="tbl" uri="/WEB-INF/table.tld" %>
 <%!
-  private static final ItemContext defaultContext = Base.createDefaultContext("position", "position,block,row,column,reporter.name,reporter.externalId");
+  private static final ItemContext defaultContext = Base.createDefaultContext("position", 
+    "position,externalId,block,row,column,reporter.name,reporter.externalId");
   private static final Item itemType = Item.FEATURE;
 %>

trunk/www/lims/arraydesigns/features/list_features.jsp

@@ -36 +36 @@
   import="net.sf.basedb.core.ArrayDesignBlock"
   import="net.sf.basedb.core.Feature"
+  import="net.sf.basedb.core.FeatureIdentificationMethod"
   import="net.sf.basedb.core.ReporterType"
   import="net.sf.basedb.core.Reporter"
@@ -88 +89 @@
   final ArrayDesign design = ArrayDesign.getById(dc, arrayDesignId);
   final boolean isAffy = design.isAffyChip();
+  final FeatureIdentificationMethod fiMethod = design.getFeatureIdentificationMethod();
   Formatter<Date> dateFormatter = FormatterFactory.getDateTimeFormatter(sc);
   
@@ -112 +114 @@
   int numListed = 0;
   %>
-  <base:page title="<%=title%>">
+<base:page title="<%=title%>">
   <base:head scripts="table.js,tabcontrol.js" styles="table.css,headertabcontrol.css,path.css">
     <script language="JavaScript">
@@ -210 +212 @@
         filterable="true"
         exportable="true"
-        show="always" 
+        show="<%=fiMethod == FeatureIdentificationMethod.EXTERNAL_ID ? "auto" : "always"%>"
+      />
+      <tbl:columndef 
+        id="externalId"
+        property="externalId"
+        datatype="string"
+        title="Feature ID"
+        sortable="true" 
+        filterable="true"
+        exportable="true"
+        show="<%=fiMethod == FeatureIdentificationMethod.EXTERNAL_ID ? "always" : "auto"%>"
       />
       <tbl:columndef 
@@ -540 +552 @@
                 <tbl:cell column="position"><div class="link" onclick="itemOnClick(event, <%=itemId%>)" 
                   title="<%=tooltip%>"><%=item.getPosition()%></div></tbl:cell>
+                <tbl:cell column="externalId"><div class="link" onclick="itemOnClick(event, <%=itemId%>)" 
+                  title="<%=tooltip%>"><%=HTML.encodeTags(item.getExternalId())%></div></tbl:cell>
                 <tbl:cell column="block"><%=block.getBlockNumber()%></tbl:cell>
                 <tbl:cell column="row"><%=item.getRow()%></tbl:cell>

trunk/www/lims/arraydesigns/features/view_feature.jsp

@@ -79 +79 @@
 
   final ArrayDesign design = ArrayDesign.getById(dc, arrayDesignId);
+  
   final boolean isAffy = design.isAffyChip();
   FeatureData feature = design.getFeatureById(itemId);
@@ -107 +108 @@
       </tr>
       <tr>
-        <td class="prompt">Affy chip</td>
-        <td><%=isAffy ? " yes" : "no"%></td>
+        <td class="prompt">Platform / Variant</td>
+        <td><base:propertyvalue item="<%=design%>" property="platform" /> / <base:propertyvalue item="<%=design%>" property="variant" /></td>
+      </tr>
+      <tr>
+        <td class="prompt">Feature identification method</td>
+        <td><%=design.getFeatureIdentificationMethod()%></td>
       </tr>
       </table>
@@ -114 +119 @@
       <h4>Feature / block</h4>
       <table class="form" cellspacing=0>
+      <tr>
+        <td class="prompt">Feature ID</td>
+        <td colspan="7"><%=HTML.encodeTags(feature.getExternalId())%></td>
+      </tr>
       <tr>
         <td class="prompt">Position</td>

trunk/www/lims/arraydesigns/list_designs.jsp

@@ -48 +48 @@
   import="net.sf.basedb.core.PermissionDeniedException"
   import="net.sf.basedb.core.PluginDefinition"
+  import="net.sf.basedb.core.FeatureIdentificationMethod"
   import="net.sf.basedb.core.query.Hql"
   import="net.sf.basedb.core.query.Restrictions"
@@ -72 +73 @@
   private static final Item itemType = Item.ARRAYDESIGN;
   private static final GuiContext guiContext = new GuiContext(itemType, GuiContext.Type.LIST);
+  private static final Enumeration<String, String> fiMethods = new Enumeration<String, String>();
+  static
+  {
+    for (FeatureIdentificationMethod fim : FeatureIdentificationMethod.values())
+    {
+      fiMethods.add(Integer.toString(fim.getValue()), fim.toString());
+    }
+    fiMethods.lock();
+  }
 %>
 <%
@@ -305 +315 @@
         filterable="true"
         exportable="true"
+      />  
+      <tbl:columndef 
+        id="featureIdentificationMethod"
+        property="featureIdentificationMethod"
+        datatype="int"
+        title="Feature ID meth."
+        tooltip="Feature identification method"
+        sortable="true" 
+        filterable="true"
+        exportable="true"
+        enumeration="<%=fiMethods%>"
       />  
       <tbl:columndef
@@ -547 +568 @@
                 <tbl:cell column="numDbFeatures"><%=item.getNumDbFeatures()%></tbl:cell>
                 <tbl:cell column="numFileFeatures"><%=item.getNumFileFeatures()%></tbl:cell>
+                <tbl:cell column="featureIdentificationMethod"><%=item.getFeatureIdentificationMethod()%></tbl:cell>
                 <tbl:cell column="batches">
                   <%

trunk/www/lims/arraydesigns/view_design.jsp

@@ -303 +303 @@
         <td class="prompt">Features</td>
         <td><%=design.hasFeatures() ? "yes (db: " + design.getNumDbFeatures() + "; file: " + design.getNumFileFeatures() + ")" : "no"%></td>
+      </tr>
+      <tr>
+        <td class="prompt">Feature identification method</td>
+        <td><%=design.getFeatureIdentificationMethod()%></td>
       </tr>
       <tr>

trunk/www/views/rawbioassays/edit_rawbioassay.jsp

@@ -509 +509 @@
     {
       parentsChanged = true;
+      <%
+      if (rawBioAssay != null && rawBioAssay.getNumDbSpots() > 0)
+      {
+        %>
+        var frm = document.forms['rawbioassay'];
+        var selectedId = frm.arraydesign_id[frm.arraydesign_id.selectedIndex].value;
+        var showFiMethod = (selectedId > 0 && selectedId != <%=currentArrayDesign == null ? 0 : currentArrayDesign.getId()%>);
+        if (showFiMethod)
+        {
+          Main.show('fiMethod');
+        }
+        else
+        {
+          Main.hide('fiMethod');
+        }
+        <%
+      }
+      %>
     }
     function selectArrayDesignOnClick()
@@ -535 +553 @@
       list.selectedIndex = 1;
       parentsChanged = true;
+      arrayDesignOnChange();
     }
     
@@ -653 +672 @@
         </td>
       </tr>
-
+      <tr id="fiMethod" style="display: none;">
+        <td></td>
+        <td>
+        <select name="fiMethod">
+        <option value="">-auto- (array design decides)
+        <option value="COORDINATES">Coordinates
+        <option value="POSITION">Position
+        <option value="FEATURE_ID">Feature ID (*)
+        </select>
+        <br>
+        Select a method for feature identification. <br>
+        (*) The Feature ID method only works if the raw data is currently
+        connected to an array design which has Feature ID values.
+        </td>
+      </tr>
       <tr>
         <td class="prompt">Protocol</td>

trunk/www/views/rawbioassays/index.jsp

@@ +1 @@
+<%@page import="net.sf.basedb.core.FeatureIdentificationMethod"%>
 <%-- $Id$
   ------------------------------------------------------------------
@@ -90 +91 @@
     private final int arrayDesignId;
     private final SessionControl sc;
-    
-    private UpdateArrayDesign(Job job, RawBioAssay rba, ArrayDesign design)
+    private final FeatureIdentificationMethod fiMethod;
+    
+    private UpdateArrayDesign(Job job, RawBioAssay rba, ArrayDesign design, FeatureIdentificationMethod fiMethod)
     {
       this.jobId = job.getId();
@@ -97 +99 @@
       this.arrayDesignId = design.getId();
       this.sc = job.getSessionControl();
+      this.fiMethod = fiMethod;
     }
     
@@ -113 +116 @@
         try
         {
-          rba.updateArrayDesign(design, progress);
+          rba.updateArrayDesign(design, fiMethod, progress);
           progress.display(99, "Committing changes to database...");
           dc.commit();
@@ -124 +127 @@
         catch (Throwable t)
         {
+          if (dc != null) dc.close();
           dc = sc.newDbControl();
           j = Job.getById(dc, jobId);
@@ -284 +288 @@
     Job job = null;
     ArrayDesign ad = null;
+    FeatureIdentificationMethod fiMethod = null;
     int arrayDesignId = Values.getInt(request.getParameter("arraydesign_id"), -1);
     if (arrayDesignId >= 0) // < 0 = denied or unchanged
@@ -290 +295 @@
       if (rba.hasData())
       {
+        try
+        {
+          fiMethod = FeatureIdentificationMethod.valueOf(request.getParameter("fiMethod"));
+        }
+        catch (Exception ex)
+        {}
         if (rba.getRawDataType().isStoredInDb() && ad != null && ad.hasFeatures())
         {
@@ -301 +312 @@
         else
         {
-          rba.updateArrayDesign(ad, null);
+          rba.updateArrayDesign(ad, fiMethod, null);
         }
       }
@@ -323 +334 @@
     {
       redirect = "../jobs/index.jsp?ID="+ID+"&cmd=ViewItem&item_id="+job.getId();
-      Thread updateThread = new Thread(new UpdateArrayDesign(job, rba, ad));
+      Thread updateThread = new Thread(new UpdateArrayDesign(job, rba, ad, fiMethod));
       updateThread.setPriority(Thread.currentThread().getPriority() - 1);
       updateThread.start();

trunk/www/views/rawbioassays/rawdata/list_rawdata.jsp

@@ -95 +95 @@
 
   final DataQuery<RawData> query = rawBioAssay.getRawData();
+  query.join(Hql.leftJoin(null, "reporter", "r", null, true));
+  query.join(Hql.leftJoin(null, "feature", "f", null, true));
+  query.join(Hql.leftJoin("f", "arrayDesignBlock", "b", null, true));
   cc.configureQuery(query, true);
   if (!"row".equals(cc.getSortProperty())) query.order(Orders.asc(Hql.property("row")));
@@ -397 +400 @@
         datatype="int"
         title="[Feat] Position" 
+        sortable="true" 
+        filterable="true" 
+        exportable="true"
+      />
+      <tbl:columndef 
+        id="feature.externalId"
+        property="feature.externalId"
+        datatype="string"
+        title="[Feat] Feature ID" 
         sortable="true" 
         filterable="true" 
@@ -704 +716 @@
                   %>
                   <tbl:cell column="feature.position"><i>- none -</i></tbl:cell>
+                  <tbl:cell column="feature.externalId"><i>- none -</i></tbl:cell>
                   <tbl:cell column="feature.row"><i>- none -</i></tbl:cell>
                   <tbl:cell column="feature.column"><i>- none -</i></tbl:cell>
@@ -721 +734 @@
                   %>
                   <tbl:cell column="feature.position"><%=feature.getPosition()%></tbl:cell>
+                  <tbl:cell column="feature.externalId"><%=HTML.encodeTags(feature.getExternalId())%></tbl:cell>
                   <tbl:cell column="feature.row"><%=feature.getRow()%></tbl:cell>
                   <tbl:cell column="feature.column"><%=feature.getColumn()%></tbl:cell>