Opened 14 years ago

Closed 14 years ago

#721 closed task (fixed)

Store data in files instead of in the database

Reported by: Nicklas Nordborg Owned by: Nicklas Nordborg
Priority: critical Milestone: BASE 2.5
Component: core Version:
Keywords: Cc:

Description (last modified by Nicklas Nordborg)

As I see it there are four basic problems:

  1. Make it possible to select files in the GUI in a flexible way. For example, if we are doing Affymetrix experiments, we should be able to select CEL and CDF files. If we are doing other experiments there are other types of files. We need the concept of a "platform" and some information about what types of files are used. Platforms can be managed by server adminitrators.
  1. We must of course be able to store the selected files and link them to the items the relate to. For example: the CEL file to a raw bioassay and the CDF file to an array design. Are there other item types that may need files?
  1. Read metadata from the files. We want file storage to be equivalent to database storage, so we need a way to extract metadata from the files. For example the number of features for an array design and the number of spots and file headers for a raw bioassay. Should the extraction of metadata be required or optional? It will be easier for users to start with a new platform if it is optional. Downstream features must be aware of this and handle three cases: no files at all, files are present but no metadata, both files and metadata is present.
  1. Import data to the database. Actually, I don't know if this is a problem or not. The existing import plug-ins could be used in this case. A slight modification is probably needed since they should use the file the user selected in step 1. It should be possible to have the data both in the database and in the file(s).

For more information:

Change History (36)

comment:1 Changed 14 years ago by Nicklas Nordborg

Owner: changed from everyone to Nicklas Nordborg
Status: newassigned

comment:2 Changed 14 years ago by Nicklas Nordborg

Description: modified (diff)

comment:3 Changed 14 years ago by Nicklas Nordborg

(In [3681]) References #721: Store data in files instead of in the database

First draft of a specification/implementation design

comment:4 Changed 14 years ago by Nicklas Nordborg

Description: modified (diff)

comment:5 Changed 14 years ago by Nicklas Nordborg

Description: modified (diff)

comment:6 Changed 14 years ago by Nicklas Nordborg

(In [3757]) References #721: Store data in files instead of in the database

Updated specification and documentation.

comment:7 Changed 14 years ago by Nicklas Nordborg

Description: modified (diff)

comment:8 Changed 14 years ago by Nicklas Nordborg

Description: modified (diff)

comment:9 Changed 14 years ago by Nicklas Nordborg

(In [3764]) References #721. Added data layer classes and some core classes.

comment:10 Changed 14 years ago by Nicklas Nordborg

(In [3793]) References #721: Many core classes in place. Work has started on the web interface for registering data file types and platforms.

comment:11 Changed 14 years ago by Nicklas Nordborg

(In [3795]) References #721: Updated documentation to reflect changes in the code [3793]

comment:12 Changed 14 years ago by Nicklas Nordborg

(In [3797]) References #721: Changes systemId to externalId

comment:13 Changed 14 years ago by Nicklas Nordborg

(In [3798]) References #721: Changed systemId to externalId in JSP as well

comment:14 Changed 14 years ago by Nicklas Nordborg

(In [3799]) References #721: Added platform variants to web gui.

comment:15 Changed 14 years ago by Nicklas Nordborg

(In [3800]) References #721: It is now possible to edit platforms and data file types throguh the web interface. Added proper permissions to all items.

comment:16 Changed 14 years ago by Nicklas Nordborg

(In [3801]) References #721: It is now possible to select files for an array design

comment:17 Changed 14 years ago by Nicklas Nordborg

(In [3802]) References #721: File validation is now in place. More robust code for users with not enough permissions.

comment:18 Changed 14 years ago by Nicklas Nordborg

(In [3804]) References #721: Most things for attaching files to array designs are now in place.

  • Added update script for updating existing installations
  • Re-coded parts of the code that depended on depracated methods
  • Changed depracated methods to be backwards compatible
  • Plugins that import to or use array designs can now use the attached files

comment:19 Changed 14 years ago by Nicklas Nordborg

(In [3807]) References #721: Most things for attaching files to rw bioassays are now in place.

comment:20 Changed 14 years ago by Nicklas Nordborg

(In [3812]) References #721: Store data in files instead of in the database

  • Re-coded most of the code that depended on depracated methods
  • Updated import plug-ins to use attached file and to save the file which file has been used
  • Minor fixes to raw bioassay platform/file selection

comment:21 Changed 14 years ago by Nicklas Nordborg

(In [3814]) References #721: Store data in files instead of in the database

  • Experiments are now aware of file-only raw data types
  • Fixed test programs
  • A few remaining Affymetrix issues

comment:22 Changed 14 years ago by Nicklas Nordborg

(In [3817]) References #721: Store data in files instead of in the database

  • Fixes remaining backwards compatibility and update issues

comment:23 Changed 14 years ago by Nicklas Nordborg

(In [3818]) References #721: Store data in files instead of in the database

  • Synchronized Javadoc with code
  • Added missing methods to some classes

comment:24 Changed 14 years ago by Nicklas Nordborg

(In [3835]) References #721. Updated documentation.

comment:25 Changed 14 years ago by Nicklas Nordborg

(In [3848]) References #721: Can now edit raw bioassays, array designs and projects where the platform/variant has been marked for removal

comment:26 Changed 14 years ago by Nicklas Nordborg

(In [3851]) References #721: Filter on file extension when selecting platform data files

comment:27 Changed 14 years ago by Nicklas Nordborg

(In [3852]) References #721: Auto-detect function is now aware of FileStoreEnabled? items and automatically selects a default file for import.

comment:28 Changed 14 years ago by Nicklas Nordborg

(In [3860]) References #721: Started to update the docbook documentation with respect to platforms

comment:29 Changed 14 years ago by Nicklas Nordborg

(In [3865]) References #721: Create doc/html/userdoc/platforms directory to make docbook work on windows.

comment:30 Changed 14 years ago by Nicklas Nordborg

(In [3867]) References #721: Store data in files instead of in the database

  • Minor GUI fixes
  • Some fixes related to Internet Explorer problems

comment:31 Changed 14 years ago by Nicklas Nordborg

(In [3873]) References #721, fixes #789: Extended test program to include more Affymetrix tests and to attach file to array designs and raw bioassays.

comment:32 Changed 14 years ago by Nicklas Nordborg

(In [3899]) References #721: Store data in files instead of in the database

Generated incorrect join on query.

comment:33 Changed 14 years ago by Nicklas Nordborg

(In [3909]) References #721: Store data in files instead of in the database

Remember most recently used platform/variant when creating raw bioassays/array designs

comment:34 Changed 14 years ago by Nicklas Nordborg

(In [3912]) References #721. Fixes NPE in autodetect file format function when used with BasicData? classes

comment:35 Changed 14 years ago by Nicklas Nordborg

(In [4006]) Fixes #844: File.isUsed() doesn't check if a file has been used in a FileSet?. References #721

comment:36 Changed 14 years ago by Nicklas Nordborg

Resolution: fixed
Status: assignedclosed
Note: See TracTickets for help on using tickets.