Run OpenMS feature finding from Proteios

[olle]

It should be possible to run external binary programs PeakPicker and FeatureFinder in the OpenMS package on an mzML file in Proteios and have the generated file uploaded to Proteios. Specifically, it is desirable to have the two programs run in a pipeline, where PeakPicker processes an input mzML file and produces a new one with selected peaks, which are used as input to FeatureFinder, that produces a resulting *.featureXML features file.

[olle]

Ticket accepted.

[olle]

Traceability note:

• The home page of the OpenMS project can be found at ​http://open-ms.sourceforge.net/openms/.
• Proteios SE support for feature finding using the msInspect external package was introduced in Ticket #668 (Run msInspect feature finding from Proteios).

[olle]

Design discussion.

• The design of support for running OpenMS feature detection from Proteios SE will be based on the design for support of running msInspect feature detection.
• The OpenMS package programs differ from msInspect, that produces a text result file with tab-separated values (.tsv file), in that PeakPicker produces an mzML file and FeatureFinder a featureXML XML file.
• The first version will support selection of the algorithm to use for PeakPicker or FeatureFinder.
• Even though programs PeakPicker and FeatureFinder differ in goal, output file format, and available options, they have similar interfaces and they will therefore be managed by a single plug-in, that will adapt to cover the differences.
• When PeakPicker and FeatureFinder should be run in a pipeline, two jobs will be created, a PeakPicker job and a FeatureFinder job, where the FeatureFinder job will have the PeakPicker job as blocker, to ensure that the PeakPicker job has finished and the PeakPicker output file exists before the FeatureFinder job is started.
• In order to create input parameters for the second job in a pipeline, that will ensure that the temporary file created by the first job is used as input to the second, but that the name of the final result file will reference the original input file, the class managing job creation will have to perform more work than many job-creating classes. This design was chosen as all necessary information is naturally available to this class. To make the differences in input parameters to the two jobs more clear in the code, the job creation part will be extracted into its own private method, so as not to obscure the former.
• When only one OpenMS program is run, the result file should be automatically uploaded to Proteios SE. When PeakPicker and FeatureFinder are run in a pipeline, the intermediary PeakPicker result file will only be uploaded to Proteios SE if explicitly requested by the user.
• Result files uploaded to Proteios SE should have an extra specifier in the file extension indicating the OpenMS program that produced it, ".peakpicker.mzML" and ".featurefinder.featureXML". When the programs are run in a pipeline, the final FeatureFinder result file gets file extension ".peakpicker.featurefinder.featureXML".
• A new file type should be added for OpenMS features, "OpenMS featureXML".

[olle]

(In [4144]) Refs #744. Support for OpenMS properties file added:

1. New OpenMS properties template file conf/openms.properties.in in client/servlet/ added.

2. New class/file OpenMsPropertiesFile.java in client/servlet/ added. It gives access to settings in OpenMS properties file openms.properties.

[olle]

(In [4145]) Refs #744. OpenMS properties template file conf/openms.properties.in in client/servlet/ updated in comment to correct file extension.

[olle]

(In [4146]) Refs #744. New directory action/openMs/ in client/servlet/ added.

[olle]

(In [4147]) Refs #744. Addition of a new FileType constant for OpenMS features:

1, Class/file core/FileType,java in api/core/ updated with new public static final String OPENMS_FEATURES.

2. Class/file core/Install.java in api/core/ updated in public static synchronized method void initDatabase(ProgressReporter progress, String rootPassword) to call private static method FileTypeData createFileType(String systemId, String name, String description) for new file type constant FileType.OPENMS_FEATURES.

[olle]

(In [4148]) Refs #744. Refs #287. First version of support for running OpenMS feature finding from Proteios SE (progress report not supported):

1. New class/file action/file/UseSpectrumFileForOpenMsSearchExtension.java in client/servlet/ added. It implements interface ContextEnabled through class FileContext. List of id values for selected spectrum files are added as hidden fields to the form, and a button is displayed that is coupled to an action link to new class RunOpenMs.

2. New class/file action/openMs/RunOpenMs.java in client/servlet/ added. It retrieves a list of id values for spectrum files from valid parameter VInteger VSPECTRUMFILEID and settings from OpenMS properties file openms.properties, after which it creates one job for each spectrum file managed by new plug-in class RunOpenMsPlugin. If OpenMS programs PeakPicker and FeatureFinder should be run in a pipeline, a second job is created that uses the result file from the first job as input file.

3. New class/file plugins/RunOpenMsPlugin.java in plugin/ added. It copies the spectrum input file to a temporary file with a time stamp in the file name to the temp folder, and then ruins the OpenMS program on the latter file. It attempts to upload the result file, after which it deletes the created temporary files. The result file will not be deleted if it is to be used as input for another program in a pipeline, or if uploading the result file to Proteios SE was unsuccessful.

4. English dictionary file locale/en/dictionary in client/servlet/ updated with new string keys.

[olle]

(In [4149]) Refs #744. Outermost Ant build.xml file updated for target "dist-jar" to include files openms.properties*, when copying files from directory client/servlet/conf/ to dist/ProteiosSE-*-dev/www/WEB-INF/classes/. Files in the latter directory are copied to directory $CATALINA_HOME/webapps/proteios/www/WEB-INF/classes/ when script "./install.sh --force" is run from directory dist/ProteiosSE-*-dev/.

[olle]

(In [4150]) Refs #744. Initialization script www/WEB-INF/init.sh in client/servlet/ updated to create openms.properties file if not existing.

[olle]

(In [4151]) Refs #744. Initialization script www/WEB-INF/init.sh in client/servlet/ updated by removing change to SQL_EXEC settings, that was unintentionally included in changeset ​[4150].

[olle]

(In [4153]) Refs #744. Proteios SE installation and update bat-files for MS Windows updated to support OpenMS properties file openms.properties:

1. Proteios SE installation bat-file install-win.bat in misc/install/ updated to create openms.properties file if not existing.

2. Proteios SE update bat-file update_proteios-win.bat in misc/install/ updated to support OpenMS properties file openms.properties.

[olle]

Design discussion regarding implementation of progress report:

• Inspection of debug output for running OpenMS programs PeakPicker and FeatureFinder revealed that the processing consists of a number of phases, 4 for PeakPicker and 17 for FeatureFinder.
• During the processing step the programs output progress information in the output stream of the type "Progress of 'loading mzML file':" and "18.15 %", where the percentage string indicates the percentage progress of the current phase. The strings starting with "Progress of" will be used to keep track of when the program enters a new phase. The total percentage used as input to the Proteios SE progress reporter will be calculated from the number of finished phases and the percentage progress of the current phase, simply assuming that all phases take the same time to process.
• Since some operations are performed after the OpenMS processing, if the percentage number is > 99%, input to the Proteios SE progress reporter should be set to 99% (the percentage value will automatically be set to 100%, when the job is completed).

[olle]

(In [4157]) Refs #744. Class/file plugins/RunOpenMsPlugin.java in plugin/ updated with progress information during OpenMS feature finding:

1. New private instance variable String openMsProgramSpecifier with public accessor methods added.

2. New private instance variable int currentPhase with public accessor methods added.

3. New private instance variable String currentPhaseStr with public accessor methods added.

4. Private method void execute(Request request, Response response, ProgressReporter progress) updated to set the value of new instance variable String openMsProgramSpecifier to the OpenMS program used.

5. Private method void openMsSearchLocal(Request request, Response response, ProgressReporter progress, ...) updated to call new StreamGobbler constructor for the OpenMS program input stream, and setting the ProgressReporter argument to the value of local variable progress.

6. Private method void updateProgressInformation(ProgressReporter progress, String line) updated to parse the program output for progress information of program phase and percentage progress in the current phase. The information is used to obtain an estimated total percentage by calling private method private int EstimatedTotalPercentage(double percentage, int phase), and the progress reporter data is updated to display new progress information.

7. Private method int EstimatedTotalPercentage(...) updated to use arguments double percentage and int phase. It estimates total percentage of time to completion. assuming a fixed number of phases for each application.

[olle]

(In ​[25]) Proteios SE manual wiki page ​Proteios SE server administration updated by addition of new section on configuration of OpenMS feature detection.

[olle]

(In [4166]) Refs #744. Refs #746. Class/file plugins/RunOpenMsPlugin.java in plugin/ updated in javadoc:

1. Public method void setInputFileSizeInBytes(long inputFileSizeInBytes) updated in javadoc by fix of typo.

[olle]

Ticket closed as a first version of support for running OpenMS programs from Proteios SE has been added.

[olle]

(In [4175]) Refs #749. Refs #668. Refs #744. Refs #356. Refs #287. First version of support for starting feature detection from the menu:

1. Class/file gui/MainMenu.java in client/servlet/ updated:
   a. Private method Node createProjectMenu() updated by adding cascade menu for performing feature detection.

2. New class/file action/file/PerformMsInspectSearch.java in client/servlet/ added. It displays the file selection form and then forwards the action to class UseSpectrumFileForMsInspectSearchExtension.

3. New class/file action/file/PerformOpenMsSearch.java in client/servlet/ added. It displays the file selection form and then forwards the action to class UseSpectrumFileForOpenMsSearchExtension.

4. English dictionary file locale/en/dictionary in client/servlet/ updated with new entries for various string keys.

[olle]

(In [4177]) Refs #748. Refs #749. Refs #744. Running feature detection using OpenMS updated to support disabling the functionality via the configuration file:

1. OpenMS properties template file conf/openms.properties.in in client/servlet/ updated with new option "openms.local.disabled", that should be set to "yes" or "no".

2. Class/file action/file/PerformOpenMsSearch.java in client/servlet/ updated to implement the ExtensionDisableInterface. New public method boolean fetchExtensionDisabledFlag() retrieves the disable flag by calling the fetchExtensionDisabledFlag() method of an instance of UseSpectrumFileForOpenMsSearchExtension.

3. Class/file action/file/UseSpectrumFileForOpenMsSearchExtension.java in client/servlet/ updated to implement the ExtensionDisableInterface. New public method boolean fetchExtensionDisabledFlag() retrieves the disable flag from the OpenMS properties file. If the properties file is missing, true is returned, while if the disable option is missing, false is returned for backwards compatibility.

[olle]

(In ​[27]) Proteios SE manual wiki page ​Proteios SE server administration updated in configuration of OpenMS feature detection with new parameter openms.local.disabled.

[olle]

(In [4181]) Refs #744. Proteios SE update bat-file for MS Windows updated to create OpenMS properties file openms.properties from template file openms.properties.in, if not existing:

1. Proteios SE update bat-file update_proteios-win.bat in misc/install/ updated to create OpenMS properties file openms.properties from template file openms.properties.in, if not existing.

[olle]

(In [4206]) Refs #751. Refs #744. Refs #614. Refs #287. Support for running OpenMS feature detection from Proteios SE updated with option to import the created feature file directly:

1. Class/file action/file/UseSpectrumFileForOpenMsSearchExtension.java in client/servlet/ updated with new check box for option to import created feature file. The check box is coupled to new valid parameter VBoolean RunOpenMs.VOPENMSIMPORTRESULTS. Default is to not import the feature file directly.

2. Class/file action/openMs/RunOpenMs.java in client/servlet/ updated:
   a. New valid parameter VBoolean VOPENMSIMPORTRESULTS added.
   b. Protected method void runMe() updated to retrieve the value of valid parameter VBoolean VOPENMSIMPORTRESULTS, and if the value is true create an import job after the feature detection job. As import of features requires a preceding feature detection job, the code for running a single OpenMS job has been updated to clarify if the job instance is a PeakPicker or FeatureFinder job. The path of the feature file to be created is transferred to the import job as a string job parameter. Import jobs are created by calling new private method Job createOpenMsImportJob(...).
   c. New private method Job createOpenMsImportJob(DbControl dc, String featureFilePathStr, String featureFileName, Job blockerJob) added. It creates an OpenMS import job.

3. English dictionary file locale/en/dictionary in client/servlet/ updated with new string keys.

[olle]

(In [4211]) Refs #761. Refs #751. Refs #744. Refs #287. Main form for running OpenMS feature detection from Proteios SE updated in main form by placing options in different sections, depending on what program the options control:

1. Class/file action/file/UseSpectrumFileForOpenMsSearchExtension.java in client/servlet/ updated in protected method void runMe() by introducing separate sections in main form for general options, PeakPicker options, and FeatureFinder options.

2. English dictionary file locale/en/dictionary in client/servlet/ updated with new string keys.

[olle]

(In [4212]) Refs #761. Refs #744. Support for running OpenMS feature detection from Proteios SE updated by change of names of valid parameters and related variables, in order to make these more readable and logical, in preparation for further additions:

1. Class/file action/openMs/RunOpenMs.java in client/servlet/ updated by change of names of valid parameters and related variables, in order to make these more readable and logical, in preparation for further additions.

2. Class/file action/file/UseSpectrumFileForOpenMsSearchExtension.java in client/servlet/ updated with new names for valid parameters in RunOpenMs and related variables.

[olle]

(In [4213]) Refs #761. Refs #744. Refs #287. First version of support for new OpenMS FeatureFinder options for algorithm "centroided":

1. Class/file action/file/UseSpectrumFileForOpenMsSearchExtension.java in client/servlet/ updated:
   a. Protected method void runMe() updated to add input fields for the new OpenMS options in the OpenMS form. The input fields are coupled to new valid parameters in class RunOpenMs.

2. Class/file action/openMs/RunOpenMs.java in client/servlet/ updated:
   a. New valid parameters added for the OpenMS options to be supported, together with constants for default values of the new parameters.
   b. Protected method void runMe() updated to obtain values of new options from valid parameters and transfer them to private method Job createOpenMsJob(DbControl dc, ...) in new arguments.
   c. Private method Job createOpenMsJob(DbControl dc, ...) updated with new arguments for the OpenMS options to support. The parameter values are transferred to the created job as job parameters.

3. Class/file plugins/RunOpenMsPlugin.java in plugin/ updated:
   a. Private method void execute(Request request, Response response, ProgressReporter progress) updated to obtain values of new parameters from job parameters and transfer them to private method void openMsSearchLocal(...).
   b. Private method void openMsSearchLocal(...) updated with arguments for new parameters and variables. It calls new private method String createOpenMsIniFile(...) to create an optional temporary OpenMS INI file. If an INI file is created, it is given as input option on the command line when starting the OpenMS program.
   c. New private method String createOpenMsIniFile(...) added. It calls new private method void updateTextFile(...) to adapt a copy of new OpenMS INI template file openms_featurefinder_centroided_ini_file_template.ini to include values of given OpenMS program parameter values.
   d. New private method void updateTextFile(String sourceFileURLStr, String targetDir, String targetFile, List<String> stringReplaceList) added. It updates a copy of a file on the server file system by replacing key strings with new text.
   e. New private convenience method void writeLine(FileWriter fw, String string) added to support creation of text files. Various instance variables with accessor methods also added to support this task.

4. New OpenMS INI template file content/openms_featurefinder_centroided_ini_file_template.ini in client/servlet/ added. It contains an OpenMS INI file for running FeatureFinder with algorithm "centroided", but with values for selected parameters exchanged for unique text markers.

5. English dictionary file locale/en/dictionary in client/servlet/ updated with new entries for various string keys.

[Fredrik Levander]

(In [4374]) Refs #744. Refs #781. Fix for bug with multiple files when running both peakpicking and featurefinding.

[Fredrik Levander]

(In [4375]) Refs #744. Refs #781. Fix for another bug with multiple files when running both peakpicking and featurefinding.