Opened 11 years ago

Closed 10 years ago

Last modified 10 years ago

#1592 closed task (fixed)

Unified installation procedure for plug-ins, extensions and more...

Reported by: Nicklas Nordborg Owned by: Nicklas Nordborg
Priority: major Milestone: BASE 3.0
Component: core Version:
Keywords: Cc:

Description (last modified by Nicklas Nordborg)

The installation procedure is quite different for plug-ins, extensions and other external code. For example:

  • Plug-ins can be installed anywhere, but it is recommended to place them in the folder specified by the 'plugins.dir' setting in the base.config file.
  • Extensions must be installed in the WEB-INF/extensions folder.
  • Some types of "plug-ins" eg. authentication and logging plug-ins are specified in the 'base.config' file but must be located in the WEB-INF/lib folder.
  • File validator plug-ins (eg. cdf/cel files) are configured in the database and can be located anywhere but the recommended location is the 'plugins.dir' folder.

Needless to say, this is a headache for packages that need to utilize several types of plug-ins+extensions. For example, the Illumina package (http://baseplugins.thep.lu.se/wiki/net.sf.basedb.illumina).

I think we need a different approach that is simpler for administrators. Eg. a single directory should be used for all types of plug-ins/extensions. BASE should be able to scan this directory and, using metadata inside the JAR, it should be possible to determine what it is. A single JAR file should of course be able to hold multiple types of plug-ins+extensions. Installation should be triggered from a single place in the web gui. It would also be nice if an initialization script could be executed directly. Using the Illumina package as an example I think steps 7-14 could be reduced to just one or two steps.

One big issue is to not break an existing BASE installation... I guess we'll have to think about that.

Related tickets:

  • #1593: Extension system for the core API
  • #1594: JarClassLoader support for JARs within JARs
  • #1598: Use the extensions system for data file validators and metadata readers
  • #1582: Extended support for external files

Change History (28)

comment:1 Changed 11 years ago by Nicklas Nordborg

Description: modified (diff)

comment:2 Changed 11 years ago by Nicklas Nordborg

Description: modified (diff)

comment:3 Changed 11 years ago by Nicklas Nordborg

My current idea is to make things a bit more restrictive:

  • The plugins.dir setting must be specified in base.config.
  • All plug-ins/extensions must be installed in this folder (except core plug-ins which are installed in WEB-INF/lib).
  • With #1594 in place a single JAR file should be enough for most plug-ins/extensions.
  • Configuration files are placed in WEB-INF/classes to allow class-path lookup.
  • Metadata+installation information about the plug-in/extension is kept in XML files in the META-INF directory inside the JAR file. This can already be done, but the XML format is different for plug-ins vs. extensions and information has to be repeated if a package contains both. This should be fixed.
  • Installation of plug-ins/extensions should be made from a single place in the web interface. The manual registration of a single plug-in should be removed and the plug-in installation wizard should be merged with the extensions installation wizard.

As far as I can tell this should not be too difficult to fix. When upgrading an existing server the admin must ensure that all plug-ins and extensions are located in the specified directory and move them if not. The upgrade script (#1591) should remove path information from from the PluginDefinitions.jar_path column (maybe we should create a new column, jar_file, instead and remove the old column?).

comment:4 Changed 11 years ago by Nicklas Nordborg

Owner: changed from everyone to Nicklas Nordborg
Status: newassigned

comment:5 Changed 11 years ago by Nicklas Nordborg

(In [5595]) References #1592: Unified installation procedure for plug-ins, extensions and more...

Application will check that plugins.dir is set and exists at startup and exists with an error message otherwise.

Removed PluginDefinition.jarPath property and replaced with PluginDefinition.jarFile. No path information is allowed in this property. Update script must copy the filename only from jar_path column in PluginDefinitions to jar_file and then remove the jar_path column.

Removed JobAgentSettings.jarPath property since it no longer makes sense to override this property. The update script should remove the jar_path column from the JobAgentSettings table.

Removed/replaced several methods from the PluginDefinition class that allows overriding the JAR path to a plug-in. External applications or plug-ins no longer has to worry about the path, but they may need to be recompiled to use the new methods.

The DataFileType has not yet been fully fixed and may not work as expected. The idea is to replace the classname+jarpath properties with some kind of extension system (#1593) similar to external file support (#1582).

In the web gui, places that are handling JAR path information has been fixed. For job agents it has been removed completely since it is no longer needed. The auto-installation wizard is probably not working at the moment. The update script should remove references to the jarPath property in context information (list filters, sort order, etc.).

comment:6 Changed 11 years ago by Nicklas Nordborg

(In [5596]) References #1592: Unified installation procedure for plug-ins, extensions and more...

When a new instance of a plug-in is created through the BASE core API using the PluginDefinition.newInstance(Class) and the given class wasn't loaded by the default classloader, we'll try to use the classloader of the given class first to try to avoid ClassCastExceptions later.

This means that the second statement below now has a chance to succeed even if the first fails.

  • X x = (X)plugin.newInstance();
  • X x = plugin.newInstance(X.class);

comment:7 Changed 10 years ago by Nicklas Nordborg

Description: modified (diff)

comment:8 Changed 10 years ago by Nicklas Nordborg

(In [5601]) References #1592: Unified installation procedure for plug-ins, extensions and more... References #1593: Extension system for the core API

Started to move the web client extensions into the core registry. So far the extensions are loaded at startup, but there is no support for adding more at runtime. The administrative interface is a bit of a mess right now, since a lot of the metadata that we used to get from the registration process is not yet available.

comment:9 Changed 10 years ago by Nicklas Nordborg

(In [5602]) References #1592: Unified installation procedure for plug-ins, extensions and more... References #1593: Extension system for the core API

Added fuctionality for metadata handling and status/error reporting. Everything is still only loaded at startup but now we can see most of the information in the admin interface.

comment:10 Changed 10 years ago by Nicklas Nordborg

(In [5603]) References #1592: Unified installation procedure for plug-ins, extensions and more... References #1593: Extension system for the core API

Added support for adding and removing extensions at runtime. There are still some minor issues to fix since it seems difficult to get the various processors to do exactly what is needed. There are also some issues when an updated version of an extension is installed over an old one.

Added a lot more statistics to both the manager and processors. It is now possible to get almost as much information as before.

comment:11 Changed 10 years ago by Nicklas Nordborg

(In [5606]) References #1593: Extension system for the core API

References #1592: Unified installation procedure for plug-ins, extensions and more...

Servlets are now processed and loaded by the extensions system. Moved handling of settings to the core. Removed classes that are no longer used.

comment:12 Changed 10 years ago by Nicklas Nordborg

(In [5610]) References #1592: Unified installation procedure for plug-ins, extensions and more...

Moved plug-in definitions into the extension definitions file. This means that we can get rid of the Plugin.getAbout() method since that information is now provided in the XML file instead. I have started to move the core plug-in definitions into the core-plugins.xml file (only a few so far).

The base-plugins.dtd was removed since it is no longer used. External plug-in packages that use base-plugins.xml must be updated to use the new format.

comment:13 Changed 10 years ago by Nicklas Nordborg

(In [5612]) References #1592: Unified installation procedure for plug-ins, extensions and more...

Moved remaining core plug-in definitions to the xml file. Removed the getAbout() method from all plugins.

comment:14 Changed 10 years ago by Nicklas Nordborg

(In [5613]) References #1592: Unified installation procedure for plug-ins, extensions and more...

Changed jarPath property of PluginType to jarFile (same as has already been done for PluginDefintion).

comment:15 Changed 10 years ago by Nicklas Nordborg

(In [5615]) References #1592: Unified installation procedure for plug-ins, extensions and more...

Added file processor for installing plug-ins as part of a scan for extensions. Trying to link information about plug-in into the tree with extensions information.

Added "disabled" property to plug-ins to make them behave more similar to extensions. There may still be synchronization issues with the in-memory information and the database information.

Changed the installation procedure somewhat to allow re-using the same code for installing plug-ins (eg. from initdb/updatedb, and scanning extensions). There are some bumps to fix yet.

comment:16 Changed 10 years ago by Nicklas Nordborg

(In [5616]) References #1592: Unified installation procedure for plug-ins, extensions and more...

Re-organized the installation procedure. The first step will now show scan for new/updated/deleted files and present a list of options for each one (eg. install/uninstall).

We keep track of each installed extension in the settings file which should be safer since no automatic installation is done any longer.

There may still be some error handling issues, and plug-in configuration are still not installed.

Re-organized the Administrate and Extensions menus in the web client. All plug-in/extensions installation is now done from Administrate -> Plug-ins & extensions -> Overview.

comment:17 Changed 10 years ago by Nicklas Nordborg

(In [5617]) References #1592: Unified installation procedure for plug-ins, extensions and more...

Added support for installing plug-in configurations. The file we are looking for has changed to META-INF/plugin-configurations.xml

Moved duplicates of the installation code to the PluginConfiguration class (from the Install class and PluginConfigurationImporter).

comment:18 Changed 10 years ago by Nicklas Nordborg

Resolution: fixed
Status: assignedclosed

All functions from the old extension system should now be working. I'll close this ticket for now and hope that no more major issues should appear.

comment:19 Changed 10 years ago by Nicklas Nordborg

(In [5637]) References #1592: Unified installation procedure for plug-ins, extensions and more...

Added missing 'PrintMapFlatFileImporter?'.

comment:20 Changed 10 years ago by Nicklas Nordborg

(In [5671]) References #1592: Unified installation procedure for plug-ins, extensions and more...

Updated UML diagram.

comment:21 Changed 10 years ago by Nicklas Nordborg

(In [5715]) References #1592: Unified installation procedure for plug-ins, extensions and more...

Added icons for flagging modified JAR/XML files to make it easier to spot that something has been updated already in the tree overview.

comment:22 Changed 10 years ago by Nicklas Nordborg

Resolution: fixed
Status: closedreopened

I found an issue with the XJspCompiler when testing the example plug-ins and extensions. The compiler is still looking for extension JAR:s in the WEB-INF/extensions directory. It must be updated to use the plugins.dir folder as configured in BASE. It is not trivial since the Application class is not on the class path, but I think we can use reflection just as we do when getting the JarClassLoader instances.

comment:23 Changed 10 years ago by Nicklas Nordborg

Resolution: fixed
Status: reopenedclosed

(In [5746]) Fixes #1592: Unified installation procedure for plug-ins, extensions and more...

Fixed the problem with the XJspCompiler.

comment:24 Changed 10 years ago by Nicklas Nordborg

(In [5760]) References #1592: Unified installation procedure for plug-ins, extensions and more...

Sort almost everything by name in the overview tree.

comment:25 Changed 10 years ago by Nicklas Nordborg

(In [5775]) References #1592: Unified installation procedure for plug-ins, extensions and more...

Do not show information about depracted plug-ins that hasn't been registered in the database.

comment:26 Changed 10 years ago by Nicklas Nordborg

(In [5777]) References #1592: Unified installation procedure for plug-ins, extensions and more...

Should have been committed as part of [5755].

comment:27 Changed 10 years ago by Nicklas Nordborg

(In [5821]) References #1592: Unified installation procedure for plug-ins, extensions and more...

Display the plug-in type in the overview tree.

comment:28 Changed 10 years ago by Nicklas Nordborg

(In [5835]) References #1592: Unified installation procedure for plug-ins, extensions and more...

Added a max height to the table listing extensions so that the buttons doesn't scroll out of view.

Note: See TracTickets for help on using tickets.