Changeset 2932


Ignore:
Timestamp:
Nov 17, 2006, 12:29:27 PM (16 years ago)
Author:
Nicklas Nordborg
Message:

Fixes #140: Write documentation: 5e) Secondary file storage

Also implemented the missing option for specifying time-point values.

Location:
trunk
Files:
6 edited

Legend:

Unmodified
Added
Removed

  • trunk/doc/admin/base.config.html

    r2648 r2932  
    5151  <b>Contents</b><br>
    5252  <ol>
    53   <li>TODO
     53  <li>More...
     54  <li><a href="#secondary">Secondary storage controller</a></li>
    5455  </ol>
    5556
    … …  
    6263</div>
    6364
    64   <h2>1. Settings</h2>
    6565
    66   <table border="1" cellspacing="0" cellpadding="2" width="100%">
    67   <tr>
    68     <th>Setting</th>
    69     <th>Default value</th>
    70     <th>Description</th>
    71   </tr>
     66  <a name="secondary"></a>
     67  <h2>2. Secondary storage controller</h2>
     68  <p>
     69    Settings related to the secondary storage controller. For more
     70    information about the secondary storage read the <a
     71    href="../development/plugins/storage/index.html">Plugins for secondary
     72    file storage</a> document.
     73  </p>
    7274 
    73 
    74   </table>
     75  <dl>
     76  <dt class="method">secondary.storage.driver</dt>
     77  <dd>
     78    The class name of the plugin that acts as the secondary storage
     79    controller. BASE ships with a simple plugin that just moves
     80    files to another directory, but it is not enabled by default.
     81    The class name of that plugin is
     82    <code>net.sf.basedb.core.InternalStorageController</code>. If no class
     83    is specified the secondary storage feature is disabled.
     84  </dd>   
     85  <dt class="method">secondary.storage.init</dt>
     86  <dd>
     87    Initialisation paramters sent to the plugin by calling the
     88    <code>init()</code> method. The syntax and meaning of this string
     89    depends on the plugin. For the internal controller this is simply
     90    the path to the seconday directory.
     91  </dd>
     92 
     93  <dt class="method">secondary.storage.interval</dt>
     94  <dd>
     95    Interval in seconds between each execution of the controller plugin. It
     96    must be a value greater than zero or the secondary storage feature will
     97    be disabled.
     98  </dd>
     99 
     100  <dt class="method">secondary.storage.time</dt>
     101  <dd>
     102    Time-point values specifying the time(s) of day that the controller
     103    should be executed. This setting overrides the interval setting if present.
     104    Time-point values are given as comma-separted list of two-digit 24-based
     105    hour and two-digit minute values. For example: 03:10,09:00,23:59.
     106  </dd>
     107 
     108  </dl>
    75109
    76110</body>

  • trunk/doc/development/plugins/storage/index.html

    r2304 r2932  
    4343  <div class="abstract">
    4444 
    45     TODO
     45    This document describes how to create a plugin for managing
     46    the secondary file storage.
    4647 
    4748    <p>
    … …  
    4950    </p>
    5051    <ol>
    51       <li>
    52 
     52      <li><a href="#overview">Primary vs. secondary storage</a></li>
     53      <li><a href="#secondary">The SecondaryStorageController interface</a></li>
     54      <li><a href="#config">Comnfiguration settings</a></li>
    5355    </ol>
    5456    <p>
    5557    <b>See also</b>
    5658    <ul>
    57     <li>
     59    <li><a href="../../../admin/base.config.html#seconary">base.config reference:
     60      Secondary storage section</a>
    5861    </ul>
    5962    </p>
    … …  
    6467  </div>
    6568
     69
     70  <a name="overview"></a>
     71  <h2>1. Primary vs. secondary storage</h2>
     72  <p>
     73    BASE has support for storing files in two locations, the primary storage and
     74    the secondary storage. The primary storage is always disk-based and must be
     75    accessible by the BASE server as a path on the file system. The path to the
     76    primary storage is configured by the <code>userfiles</code> setting in the
     77    <code>base.config</code> file. The primary storage is internal to the core.
     78    Client applications don't get access to read or manipulate the files directly
     79    from the file system.
     80  </p>
     81 
     82  <p>
     83    The secondary storage can be anything that can store files. It could for
     84    example be another directory, a remote FTP server, or a tape based
     85    archiving system. A file located in the secondary storage is not accessible
     86    by the core or client applications. The secondary storage can only be accessed
     87    by the secondary storage controller and the core (and client) applications uses
     88    flags on the <code>File</code> item to handle the interaction with the secondary
     89    storage.
     90  </p>
     91
     92  <p>
     93    Each file has an <code>action</code> attribute which default's to
     94    <code>File.Action.NOTHING</code>. It can take two other values:
     95  </p>
     96 
     97  <ol>
     98  <li><code>File.Action.MOVE_TO_SECONDARY</code></li>
     99  <li><code>File.Action.MOVE_TO_PRIMARY</code></li>
     100  </ol>
     101 
     102  <p>
     103    All files with the action attribute set to <code>MOVE_TO_SECONDARY</code>
     104    should be moved to the secondary storage by the controller, and all files
     105    with the action attribute set to <code>MOVE_TO_PRIMARY</code> should be
     106    brought back to primary storage.
     107  </p>
     108 
     109  <p>
     110    The moving of files between primary and secondary storage doesn't happen
     111    immediately. It is up to the server administrator to configure how often
     112    and at what times the controller should check for files that should be moved.
     113    This is configured by the <code>secondary.storage.interval</code> and
     114    <code>secondary.storage.time</code> settings in the <code>base.config</code>
     115    file.
     116  </p>
     117
     118  <a name="secondary"></a>
     119  <h2>2. The <code>SecondaryStorageController</code> interface</h2>
     120
     121  <p>
     122    All you have to do to create a secondary storage controller is
     123    to create a class that implements the <code>SecondaryStorageController</code>
     124    interface. In you <code>base.config</code> file you then specify the
     125    class name in the <code>secondary.storage.driver</code> setting and it's
     126    initialisation parameters in the <code>secondary.storage.driver</code>
     127    setting.
     128  </p>
     129 
     130  <p>
     131    Your class must have a public no-argument constructor. The BASE application will
     132    create only one instance of the class for lifetime of the BASE server.
     133    Here are the method that you must implement:
     134  </p>
     135 
     136  <dl>
     137  <dt class="method">public void init(String settings);</dt>
     138  <dd>
     139    This method is called just after the object has been created with
     140    it's argument taken from the <code>secondary.storage.driver</code>
     141    setting in your <code>base.config</code> file. This method is only
     142    called once for an object.
     143  </dd>
     144 
     145  <dt class="method">public void run();</dt>
     146  <dd>
     147    This method is called whenever the core thinks it is time to
     148    do some management of the secondary storage. How often the run() method
     149    is called is controlled by the <code>secondary.storage.interval</code> and
     150    <code>secondary.storage.time</code> settings in the <code>base.config</code> file.
     151    <p>
     152      When this method is called the controller should:
     153    </p>
     154 
     155    <ul>
     156    <li>Move all files which has <code>action=MOVE_TO_SECONDARY</code>
     157      to the secondary storage. When the file has been moved call
     158      <code>File.setLocation(Location.SECONDARY)</code> to tell the core
     159      that the file is now in the secondary storage. You should also call
     160      <code>File.setAction(File.Action.NOTHING)</code> to reset the action
     161      attribute.
     162
     163    <li>Restore all files which has <code>action=MOVE_TO_PRIMARY</code>.
     164      The core will set the location attribute automatically, but
     165      you should call <code>File.setAction(File.Action.NOTHING)</code> to reset the
     166      action attribute.
     167   
     168    <li>Delete all files from the secondary storage that are not present in the
     169      database with <code>location=Location.SECONDARY</code>. This includes files
     170      which has been deleted and files that have been moved offline or re-uploaded.
     171
     172    </ul>
     173 
     174    <p>
     175      As a final act the method should send a message to each user
     176      owning files that has been moved from one location to the other. The
     177      message should include a list of files that has been moved to the
     178      secondary storage and a list of files moved from the secondary storage
     179      and a list of files that has been deleted due to some of the reasons above.
     180    </p>
     181  </dd>
     182
     183  <dt class="method">public void close();</dt>
     184  <dd>
     185    This method is called when the server is closing down. After this the
     186    object is never used again.
     187  </dd>
     188  </dl>
     189 
     190  <a name="config"></a>
     191  <h2>3. Configuration setting</h2>
     192
     193  <p>
     194    The configuration settings for the secondary storage controller is
     195    location in the <code>base.config</code> file. Here is an overview of
     196    the settings. For more information read the
     197    <a href="../../../admin/base.config.html#seconary">base.config reference</a>
     198  </p>
     199 
     200  <dl>
     201  <dt>secondary.storage.driver</dt>
     202  <dd>
     203    The class name of the secondary storage plugin.
     204  </dd>
     205 
     206  <dt>secondary.storage.init</dt>
     207  <dd>
     208    Initialisation paramters sent to the plugin by calling the
     209    <code>init()</code> method.
     210  </dd>
     211 
     212  <dt>secondary.storage.interval</dt>
     213  <dd>
     214    Interval in seconds between each execution of the controller plugin.
     215  </dd>
     216 
     217  <dt>secondary.storage.time</dt>
     218  <dd>
     219    Time points per day when the controller plugin should be executed.
     220  </dd>
     221  </dl>
     222 
     223 
    66224</body>
    67225</html>

  • trunk/src/base.config.in

    r2648 r2932  
    125125# secondary.storage.interval = 3600
    126126
     127# Comma-separated list of time values (hours:minutes) when the controller
     128# should be executed; overrides the interval setting if present
     129# secondary.storage.time = 18:15,07:30
     130
    127131# ===============
    128132# General section

  • trunk/src/core/net/sf/basedb/core/Application.java

    r2822 r2932  
    2929import net.sf.basedb.util.timer.Scheduler;
    3030
     31import java.util.Calendar;
     32import java.util.GregorianCalendar;
    3133import java.util.Map;
    3234import java.util.HashMap;
    … …  
    316318    log.info("Starting BASE");
    317319
    318     // Read settings from the configuration file
    319     Config.init();
    320     authenticationDriver = Config.getString("auth.driver");
    321     log.info("auth.driver = " + authenticationDriver);
    322    
    323     secondaryStorageDriver = Config.getString("secondary.storage.driver");
    324     log.info("secondary.storage.driver = " + secondaryStorageDriver);
    325    
    326     sessionCacheTimeout = Config.getInt("cache.timeout", 20); // minutes
    327     log.info("cache.timeout = " + sessionCacheTimeout + " minutes");
    328    
    329     permissionTimeout = Config.getInt("permission.timeout", 10); // minutes
    330     log.info("permission.timeout = " + permissionTimeout + " minutes");
    331    
    332     userFilesDirectory = new java.io.File(Config.getString("userfiles"));
    333     log.info("userfiles = " + userFilesDirectory);
    334    
    335     log.info("db.dialect = " + Config.getString("db.dialect"));
    336     log.info("db.driver = " + Config.getString("db.driver"));
    337     log.info("db.url = " + Config.getString("db.url"));
    338     log.info("db.username = " + Config.getString("db.username"));
    339     log.debug("db.password = " + Config.getString("db.password"));
    340    
    341     queryFile = Config.getString("db.queries");
    342     log.info("db.queries = " + queryFile);
    343    
    344     extendedPropertiesFile = Config.getString("db.extended-properties");
    345     log.info("db.extended-properties = " + extendedPropertiesFile);
    346    
    347     rawDataTypesFile = Config.getString("db.raw-data-types");
    348     log.info("db.raw-data-types = " + rawDataTypesFile);
    349    
    350     dynamicCatalog = Config.getString("db.dynamic.catalog");
    351     log.info("db.dynamic.catalog = " + dynamicCatalog);
    352    
    353     dynamicSchema = Config.getString("db.dynamic.schema");
    354     log.info("db.dynamic.schema = " + dynamicSchema);
    355     log.info("db.batch-size = " + Config.getString("db.batch-size"));
    356 
    357     // Create a cache for SessionControl objects
    358     sessionCache = Collections.synchronizedMap(new HashMap<String,SessionControl>());
    359    
    360     // Initialise other utility classes
    361     ExtendedProperties.init();
    362     RawDataTypes.init();
    363     RawDataUtil.init();
    364     HibernateUtil.init();
    365     QueryRuntimeFilterFactory.init();
    366     PredefinedQuery.init();
    367     BatchUtil.init();
    368     Metadata.init();
    369 
    370     if (verifySchemaVersion)
    371     {
    372       int schemaVersion = getSchemaVersion();
    373       if (schemaVersion != Install.NEW_SCHEMA_VERSION)
     320    try
     321    {
     322      // Read settings from the configuration file
     323      Config.init();
     324      authenticationDriver = Config.getString("auth.driver");
     325      log.info("auth.driver = " + authenticationDriver);
     326     
     327      secondaryStorageDriver = Config.getString("secondary.storage.driver");
     328      log.info("secondary.storage.driver = " + secondaryStorageDriver);
     329     
     330      sessionCacheTimeout = Config.getInt("cache.timeout", 20); // minutes
     331      log.info("cache.timeout = " + sessionCacheTimeout + " minutes");
     332     
     333      permissionTimeout = Config.getInt("permission.timeout", 10); // minutes
     334      log.info("permission.timeout = " + permissionTimeout + " minutes");
     335     
     336      userFilesDirectory = new java.io.File(Config.getString("userfiles"));
     337      log.info("userfiles = " + userFilesDirectory);
     338     
     339      log.info("db.dialect = " + Config.getString("db.dialect"));
     340      log.info("db.driver = " + Config.getString("db.driver"));
     341      log.info("db.url = " + Config.getString("db.url"));
     342      log.info("db.username = " + Config.getString("db.username"));
     343      log.debug("db.password = " + Config.getString("db.password"));
     344     
     345      queryFile = Config.getString("db.queries");
     346      log.info("db.queries = " + queryFile);
     347     
     348      extendedPropertiesFile = Config.getString("db.extended-properties");
     349      log.info("db.extended-properties = " + extendedPropertiesFile);
     350     
     351      rawDataTypesFile = Config.getString("db.raw-data-types");
     352      log.info("db.raw-data-types = " + rawDataTypesFile);
     353     
     354      dynamicCatalog = Config.getString("db.dynamic.catalog");
     355      log.info("db.dynamic.catalog = " + dynamicCatalog);
     356     
     357      dynamicSchema = Config.getString("db.dynamic.schema");
     358      log.info("db.dynamic.schema = " + dynamicSchema);
     359      log.info("db.batch-size = " + Config.getString("db.batch-size"));
     360 
     361      // Create a cache for SessionControl objects
     362      sessionCache = Collections.synchronizedMap(new HashMap<String,SessionControl>());
     363     
     364      // Initialise other utility classes
     365      ExtendedProperties.init();
     366      RawDataTypes.init();
     367      RawDataUtil.init();
     368      HibernateUtil.init();
     369      QueryRuntimeFilterFactory.init();
     370      PredefinedQuery.init();
     371      BatchUtil.init();
     372      Metadata.init();
     373 
     374      if (verifySchemaVersion)
    374375      {
    375         String msg = "Schema version mismatch: database schema version=" +
    376           schemaVersion + "; expected=" + Install.NEW_SCHEMA_VERSION;
     376        int schemaVersion = getSchemaVersion();
     377        if (schemaVersion != Install.NEW_SCHEMA_VERSION)
     378        {
     379          String msg = "Schema version mismatch: database schema version=" +
     380            schemaVersion + "; expected=" + Install.NEW_SCHEMA_VERSION;
     381         
     382          if (schemaVersion < Install.NEW_SCHEMA_VERSION)
     383          {
     384            msg += "; Please update the database with the 'updatedb.sh' script before starting BASE.";
     385          }
     386          else
     387          {
     388            msg += "; Please update the BASE code before starting.";
     389          }
     390          log.error(msg);
     391          throw new BaseException(msg);
     392        }
     393      }
     394     
     395      if (!installation)
     396      {
     397        SystemItems.init();
     398        Keyring.init();
     399        HibernateUtil.testTransactions();
     400 
     401        // Adding a task that cleans the session control cache at regular intervale
     402        long milliSeconds = 60*1000*sessionCacheTimeout;
     403        getScheduler().schedule(new SessionControlCacheCleaner(), milliSeconds, milliSeconds, false);
    377404       
    378         if (schemaVersion < Install.NEW_SCHEMA_VERSION)
     405        if (useInternalJobQueue == null) useInternalJobQueue = Config.getBoolean("jobqueue.internal.enabled");
     406        if (useInternalJobQueue)
    379407        {
    380           msg += "; Please update the database with the 'updatedb.sh' script before starting BASE.";
     408          long checkInterval = 1000 * Config.getInt("jobqueue.internal.checkinterval", 30);
     409          internalJobQueue = new InternalJobQueue();
     410          getScheduler().schedule(internalJobQueue, checkInterval, checkInterval, false);
    381411        }
    382         else
     412       
     413        if (hasSecondaryStorage())
    383414        {
    384           msg += "; Please update the BASE code before starting.";
    385         }
    386         log.error(msg);
    387         throw new BaseException(msg);
    388       }
    389     }
    390    
    391     if (!installation)
    392     {
    393       SystemItems.init();
    394       Keyring.init();
    395       HibernateUtil.testTransactions();
    396 
    397       // Adding a task that cleans the session control cache at regular intervale
    398       long milliSeconds = 60*1000*sessionCacheTimeout;
    399       getScheduler().schedule(new SessionControlCacheCleaner(), milliSeconds, milliSeconds, false);
    400      
    401       if (useInternalJobQueue == null) useInternalJobQueue = Config.getBoolean("jobqueue.internal.enabled");
    402       if (useInternalJobQueue)
    403       {
    404         long checkInterval = 1000 * Config.getInt("jobqueue.internal.checkinterval", 30);
    405         internalJobQueue = new InternalJobQueue();
    406         getScheduler().schedule(internalJobQueue, checkInterval, checkInterval, false);
    407       }
    408      
    409       if (hasSecondaryStorage())
    410       {
    411         long checkInterval = 1000 * Config.getInt("secondary.storage.interval", 0);
    412         secondaryStorageController = getSecondaryStorageController();
    413         TimerTask sss = new TimerTask()
    414         {
    415           public void run()
     415          long checkInterval = Config.getInt("secondary.storage.interval", 0);
     416          String timePoints = Config.getString("secondary.storage.time");
     417          TimerTask sss = new SecondaryStorageControllerTask(getSecondaryStorageController());
     418          Scheduler s = getScheduler();
     419          if (timePoints != null)
    416420          {
    417             secondaryStorageController.run();
    418           }
    419         };
    420         getScheduler().schedule(sss, checkInterval, checkInterval, false);
    421       }
    422      
    423       // Add shutdown hook so we can clean up if System.exit is called.
    424       Runtime.getRuntime().addShutdownHook(
    425         new Thread(
    426           new Runnable()
    427           {
    428             public void run()
     421            Calendar c = new GregorianCalendar();
     422            long EVERY_DAY = 24l * 3600l * 1000l;
     423            Pattern p = Pattern.compile("(\\d\\d):(\\d\\d)");
     424            for (String timePoint : timePoints.split(","))
    429425            {
    430               log.info("System is shutting down. Cleaning up as much as possible");
    431               Application.stop();
     426              Matcher m = p.matcher(timePoint);
     427              if (m.matches())
     428              {
     429                int hour = Integer.parseInt(m.group(1));
     430                int minute = Integer.parseInt(m.group(2));
     431                log.info("Scheduling secondary storage driver at " + timePoint + " every day");
     432                c.set(Calendar.HOUR_OF_DAY, hour);
     433                c.set(Calendar.MINUTE, minute);
     434                c.set(Calendar.SECOND, 0);
     435                s.scheduleAtFixedRate(sss, c.getTime(), EVERY_DAY, false);
     436              }
     437              else
     438              {
     439                log.warn("Time-point " + timePoint + " is not a valid value; must match hh:mm");
     440              }
    432441            }
    433442          }
    434         )
    435       );
    436     }
    437     log.info("BASE is up and running");
    438     isRunning = true;
    439   }
    440 
     443          else if (checkInterval > 0)
     444          {
     445            log.info("Scheduling secondary storage driver every " + checkInterval + " seconds");
     446            s.schedule(sss, 1000 * checkInterval, 1000 * checkInterval, false);
     447          }
     448        }
     449       
     450        // Add shutdown hook so we can clean up if System.exit is called.
     451        Runtime.getRuntime().addShutdownHook(
     452          new Thread(
     453            new Runnable()
     454            {
     455              public void run()
     456              {
     457                try
     458                {
     459                  log.info("System is shutting down. Cleaning up as much as possible");
     460                }
     461                catch (Throwable t)
     462                {}
     463                Application.stop();
     464              }
     465            }
     466          )
     467        );
     468      }
     469      log.info("BASE is up and running");
     470      isRunning = true;
     471    }
     472    catch (Throwable t)
     473    {
     474      log.error("Exception when starting...", t);
     475      isRunning = true; // Or the stop() method won't do anything
     476      stop();
     477      throw new BaseException(t);
     478    }
     479  }
    441480
    442481  /**
    … …  
    834873    public void run()
    835874    {
     875      log.info("Cleaning session control cache");
    836876      cleanSessionControlCache(false);
    837877    }
    838878    // -------------------------------------------
    839879  }
     880 
     881  private static class SecondaryStorageControllerTask
     882    extends TimerTask
     883  {
     884   
     885    private final SecondaryStorageController ssc;
     886    private boolean isRunning;
     887   
     888    private SecondaryStorageControllerTask(SecondaryStorageController ssc)
     889    {
     890      this.ssc = ssc;
     891      this.isRunning = false;
     892    }
     893   
     894    /*
     895      From the TimerTask class
     896      -------------------------------------------
     897    */
     898    public void run()
     899    {
     900      if (isRunning) return;
     901      isRunning = true;
     902      log.info("Executing secondary storage controller: " + ssc);
     903      ssc.run();
     904      isRunning = false;
     905    }
     906    // -------------------------------------------
     907  }
     908 
    840909}

  • trunk/src/core/net/sf/basedb/core/InternalStorageController.java

    r2304 r2932  
    9090  */
    9191  private static final org.apache.log4j.Logger log =
    92     org.apache.log4j.LogManager.getLogger("net.sf.basedb.core.storage");
     92    org.apache.log4j.LogManager.getLogger("net.sf.basedb.core.storage.InternalStorageController");
    9393
    9494  /**

  • trunk/src/core/net/sf/basedb/core/SecondaryStorageController.java

    r2304 r2932  
    3939  <code>secondary.storage.interval</code> setting. The interval is given
    4040  in seconds. Alternatively the setting <code>secondary.storage.time</code>
    41   may contain a comma-separated list of time-points when the storage controller
    42   should be invoked (TODO - this is not yet implemented).
     41  may contain a comma-separated list of time-points (hour:minutes) when the storage
     42  controller should be invoked. If time-points are given it overrides the interval
     43  setting which is ignored. Time-point valus must be given with time two-digit 24-based
     44  hour and two-digit minute values. For example: 03:10,09:00,23:59.
    4345  <p>
    4446  When BASE is stopped the {@link #close()} method is called.
Note: See TracChangeset for help on using the changeset viewer.