| 1 | <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd"> |
|---|
| 2 | <!-- |
|---|
| 3 | $Id: jobqueue.html 4509 2008-09-11 20:01:44Z jari $ |
|---|
| 4 | |
|---|
| 5 | Copyright (C) 2005 Jari Hakkinen, Nicklas Nordborg |
|---|
| 6 | Copyright (C) 2006 Jari Hakkinen |
|---|
| 7 | |
|---|
| 8 | This file is part of BASE - BioArray Software Environment. |
|---|
| 9 | Available at http://base.thep.lu.se/ |
|---|
| 10 | |
|---|
| 11 | BASE is free software; you can redistribute it and/or |
|---|
| 12 | modify it under the terms of the GNU General Public License |
|---|
| 13 | as published by the Free Software Foundation; either version 3 |
|---|
| 14 | of the License, or (at your option) any later version. |
|---|
| 15 | |
|---|
| 16 | BASE is distributed in the hope that it will be useful, |
|---|
| 17 | but WITHOUT ANY WARRANTY; without even the implied warranty of |
|---|
| 18 | MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the |
|---|
| 19 | GNU General Public License for more details. |
|---|
| 20 | |
|---|
| 21 | You should have received a copy of the GNU General Public License |
|---|
| 22 | along with BASE. If not, see <http://www.gnu.org/licenses/>. |
|---|
| 23 | --> |
|---|
| 24 | <html> |
|---|
| 25 | <head> |
|---|
| 26 | <title>BASE - Core specification - Job controller / queue system</title> |
|---|
| 27 | <link rel=stylesheet type="text/css" href="../../styles.css"> |
|---|
| 28 | </head> |
|---|
| 29 | <body> |
|---|
| 30 | |
|---|
| 31 | <div class="navigation"> |
|---|
| 32 | <a href="../../index.html">BASE</a> |
|---|
| 33 | <img src="../../next.gif"> |
|---|
| 34 | <a href="index.html">Core specification</a> |
|---|
| 35 | <img src="../../next.gif"> |
|---|
| 36 | Job controller / queue system |
|---|
| 37 | </div> |
|---|
| 38 | |
|---|
| 39 | <h1>Job controller / queue system</h1> |
|---|
| 40 | |
|---|
| 41 | <div class="abstract"> |
|---|
| 42 | <p> |
|---|
| 43 | This document covers the details of how the job controller / queuing system works. |
|---|
| 44 | </p> |
|---|
| 45 | |
|---|
| 46 | <b>Contents</b><br> |
|---|
| 47 | <ol> |
|---|
| 48 | <li><a href="#queue">The queue</a> |
|---|
| 49 | <li><a href="#manager">The queue manager / job controller</a> |
|---|
| 50 | </ol> |
|---|
| 51 | |
|---|
| 52 | <p class="authors"> |
|---|
| 53 | <b>Last updated:</b> $Date: 2008-09-11 20:01:44 +0000 (Thu, 11 Sep 2008) $ |
|---|
| 54 | </p> |
|---|
| 55 | <p> |
|---|
| 56 | </div> |
|---|
| 57 | |
|---|
| 58 | <a name="queue"> |
|---|
| 59 | <h2>1. The queue</h2> |
|---|
| 60 | </a> |
|---|
| 61 | |
|---|
| 62 | <ol> |
|---|
| 63 | <li>The queuing system is used for operations that the core or |
|---|
| 64 | a client application estimates to have a relativeley long execution time, |
|---|
| 65 | for example importing a raw data set. |
|---|
| 66 | |
|---|
| 67 | <li>The queue can also be used to hold operations that need to |
|---|
| 68 | be executed at regular intervals, for example cleaning and |
|---|
| 69 | house-keeping operations. |
|---|
| 70 | |
|---|
| 71 | <li>The queue holds all information that is needed to complete |
|---|
| 72 | the operation, including: |
|---|
| 73 | <ul> |
|---|
| 74 | <li>the command to run |
|---|
| 75 | <li>the input parameters needed |
|---|
| 76 | <li>a priority value |
|---|
| 77 | <li>the user that added the operaion to the queue |
|---|
| 78 | <li>a repeat interval if desired |
|---|
| 79 | </ul> |
|---|
| 80 | |
|---|
| 81 | </ol> |
|---|
| 82 | |
|---|
| 83 | |
|---|
| 84 | <a name="manager"> |
|---|
| 85 | <h2>2. The queue manager / job controller</h2> |
|---|
| 86 | </a> |
|---|
| 87 | |
|---|
| 88 | <ol> |
|---|
| 89 | <li>An external application (queue manager) running in the background |
|---|
| 90 | regularly checks the queue for new entries. |
|---|
| 91 | |
|---|
| 92 | <li>The queue manager doesn't have to run on the same server as the web server |
|---|
| 93 | or database. |
|---|
| 94 | |
|---|
| 95 | <li>It should be possible to have more than one queue manager running |
|---|
| 96 | on different servers. Each one may be configured to only execute |
|---|
| 97 | certain type of operations or all types of operations. For example |
|---|
| 98 | |
|---|
| 99 | <li>The queue manager executes the operations in the order of priority |
|---|
| 100 | and the time they were added to the queue. |
|---|
| 101 | |
|---|
| 102 | <li>The queue manager must execute an operation as if it were logged in |
|---|
| 103 | as the user who added the operation to the queue. |
|---|
| 104 | |
|---|
| 105 | <li>The queue manager should avoid overloading the server. For |
|---|
| 106 | example by: |
|---|
| 107 | <ul> |
|---|
| 108 | <li>Only allow a maximum number of threads to be executed at the same time. |
|---|
| 109 | <li>Executing the operations with a lower thread priority |
|---|
| 110 | </ul> |
|---|
| 111 | |
|---|
| 112 | <li>While an operation is executing it should, if possible, report |
|---|
| 113 | the progress (ie. percent done) to the core. |
|---|
| 114 | |
|---|
| 115 | <li>The queue manager should be a flexible application that easily can |
|---|
| 116 | be extended by plugins to allow more complex server configurations. |
|---|
| 117 | For example executing operations on a cluster. |
|---|
| 118 | |
|---|
| 119 | <li>If an operation fails because of an unexpected failure it should |
|---|
| 120 | be possible to re-execute the operation at a later time. |
|---|
| 121 | |
|---|
| 122 | |
|---|
| 123 | </ol> |
|---|
| 124 | |
|---|
| 125 | |
|---|
| 126 | </body> |
|---|
| 127 | </html> |
|---|
| 128 | |
|---|
