source: trunk/doc/historical/development/overview/index.html @ 4889

Last change on this file since 4889 was 4889, checked in by Nicklas Nordborg, 14 years ago

References #1290: Change source files to UTF-8

Changed 'Hakkinen' to 'Häkkinen'.
  • Property svn:eol-style set to native
  • Property svn:keywords set to Id Date
File size: 12.9 KB
Line 
1<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
2<!--
3  $Id: index.html 4889 2009-04-06 12:52:39Z nicklas $
4
5  Copyright (C) 2005 Nicklas Nordborg
6  Copyright (C) 2006 Jari Häkkinen, Nicklas Nordborg
7
8  This file is part of BASE - BioArray Software Environment.
9  Available at http://base.thep.lu.se/
10
11  BASE is free software; you can redistribute it and/or
12  modify it under the terms of the GNU General Public License
13  as published by the Free Software Foundation; either version 3
14  of the License, or (at your option) any later version.
15
16  BASE is distributed in the hope that it will be useful,
17  but WITHOUT ANY WARRANTY; without even the implied warranty of
18  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
19  GNU General Public License for more details.
20
21  You should have received a copy of the GNU General Public License
22  along with BASE. If not, see <http://www.gnu.org/licenses/>.
23-->
24<html>
25  <head>
26    <title>BASE - Development information - Schematic overview</title>
27  <link rel=stylesheet type="text/css" href="../../styles.css">
28  </head>
29<body>
30
31<div class="navigation">
32  <a href="../../index.html">BASE</a>
33  <img src="../../next.gif">
34  <a href="../index.html">Development information</a>
35  <img src="../../next.gif">
36  Schematic overview
37</div>
38
39  <h1>Schematic overview</h1>
40
41  <div class="abstract">
42    <p>
43      This document gives a brief overview of the Base application.
44    </p>
45   
46    <b>Contents</b><br>
47    <ol>
48    <li><a href="#overview">Overview</a></li>
49    <li><a href="#databases">Fixed database vs. dynamic database</a></li>
50    <li><a href="#hibernate">Hibernate</a></li>
51    <li><a href="#batchapi">Batch API</a></li>
52    <li><a href="#classes">Data classes vs. item classes</a></li>
53    <li><a href="#query">Query API</a></li>
54    <li><a href="#controller">Controller API</a></li>
55    <li><a href="#plugins">Plugin system</a></li>
56    <li><a href="#clients">Client applications</a></li>
57    </ol>
58   
59    <p class="authors">
60    <b>Last updated:</b> $Date: 2009-04-06 12:52:39 +0000 (Mon, 06 Apr 2009) $
61    </p>
62  </div>
63
64  <a name="overview"></a>
65  <h2>1. Overview</h2>
66  <p>
67    <img src="overview.png">
68  </p>
69 
70  <a name="databases"></a>
71  <h2>2. Fixed database vs. dynamic database</h2>
72  <p>
73    Base 2 stores most of it's data in a database. The database is divided into
74    two parts, one fixed and one dynamic part.
75  </p>
76   
77  <p>
78    The fixed part contains tables that corresponds
79    to the various items found in Base. There is, for example, one table
80    for users, one table for groups and one table for reporters. Some items
81    share the same table. Biosources, samples, extracts and labeled extracts are
82    all biomaterials and share the <code>BioMaterials</code> table. The access
83    to the fixed part of the database goes through Hibernate or in some cases
84    through the Batch API.
85  </p>
86  <p>
87    The dynamic part of the database contains tables for storing analyzed data. 
88    Each experiment has it's own set of tables and it is not possible to mix data
89    from two experiments. The dynamic part of the database can only be accessed
90    by the Batch API and the Query API using SQL and JDBC.
91  </p>
92
93  <p class="note">
94    NOTE! The actual location of the two parts depends on the database that is used.
95    MySQL uses two separate databases, Postgres uses one database with two schemas.
96  </p>
97
98  <p>
99    <b>More information</b>
100  </p>
101  <ul>
102    <li><a href="dynamic/index.html">Dynamic API overview</a></li>
103  </ul>
104
105
106  <a name="hibernate"></a>
107  <h2>3. Hibernate</h2>
108 
109  <p>
110    Hibernate (<a href="http://www.hibernate.org">www.hibernate.org</a>) is an
111    object/relational mapping software package. It takes plain Java objects
112    and stores them in a database. All we have to do is to set the properties
113    on the objects (for example: <code>user.setName("A name")</code>). Hibernate
114    will take care of the SQL generation and database communication for us.
115    But, this is not a magic or automatic process. We have to provide mapping
116    information about what objects goes into which tables and what properties
117    goes into which columns, and other stuff like caching and proxying settings, etc.
118    This is done by annotating the code with Javadoc comments. The classes
119    that are mapped to the database are found in the <code>net.sf.basedb.core.data</code> 
120    package, which is shown as the <code>Data classes</code> box in the image above.
121  </p>
122
123  <p>
124    Hibernate supports many different database systems. In theory, this means
125    that Base 2 should work with all those databases. In practice we have, however,
126    found that this is not the case. For example, Oracle, converts empty strings
127    to <code>null</code> values, which breaks some parts of our code that
128    expects non-null values. And, our Batch and Query API:s generate some SQL as well.
129    They try to use database dialect information from Hibernate, but it is not
130    always possible. The <code>net.sf.basedb.core.dbengine</code> contains code
131    for generating the SQL that Hibernate can't help us with. There is a generic ANSI
132    driver and special drivers for MySQL and Postgres. We don't expect Base 2 to work
133    with other databases without modifications.
134  </p>
135
136  <p>
137    <b>More information</b>
138  </p>
139  <ul>
140    <li><a href="../coding/data/index.html">Coding rules and guidelines for the data layer</a></li>
141    <li><a href="http://www.hibernate.org">www.hibernate.org</a></li>
142  </ul>
143
144
145  <a name="batchapi"></a>
146  <h2>4. Batch API</h2>
147  <p>
148    Hibernate comes with a price. It affects performance and uses a lot
149    of memory. This means that those parts of Base 2 that often handles
150    lots of items at the same time doesn't work well with Hibernate. This
151    is for example reporters, array design features and raw data. We
152    have created the Batch API to solve these problems.
153  </p>
154
155  <p>
156    The Batch API uses JDBC and SQL directly against the database. However, we
157    still use metadata and database dialect information available from Hibernate
158    to generate the SQL we need. This should make the Batch API just as
159    database-independent as Hibernate is. The Batch API can be used for any
160    <code>BatchableData</code> class in the fixed part of the database and is the
161    only way for adding data to the dynamic part.
162  </p>
163 
164  <p class="note">
165    NOTE!
166    The main reason for the Batch API is to avoid the internal caching
167    of Hibernate which eats lots of memory when handling thousands of items.
168    Hibernate 3.1 introduced a new stateless API which among other things doesn't
169    do any caching. This version was released after we had created the Batch API.
170    We made a few tests to check if it would be better for us to switch back to Hibernate
171    but found that it didn't perform as well as our own Batch API (it was about 2 times slower).
172    Future versions of Hibernate may perform better so the Batch API may have to be revised
173    again.
174  </p>
175  <p>
176    <b>More information</b>
177  </p>
178  <ul>
179    <li><a href="../coding/batch/index.html">Coding rules and guidelines for batch classes</a></li>
180    <li><a href="core/batchprocessing.html">Overview of the core API: Batch processing</a></li>
181  </ul>
182
183  <a name="classes"></a>
184  <h2>5. Data classes vs. item classes</h2>
185 
186  <p>
187    The data classes are, with few exceptions, for internal use. These are the classes
188    that are mapped to the database with Hibernate mapping files. They are very simple
189    and contains no logic at all. They don't do any permission checks or any data
190    validation.
191  </p>
192 
193  <p>
194    Most of the data classes has a corresponding item class. For example: <code>UserData</code>
195    and <code>User</code>, <code>GroupData</code> and <code>Group</code>, but there is no
196    corresponding item class for the <code ReporterData</code> class. The item classes are what
197    the client applications can see and use. They contain logic for permission checking
198    (for example if the logged in user has WRITE permission) and data validation (for example
199    setting a required property to null).
200  </p>
201 
202  <p>
203    The only exception to this setup is that batchable data classes doesn't
204    have a corresponding item class. The reason is that the data/item class
205    relation needs the caching system, but in the Batch API we want to cache as little
206    as possible. Hence, it doesn't make sense to have an item class. This creates
207    another problem since we still need to do permission checking and data validation.
208    This was solved by moving that part of the code to the batcher classes
209    (ie. <code>ReporterBatcher</code>).
210  </p>
211 
212  <p>
213    <b>More information</b>
214  </p>
215  <ul>
216    <li><a href="../coding/data/index.html">Coding rules and guidelines for the data layer</a></li>
217    <li><a href="../coding/item/index.html">Coding rules and guidelines for item classes</a></li>
218    <li><a href="../coding/batch/index.html">Coding rules and guidelines for batch classes</a></li>
219    <li><a href="core/accesspermissions.html">Overview of the core API: Access permission to items</a></li>
220    <li><a href="core/datavalidation.html">Overview of the core API: Data validation</a></li>
221    <li><a href="core/batchprocessing.html">Overview of the core API: Batch processing</a></li>
222  </ul>
223
224
225  <a name="query"></a>
226  <h2>6. Query API</h2>
227  <p>
228    The Query API is used to build and execute queries against the data in the
229    database. It builds a query by using objects that represents certain
230    operations. For example, there is an <code>EqRestriction</code> object
231    which tests if two expressions are equal and there is an <code>AddExpression</code>
232    object which adds two expressions. In this way it is possible to build
233    very complex queries without using SQL or HQL.
234  </p>
235 
236  <p>
237    The Query API knows can work both via Hibernate and via SQL. In the first case it
238    generates HQL (Hibernate Query Language) statements which Hibernate then
239    translates into SQL. In the second case SQL is generated directly.
240    In most cases HQL and SQL are identical, but not
241    always. Some situations are solved by having the Query API generate
242    slightly different query strings. Some query elements can only be used
243    with one of the query types.
244  </p>
245 
246  <p class="note">
247    NOTE! The object-based approach makes it a bit difficult to store
248    a query for later reuse. The <code>net.sf.basedb.util.jep</code> 
249    package contains an expression parser that can be used to convert
250    a string to <code>Restriction</code>:s and <code>Expression</code>:s for
251    the Query API. While it doesn't cover 100% of the cases it should be
252    useful for the <code>WHERE</code> part of a query.
253  </p>
254 
255  <p>
256    <b>More information</b>
257  </p>
258  <ul>
259    <li><a href="query/index.html">Overview of queries and the Query API</a></li>
260  </ul>
261 
262  <a name="controller"></a>
263  <h2>7. Controller API</h2>
264  <p>
265    The Controller API is the very heart of the Base 2 system. This part
266    of the core is used for boring but essential details, such as
267    user authentication, database connection management, transaction
268    management, data validation, and more. We don't write more about this
269    part here, but recommends reading the documents below.
270  </p>
271 
272  <p>
273    <b>More information</b>
274  </p>
275  <ul>
276    <li><a href="core/index.html">Overview of the the core API</a></li>
277  </ul>
278   
279  <a name="plugins"></a>
280  <h2>8. Plugin system</h2>
281  <p>
282    From the core code's point of view a plugin is just another client
283    application. A plugin doesn't have more powers and doesn't have
284    access to some special API that allows it to do cool stuff that other
285    clients can't.
286  </p>
287
288  <p>
289    However, the core must be able to control when and where a plugin is
290    executed. Some plugins may take a long time doing their calculations
291    and may use a lot of memory. It would be bad if a 100+ users started
292    to execute a resource-demanding plugin at the same time. This problem is
293    solved by adding a job queue. Each plugin that should be executed is
294    registered as <code>Job</code> in the database. A job controller is
295    checking the job queue at regular intervals. The job controller can then
296    choose if it should execute the plugin or wait depending on the current
297    load on the server.
298  </p>
299 
300  <p class="note">
301    NOTE! This part of the core is not yet very developed. The only existing
302    job controller is the internal job controller which just starts another thread
303    in the same process. This means that a badly written plugin may crash the
304    entire web server. For example, do not call <code>System.exit()</code> 
305    in the plugin code, since it shuts down Tomcat as well. This part of the core
306    is something that must be changed in the near future.
307  </p>
308
309  <p>
310    <b>More information</b>
311  </p>
312  <ul>
313    <li><a href="core/plugins.html">Overview of the core API: Plugin execution</a></li>
314    <li><a href="core/jobs.html">Overview of the core API: Job queue management</a></li>
315    <li><a href="../index.html#plugins">Plug-ins</a></li>
316  </ul>
317
318 
319  <a name="clients"></a>
320  <h2>9. Client applications</h2>
321  <p>
322    Client applications are application that use the Base 2 core API. This document hasn't
323    much to say about them. The current web application is built with Java Server Pages
324    (JSP). It is supported by several application server but we have only tested
325    it with Tomcat.
326  </p>
327 
328  <p>
329    Another client application is the migration tool that migrates Base 1.2.x data
330    to Base 2.
331  </p>
332
333</body>
334</html>
Note: See TracBrowser for help on using the repository browser.