source: trunk/doc/historical/development/overview/index.html @ 4902

Last change on this file since 4902 was 4902, checked in by Nicklas Nordborg, 14 years ago

Fixes #550: Write "Overview of BASE" section in Developer documentation part

  • Property svn:eol-style set to native
  • Property svn:keywords set to Id Date
File size: 13.1 KB
1<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "">
3  $Id: index.html 4902 2009-04-24 10:56:19Z nicklas $
5  Copyright (C) 2005 Nicklas Nordborg
6  Copyright (C) 2006 Jari Häkkinen, Nicklas Nordborg
8  This file is part of BASE - BioArray Software Environment.
9  Available at
11  BASE is free software; you can redistribute it and/or
12  modify it under the terms of the GNU General Public License
13  as published by the Free Software Foundation; either version 3
14  of the License, or (at your option) any later version.
16  BASE is distributed in the hope that it will be useful,
17  but WITHOUT ANY WARRANTY; without even the implied warranty of
19  GNU General Public License for more details.
21  You should have received a copy of the GNU General Public License
22  along with BASE. If not, see <>.
25  <head>
26    <title>BASE - Development information - Schematic overview</title>
27  <link rel=stylesheet type="text/css" href="../../styles.css">
28  </head>
29<body class="old">
31<div class="navigation">
32  <a href="../../index.html">BASE</a>
33  <img src="../../next.gif">
34  <a href="../index.html">Development information</a>
35  <img src="../../next.gif">
36  Schematic overview
39  <h1>Schematic overview</h1>
40  <div class="warning">
41    NOTE! This document is outdated and has been replaced with newer
42    documentation. See <a href="../../../html/developerdoc/develop_overview/base_develop_overview.html">Developer overview of BASE</a>
43  </div>
44  <div class="abstract">
45    <p>
46      This document gives a brief overview of the Base application.
47    </p>
49    <b>Contents</b><br>
50    <ol>
51    <li><a href="#overview">Overview</a></li>
52    <li><a href="#databases">Fixed database vs. dynamic database</a></li>
53    <li><a href="#hibernate">Hibernate</a></li>
54    <li><a href="#batchapi">Batch API</a></li>
55    <li><a href="#classes">Data classes vs. item classes</a></li>
56    <li><a href="#query">Query API</a></li>
57    <li><a href="#controller">Controller API</a></li>
58    <li><a href="#plugins">Plugin system</a></li>
59    <li><a href="#clients">Client applications</a></li>
60    </ol>
62    <p class="authors">
63    <b>Last updated:</b> $Date: 2009-04-24 10:56:19 +0000 (Fri, 24 Apr 2009) $
64    </p>
65  </div>
67  <a name="overview"></a>
68  <h2>1. Overview</h2>
69  <p>
70    <img src="overview.png">
71  </p>
73  <a name="databases"></a>
74  <h2>2. Fixed database vs. dynamic database</h2>
75  <p>
76    Base 2 stores most of it's data in a database. The database is divided into
77    two parts, one fixed and one dynamic part.
78  </p>
80  <p>
81    The fixed part contains tables that corresponds
82    to the various items found in Base. There is, for example, one table
83    for users, one table for groups and one table for reporters. Some items
84    share the same table. Biosources, samples, extracts and labeled extracts are
85    all biomaterials and share the <code>BioMaterials</code> table. The access
86    to the fixed part of the database goes through Hibernate or in some cases
87    through the Batch API.
88  </p>
89  <p>
90    The dynamic part of the database contains tables for storing analyzed data. 
91    Each experiment has it's own set of tables and it is not possible to mix data
92    from two experiments. The dynamic part of the database can only be accessed
93    by the Batch API and the Query API using SQL and JDBC.
94  </p>
96  <p class="note">
97    NOTE! The actual location of the two parts depends on the database that is used.
98    MySQL uses two separate databases, Postgres uses one database with two schemas.
99  </p>
101  <p>
102    <b>More information</b>
103  </p>
104  <ul>
105    <li><a href="dynamic/index.html">Dynamic API overview</a></li>
106  </ul>
109  <a name="hibernate"></a>
110  <h2>3. Hibernate</h2>
112  <p>
113    Hibernate (<a href=""></a>) is an
114    object/relational mapping software package. It takes plain Java objects
115    and stores them in a database. All we have to do is to set the properties
116    on the objects (for example: <code>user.setName("A name")</code>). Hibernate
117    will take care of the SQL generation and database communication for us.
118    But, this is not a magic or automatic process. We have to provide mapping
119    information about what objects goes into which tables and what properties
120    goes into which columns, and other stuff like caching and proxying settings, etc.
121    This is done by annotating the code with Javadoc comments. The classes
122    that are mapped to the database are found in the <code></code> 
123    package, which is shown as the <code>Data classes</code> box in the image above.
124  </p>
126  <p>
127    Hibernate supports many different database systems. In theory, this means
128    that Base 2 should work with all those databases. In practice we have, however,
129    found that this is not the case. For example, Oracle, converts empty strings
130    to <code>null</code> values, which breaks some parts of our code that
131    expects non-null values. And, our Batch and Query API:s generate some SQL as well.
132    They try to use database dialect information from Hibernate, but it is not
133    always possible. The <code>net.sf.basedb.core.dbengine</code> contains code
134    for generating the SQL that Hibernate can't help us with. There is a generic ANSI
135    driver and special drivers for MySQL and Postgres. We don't expect Base 2 to work
136    with other databases without modifications.
137  </p>
139  <p>
140    <b>More information</b>
141  </p>
142  <ul>
143    <li><a href="../coding/data/index.html">Coding rules and guidelines for the data layer</a></li>
144    <li><a href=""></a></li>
145  </ul>
148  <a name="batchapi"></a>
149  <h2>4. Batch API</h2>
150  <p>
151    Hibernate comes with a price. It affects performance and uses a lot
152    of memory. This means that those parts of Base 2 that often handles
153    lots of items at the same time doesn't work well with Hibernate. This
154    is for example reporters, array design features and raw data. We
155    have created the Batch API to solve these problems.
156  </p>
158  <p>
159    The Batch API uses JDBC and SQL directly against the database. However, we
160    still use metadata and database dialect information available from Hibernate
161    to generate the SQL we need. This should make the Batch API just as
162    database-independent as Hibernate is. The Batch API can be used for any
163    <code>BatchableData</code> class in the fixed part of the database and is the
164    only way for adding data to the dynamic part.
165  </p>
167  <p class="note">
168    NOTE!
169    The main reason for the Batch API is to avoid the internal caching
170    of Hibernate which eats lots of memory when handling thousands of items.
171    Hibernate 3.1 introduced a new stateless API which among other things doesn't
172    do any caching. This version was released after we had created the Batch API.
173    We made a few tests to check if it would be better for us to switch back to Hibernate
174    but found that it didn't perform as well as our own Batch API (it was about 2 times slower).
175    Future versions of Hibernate may perform better so the Batch API may have to be revised
176    again.
177  </p>
178  <p>
179    <b>More information</b>
180  </p>
181  <ul>
182    <li><a href="../coding/batch/index.html">Coding rules and guidelines for batch classes</a></li>
183    <li><a href="core/batchprocessing.html">Overview of the core API: Batch processing</a></li>
184  </ul>
186  <a name="classes"></a>
187  <h2>5. Data classes vs. item classes</h2>
189  <p>
190    The data classes are, with few exceptions, for internal use. These are the classes
191    that are mapped to the database with Hibernate mapping files. They are very simple
192    and contains no logic at all. They don't do any permission checks or any data
193    validation.
194  </p>
196  <p>
197    Most of the data classes has a corresponding item class. For example: <code>UserData</code>
198    and <code>User</code>, <code>GroupData</code> and <code>Group</code>, but there is no
199    corresponding item class for the <code ReporterData</code> class. The item classes are what
200    the client applications can see and use. They contain logic for permission checking
201    (for example if the logged in user has WRITE permission) and data validation (for example
202    setting a required property to null).
203  </p>
205  <p>
206    The only exception to this setup is that batchable data classes doesn't
207    have a corresponding item class. The reason is that the data/item class
208    relation needs the caching system, but in the Batch API we want to cache as little
209    as possible. Hence, it doesn't make sense to have an item class. This creates
210    another problem since we still need to do permission checking and data validation.
211    This was solved by moving that part of the code to the batcher classes
212    (ie. <code>ReporterBatcher</code>).
213  </p>
215  <p>
216    <b>More information</b>
217  </p>
218  <ul>
219    <li><a href="../coding/data/index.html">Coding rules and guidelines for the data layer</a></li>
220    <li><a href="../coding/item/index.html">Coding rules and guidelines for item classes</a></li>
221    <li><a href="../coding/batch/index.html">Coding rules and guidelines for batch classes</a></li>
222    <li><a href="core/accesspermissions.html">Overview of the core API: Access permission to items</a></li>
223    <li><a href="core/datavalidation.html">Overview of the core API: Data validation</a></li>
224    <li><a href="core/batchprocessing.html">Overview of the core API: Batch processing</a></li>
225  </ul>
228  <a name="query"></a>
229  <h2>6. Query API</h2>
230  <p>
231    The Query API is used to build and execute queries against the data in the
232    database. It builds a query by using objects that represents certain
233    operations. For example, there is an <code>EqRestriction</code> object
234    which tests if two expressions are equal and there is an <code>AddExpression</code>
235    object which adds two expressions. In this way it is possible to build
236    very complex queries without using SQL or HQL.
237  </p>
239  <p>
240    The Query API knows can work both via Hibernate and via SQL. In the first case it
241    generates HQL (Hibernate Query Language) statements which Hibernate then
242    translates into SQL. In the second case SQL is generated directly.
243    In most cases HQL and SQL are identical, but not
244    always. Some situations are solved by having the Query API generate
245    slightly different query strings. Some query elements can only be used
246    with one of the query types.
247  </p>
249  <p class="note">
250    NOTE! The object-based approach makes it a bit difficult to store
251    a query for later reuse. The <code>net.sf.basedb.util.jep</code> 
252    package contains an expression parser that can be used to convert
253    a string to <code>Restriction</code>:s and <code>Expression</code>:s for
254    the Query API. While it doesn't cover 100% of the cases it should be
255    useful for the <code>WHERE</code> part of a query.
256  </p>
258  <p>
259    <b>More information</b>
260  </p>
261  <ul>
262    <li><a href="query/index.html">Overview of queries and the Query API</a></li>
263  </ul>
265  <a name="controller"></a>
266  <h2>7. Controller API</h2>
267  <p>
268    The Controller API is the very heart of the Base 2 system. This part
269    of the core is used for boring but essential details, such as
270    user authentication, database connection management, transaction
271    management, data validation, and more. We don't write more about this
272    part here, but recommends reading the documents below.
273  </p>
275  <p>
276    <b>More information</b>
277  </p>
278  <ul>
279    <li><a href="core/index.html">Overview of the the core API</a></li>
280  </ul>
282  <a name="plugins"></a>
283  <h2>8. Plugin system</h2>
284  <p>
285    From the core code's point of view a plugin is just another client
286    application. A plugin doesn't have more powers and doesn't have
287    access to some special API that allows it to do cool stuff that other
288    clients can't.
289  </p>
291  <p>
292    However, the core must be able to control when and where a plugin is
293    executed. Some plugins may take a long time doing their calculations
294    and may use a lot of memory. It would be bad if a 100+ users started
295    to execute a resource-demanding plugin at the same time. This problem is
296    solved by adding a job queue. Each plugin that should be executed is
297    registered as <code>Job</code> in the database. A job controller is
298    checking the job queue at regular intervals. The job controller can then
299    choose if it should execute the plugin or wait depending on the current
300    load on the server.
301  </p>
303  <p class="note">
304    NOTE! This part of the core is not yet very developed. The only existing
305    job controller is the internal job controller which just starts another thread
306    in the same process. This means that a badly written plugin may crash the
307    entire web server. For example, do not call <code>System.exit()</code> 
308    in the plugin code, since it shuts down Tomcat as well. This part of the core
309    is something that must be changed in the near future.
310  </p>
312  <p>
313    <b>More information</b>
314  </p>
315  <ul>
316    <li><a href="core/plugins.html">Overview of the core API: Plugin execution</a></li>
317    <li><a href="core/jobs.html">Overview of the core API: Job queue management</a></li>
318    <li><a href="../index.html#plugins">Plug-ins</a></li>
319  </ul>
322  <a name="clients"></a>
323  <h2>9. Client applications</h2>
324  <p>
325    Client applications are application that use the Base 2 core API. This document hasn't
326    much to say about them. The current web application is built with Java Server Pages
327    (JSP). It is supported by several application server but we have only tested
328    it with Tomcat.
329  </p>
331  <p>
332    Another client application is the migration tool that migrates Base 1.2.x data
333    to Base 2.
334  </p>
Note: See TracBrowser for help on using the repository browser.