Persistence mechanism trials and Compass Framework revolution

We've used number of different technologies to provide persistence service in our products. Our main concerns/requirements can be listed as:

1. performance. The persistence mechanism should perform well since we manage real time monitoring data. Number of write/read operations per second as well as performance of the queries are key performance aspects for us.
2. A fixed/static schema is not suitable for our products since they support dynamic modeling. Mapping from Object to its storage should be flexible.
3. Querying the object storage should be easy and should not require developer skills.

In the first iteration of our products, we had used a database. However, relational databases have a rigid schema structure, hence modification of the schema by the users was problematic. The performance of the solution was also not ideal. Both read/write and search performance were not adequate for many use cases that we have.

In the next iteration, we've kept the data in-memory and implemented our own persistence layer using Prevayler which is a very good and simple framework. Prevayler allows using in memory data structures like maps as main storage and persists every write operation by logging this operation to a file. When it is restarted it reads the data from these files and applies transactions in same order, and constructs in memory data. It increased performance dramatically in our applications, since all of the content is stored in memory. Also, since it allows to use any kind of memory structure to hold data, there was no need to map object to a different storage structure.

Prevayler also has some obvious drawbacks. Since everything is kept in memory, you need a lot of memory. This was not a major problem for our applications since real-time data is typically not that high and we've used a relatinal database to store the historical data.
Another drawback was the querying interface. We had to provide a query mechanism to search our internal storage and we implemented it ourselves using javacc. We also implemented our own expression language and provided a querying mechanism for our internal storage.

We used the prevayler based in memory object storage in several products which can be listed as RapidInsight , RapidInformer , RapidConnector .

After deciding to join open source community and starting to implement RapidCMDB we've decided to build our solutions on top of established frameworks, instead of continuing to build our own as we had done previously. After a survey of the field, we've decided on using Grails as our development framework since there were number of common components and the philosophy of the framework closest to our own.

Grails provides a persistence layer and a object relational mapping system which is named GORM . and provides persistence by using Hibernate . It looked like a good match for our requirements at first, but we've run into difficulties as we went further into developing our solution. It introduced some overhead that did not add much value for us and were inappropriate. We had some custom requirements and we tried to modify some of the GORM  methods. Because of our somewhat different requirements and necessary modifications, we had a lot of problems with GORM  basics. We've released first versions of RapidCMDB  based on GORM  persistence layer. We tried to get rid of problems and bugs we introduced while changing GORM  structure and as well as others which GORM  have, but this slowed the paste of our development significantly.

Finally, we discovered Grails Searchable Plugin  and Compass Framework . Grails Searchable Plugin  is a plugin which embeds Compass Search Framework in grails. Searchable Plugin is not designed as a persistence mechanism. Its main purpose is providing Google like search engine capabilities for already persisted data in database via GORM.
However, after a length research period and numerous design meetings and tests, we've decided to explore using it as our persistence mechanism. The main benefits we saw for us can be listed as:

1. Flexibility. Searchable Plugin (Compass) does not require a static schema. A mapping is required but this mapping is already done in Grails Searchable Plugin .
2. Simple yet powerful query mechanism. Compass querying api is very powerful. It is quite close to Google search engine query syntax.
3. Indexing mechanism is very good and provides high performance.
4. It is not memory based. Therefore we can store lots of data on its indexes.
5. Its read/write performance is reasonable. We find the performance quite close to GORM performance even with the defaults (using one transaction for every write operation) and can be significantly improved by using the right configuration.

As a result, the latest version of RapidCMDB (0.3.4) is a Grails application that does not use a database (or GORM), and uses searchable plugin and Compass to store and access data. We think we're barely scratching the surface so far, and continue to be amazed by the new Compass/Lucene wonders we discover every day.