eXist-db: Current State, Features and Roadmap

ModuleStatusPriorityTest CoverageProgressWho
1. Document Storage
1.1. Collection storageStable, but subject to redesignHighNo tests0%wolf

The current organization of collections and resources causes a number of problems with respect to (a) locking, (b) query performance, (c) update performance. Right now, documents are tightly bound to the collection in which they are contained. Any operation on a document has to go through its parent collection. As a result, locking and access control becomes quite complex as we need to take care of the document and the collection.

There's also a direct dependency between the size of a collection (in terms of the number of documents stored in it) and document update speed. If a collection has a large number of documents, removing a single document becomes very slow.

This problem can be solved by physically decoupling documents and collections:

  • a collection should be modelled as a logical unit, not a physical one.
  • current integer document ids should be replaced by hierarchical ids, which model the entire collection/document path.
  • the link between the resource and the collection is only done through the hierarchical document id.
  • locking and access control should be delegated to a central lock manager, which just contains a map DocumentProxy -> Lock
1.2. Allow metadata to be associated with a documentOpenAvgN/A0%

Metadata could include system properties like last-modification date or user-defined metadata. Preferably, metadata records should be ordinary XML documents. The format should not be restricted.

2. Indexing

Since version 1.2, there are now alternative index configuration methods which support the optimizer in rewriting a query for best performance. A new modularized indexing architecture allows to plug in arbitrary new indexes into the indexing pipeline. A N-gram and a spatial index module were added as prototypes to test the new architecture.

There will be other index types added in the future, for example:

  • combined path indexes to speed up frequently-used XPath expressions
  • indexes to support operations on atomic-values, e.g. in order-by expressions or the distinct-values function
  • integration of specialized, external index types like the spatial index module mentioned above.
2.1. XQuery and XPath Full Text 1.0OpenAvgN/A15%

Implement the W3C's full text extensions to XQuery. Many features could be easily implemented on top of Lucene.

The grammar for the full text extensions has already been merged into the XQuery parser as part of a Google Summer of Code project and is available in an SVN branch. It just needs to be filled with life.

2.2. Index-support for order-by, distinct-valuesOpenAvgN/A0%

Order-by expressions and other functions that need to access atomized nodes are not supported by indexes.

2.3. Collation-driven indexingOpenAvgN/A0%

Maybe part of FT index redesign.

3. Transactions and Recovery

The journal log and the recovery manager should be stable and are covered by extensive tests. However, recovery failures can not be excluded entirely. The tests can't reproduce every possible real-world scenario.

However, some steps remain for eXist to become a fully transactional database system. Transaction support is currently limited to the functionality needed for crash recovery. Though we maintain transactions internally, they are currently not exposed to applications. Also, read operations are not transactional right now.

In order to allow user-defined ACID transactions with support for rollback, all index files would need to be protected by the journaling log. The required functionality is basically available, but the feature is currently not regarded as high-priority.

3.1. Internal transaction managementStablexTested100%

Transactions are maintained internally, but they are not exposed to applications. eXist-db does not yet support full ACID transactions. Read-only operations bypass the transaction system.

3.2. User-definable transactionsOpenLowN/A0%

Journal logs are limited to critical data required for recovery. No transaction rollbacks.

4. Backup / Restore
4.1. Store configuration into backupOpenxNo tests0%

The backup utility should also include a copy of the relevant server configuration files (mainly conf.xml) into the created backup. Settings like page size, additional XQuery or index modules etc. are important.

5. Configuration
5.1. Dynamic configuration of the database via Java Management Extensions (JMX)OpenxNo tests0%

Main problem: access control and security.

6. Node-level updates
6.1. XUpdateStablexTested100%

No remarks available.

6.2. XQuery Update ExtensionsStable, but subject to redesignxTested75%

eXist's XQuery Update Extensions were implemented before the first W3C drafts were published. The differences between the W3C recommendation and our implementation are not that big, though there are some subtle differences concerning the processing model. In fact, the recommendation simplifies a few issues.

http://www.w3.org/TR/xquery-update-10/

7. Access-Control

The currently implemented Unix-like access control scheme is sufficient to protect resources and collections in a multi-user environment. However, it might be too coarse-grained for some types of applications. A more dynamic ACL implementation could help here. Right now, security management forms part of the database core. This is unnecessary. A more modular architecture would allow different security managers to be plugged in. It would be the responsibility of the security manager implementation to handle ACL lists.

Since version 1.1, eXist supports the XACML standard for fine-grained access control to stored XQueries, Java classes etc.

8. Schema Validation
8.1. XQuery validation featuresOpenAvgN/A0%

No remarks available.

8.2. Store PSVI with the node tree in the databaseOpenLowN/A0%dizzzz

No remarks available.

8.3. Static typing based on PSVIOpenLowN/A0%dizzzz

No remarks available.

9. XQuery

The XQuery engine as well as the standard function libraries should be updated to align with the XQuery 3.0 recommendation. Basically, almost all XQuery 1.0 core language features are implemented, excluding schema related features, which are currently beyond eXist-db's scope.

XQuery support in eXist-db is covered by the official W3C XQuery Test Suite (XQTS) 1.02. Implementing the official XQTS XQuery test suite was a top priority in order to guarantee standard conformance and avoid future regressions.

9.1. Core XPath and XQueryStablextested100%

Updated to the XPath 2.0 and XQuery 1.0 recommendations. Stable, excluding schema-related features.

9.2. XPath and XQuery atomic value typesStableAvgtested99.4%
9.3. XPath and XQuery function librariesStableHightested99.4%

Updated to XPath 2.0 and XQuery 1.0 recommendations. Stable, excluding schema-related features.

9.4. XPath and XQuery function librariesStableHightested99.4%

Updated to XPath 2.0 and XQuery 1.0 recommendations.

9.5. XQuery serializationStable, but subject to redesignAvgtested80%

Though we implement most of the serialization options specified in the XQuery and XSLT serialization spec, some options need to be reworked and should be covered by tests.

9.6. Error reportingStable, but subject to redesignAvgN/A75%

Error reports by the XQuery parser and compiler need to be improved.

10. XIncludeStableLowNo tests80%

XInclude expansion happens at serialization time. Queries across the included document fragments are not possible. Stable, but limited.

11. Interfaces
11.1. XML:DB APIStablexTested100%

No remarks available.

11.2. XML-RPCStablexPartially tested100%

Exposes the entire database functionality.

11.3. RESTStable, but subject to redesignLowPartially tested90%delirium

Does not cover administrative functions, e.g. user-management and permissions. Stable, but further functionality could be exposed.

The XQuery API for Java (XQJ) implementation makes heavy use of REST and improves the interface in some aspects. However, the XQJ branch has not yet been merged into trunk.

11.4. SOAPStableLowNo tests90%
12. Documentation
12.1. 1.5OpenLowN/A0%

Based on Java 6.

13. Other Tasks
13.1. i18nOpenLowN/A20%

Provide translations for error messages, console outputs etc. At least, resource bundles should be used, so others can translate them if they want.

13.2. Clean up/upgrade librariesBetaLowN/A60%

All libraries included with eXist need to be checked.

13.3. Move to ANTLR3 parserInstableHighN/A60%ljo

Change the parser to ANTLR3 which is better performant, LL* lookahead capable and processes whitespace in a better manner than ANTLR2 which we currently are using.

Maybe use gunit for testing?

Legend

PercentageDescription
0work not started
201-20 Percentage of completion
4021-40 Percentage of completion
6041-60 Percentage of completion
8061-80 Percentage of completion
9981-99 Percentage of completion
Done100 Percentage of completion
PriorityDescription
1. HighestVery important
2. HighImportant
3. AvgNice to have
4. LowNot very important
5. xNot yet decided