pyDocDB

Background The inspiration to this project is the CouchDB document database. CouchDB is a document database storing key-value based documents and using map-functions to build up indexes. It supports network access via an json-based REST-API. CouchDB is based on Erlang.

Even though CouchDB is based on Erlang the map functions is natively written in Javascript. It is also possible to use alternative languages for the map-function using a very simple Map Server API (different Map Servers have been released, for instance a Python-based one).

The Erlang OPT platforms enables CouchDB to have great concurrency features and it can be distributed over several hosts for load-balancing and replication.

Suggestion This implementation would be a python implementation with less focus on concurrency and more focus on easy of use, at least from a pythonic point of view.

The main storage would store documents in a BTree. The id of the document would be the key and the value a dictionary with key-value pairs representing the data. Dictionaries may contain primitive data types that should be representable in JSON (list, tuples, dictionaries, strings, numbers and dates in string format). E.g. no direct Python objects should be stored in the documents.

It could be based on ZODB or SQLite as the underlying database infrastructure.

Views with Map and Reduce Both a map and reduce process for creating interesting views to the database. The Map-function takes a document and returns one or more key-value pairs, which will optionally be saved as an index in its own BTree. (Note: I’m not sure weather identical keys would be required, I just know to little about Map-Reduce to be sure how to deal with that yet.)

A reduce function takes key-value from a map-function and generate another list of key-value pairs, also optionally saved in another BTree. Possible several map/reduce methods can be chained.

Maps could be used for stuff like word count operations, reduce to sum it up, following the application structure of for instance Googles Map-Reduce system. (I’m finding this lacking in CouchDB, were they have chosen to integrate with Lucene for full-text indexing, which could be done native IMHO.)

Relation Maps could also be used to represent relations between documents can be calculated via Map functions.

Database development Map- and Reduce-functions can be stored in special design documents. Template for documents may also be stored as special design documents.

External API API will be JSON, which means that only primary types will be allowed. Possible extension could be made for a Python only version using a PySON API, allowing native Python object, but that is not the goal for the first iterations.