Polyglot Persistent – Tools for the right job

After years of working with IT, as developers we tend to solve problems in similar ways, off course we are humans and we tend to think that the same hammer applies to all kind of problems, but the Internet of Things, where applications are not as straightforward as they used to be, are forcing us to use our brains.

Recently I have been tasked to create an ecommerce application allowing customers to pick from a range of configurations and enable them to provision their services without any other help than the web app, these services include Virtual machines, email, Active directory configuration, etc. Part of the requirement is to enable the platform to grow and be flexible to include new configurations and options based on the customers requirements.

This last requirement requires a system that adapts to changes, like adding new fields, or data that helps the system to achieve the wide range of parameters required for each service and mix of configurations. Also been an ecommerce app, that will charge money to customers, requires consistency and reliability.

How to achieve flexibility in terms of fields without having a deployment problem for every new change? Metadata programming to the rescue, instead of hard coding screens and options we created a set of tables, namespaces as they have been stored in djondb, defining the screen structures, fields, validations, etc. And code an engine that draws everything based on this metadata. This metadata programming is something really hard to archieve using RDBMS, as any new field requires alter tables and update a lot of data, meaning stopping services for new patches, but doing this with djondb is simple and doesn’t require breaking your head, and production at the same time.

On the other hand there’s data structure that is pretty much static and it does not require hard changes over time, things like customer information, orders (who, when, price, etc), etc. Most of these are perfect for a Relational Database, also it will be required to be reported, etc.

Metadata problem

An oversimplified version of the metadata looks like this:

Form: Server Order
Fields: [ RAM, CPU, OS, Data Center, Admin Password, IP Address ]

Off course, each field will have type, required flag, Min length, etc. So our simple structure will really look like this:

  form: 1,
  description: "Server Order",
  fields: {
      description: { type: "string", caption: "Description", len: 100, required: true },
      RAM: { type: "dropdown", caption: "RAM", list: "ram", required: true },
      CPU: { type: "dropdown", caption: "CPU", list: "cputypes", required: true},
      AdminPassword: { type: "password", caption: "Password", required: true }

Now, it’s really simple to create a routine in jquery rendering the above dynamically, and sending the data back to the server in the form of a JSON structure, both structures may grow without changing the code. The code will require changes if a new field type is supported, like “radio”, or others, but both server and client are flexible to render and store the information. Here I’m just showing one of the possible order type, now extend the concept to Email, where you will need to store the user name, password, size, etc. and a full range of other services that users will be able to provision using the platform. On the other hand adding new fields to djondb is not a big problem, just send the new JSON structure and djondb will take care of it.

Processing the order requires a similar approach, you will script some plans based on the above, so using a templating system at the server side that replaces values into script and executes the script is a no brainer. Here using a transactional system over this data makes sense as you need to update the order as soon as the provision is ready, but also store the provision information for later maintenance. No Problem with djondb you have both, you can store atomic multi document transactions and use JSON at the same time.

Relational Data goes to a RDBMS

NoSQL is not really a tool for replacing your full system, it is one of the many tools that you will use to build your application. So we kept relational in a relational database, their infrastructure is based on Microsoft, therefore SQL Server is their preferred choice, information like invoices, customer data, details, etc. where the data structure are almost fixed will live into a RDBMS, linking keys across environments is not really a problem. When dealing with dynamic information that will change over time and requires constant changes will go to djondb, whilst fixed structures are processed in SQL Server.

Another example on where to use the right tool for the right problem is text searching, nor SQL Server or NoSQL are really good for searching in a google style search, for this elastic search helps you to index your texts and provide results that will be linked to djondb and others.

Key decisions to support your architecture

– Fixed data, with no changing structure whatsoever, RDBMS (Pick your choice, SQL Server, Oracle, MySQL, etc)
– Unstructured data requiring transactions (Not too much choice in here, djondb)
– Text search capabilities (Elastic Search)

Now mix all of this in your application and you will use the right tool for the right job, implement your system without any hassle and make your customers happy, using multiple persistence storages into the same application is known as Polyglot persistence.

Hope you like this and let me know your comments, and your approaches to similar problems.

Posted in Uncategorized

Creating a product catalog with djondb, PHP Restler and Backbone JS Part V – Adding new features


In our previous post, we created the user interface components of the application. In this post, we’ll make some changes at database level and we’ll see how our application is impacted due to these changes.

In a relational database world

Now, let’s say that we want to include a new product type in the catalog. Now we want to sell phones too. What should we do in our application to support that?

In a traditional relational database, may be we should create a new table Phone to store its properties such as Manufacturer and Capacity.

But, we don’t want to repeat common properties as Name, Price and Picture, so we better create a Product table to store those properties and make a relation to Book and Phone Tables:

Read more ›

Posted in djondb application, NoSQL Samples Tagged with:

Creating a product catalog with djondb, PHP Restler and Backbone JS Part IV – UI


In our previous post, we created the services layer of our application using PHP Restler. In this post, we’ll create the user interface components using backbone to consume the services we’ve created.

Creating Backbone models

First let’s create a file called models.js and define a model for a Product and for ProductCollection. Also, we’ll register here the REST URLs for each model.

This is the code for Backbone Product model:

window.Product = Backbone.Model.extend({

    urlRoot: "rest/products",
    idAttribute: "_id",

    defaults: {
        _id : null,
        prodtype: "",
        name: "",
        picture: null,
        author: "",
        price: "",
        releasedate: "",
        isbn: ""

And this is the code for Backbone ProductCollection model:

window.ProductCollection = Backbone.Collection.extend({
    model: Product,
    url: "rest/products"

Read more ›

Posted in djondb application, NoSQL Samples Tagged with:

Creating a product catalog with djondb, PHP Restler and Backbone JS Part III – Services layer


In our previous post, we installed djondb database and used the console to add and retrieve products. In this post, we’ll create the services layer of our application using PHP Restler.

Installing PHP Driver for djondb

Download the djondb PHP driver from djondb downloads page: http://www.djondb.com/downloads.html. In this example we will use the OSX version of djondb’s PHP 5.4 driver. Just open the OSX zip file djondb_phpext_Darwin_i386.zip to extract its content (no matter where).

We’ll see the djonwrapper.php file and the modules folder with the djonwrapper.so file as shown in the following image:

Copy the djonwrapper.so file that is in the modules folder to this folder: /usr/lib/php/extensions/no-debug-non-zts-20100525/ (the folder may change based on your installation) as shown in the image below. if you are not sure where the folder is, you can execute the command php-config –extension-dir and it will show you the folder configured for extensions.

Read more ›

Posted in djondb application, NoSQL Samples Tagged with:

Creating a product catalog with djondb, PHP Restler and Backbone JS Part II – Setting up


In our previous post, we have explained some aspects of the application we are building with djondb. In this post, we’ll install djondb, the enterprise class NoSQL database and we’ll play around a little bit with the console.

Installing djondb

First of all, download and install djondb. You can get the djondb installer from djondb downloads page: http://www.djondb.com/downloads.html. In this example we will use the OSX version. Just open the OSX package and follow the instructions of the installer.

The installer does not create the database folder, so you will need to do this manually, just run the following command in a terminal window:

sudo mkdir /var/djondb
sudo chown ‘id -u’ /var/djondb

Sudo allows you to execute admin tasks in your machine, so you can create the folder in the protected area “var”, which is normally used to store the files used by OSX applications.

Once you have the folder create you can run djondb server. Using the terminal type:

djondbd -n
INFO:7fff7b008310: djondbd version 3.5.60107 is starting up.
INFO:7fff7b008310: DBController initializing. Data dir: /var/djondb/
INFO:1042f0000: djondb server ready. Port: 1243
INFO:104373000: web interface ready. Port: 8090. Web Folder: /var/webconsole

Congratulations, you have djondb running and ready to server your application, the -n allows you to run the server in interactive mode, please don’t close this console session or it will shutdown the database. For any other new terminar related elements open a new session of the terminal.

Read more ›

Posted in djondb application, NoSQL Samples Tagged with:

Creating a product catalog with djondb, PHP Restler and Backbone JS Part I – NoSQL Concepts


In these series of posts, we’ll create a sample application using djondb database. You’ll see how easy is to create applications using a NoSQL database, especially with djondb, compared to a relational database.

In the first part, we’ll start designing our application and understanding the architecture.

In second part, we’ll install djondb and we’ll use the djondb console.

In third part, we’ll create the REST services that provide the business logic of our sample application.

In fourth part, we’ll create the client application for the final user.

And finally, in fifth part, we’ll add a new feature from database level and see how easy is to make this kind of changes using a NoSQL approach.

So, let’s get started…

Designing the application

The application we’ll create is a simple product catalog. The user will be able to:

  1. Browse available products
  2. See product details
  3. Add a new product to the catalog

We want to use a web browser to use the application, so the client will be an HTML, CSS and JavaScript application created with Twitter Bootstrap and Backbone.js.

Read more ›

Posted in djondb application, NoSQL Samples Tagged with:

Version 3.5 Release

The production ready version 3.5 is now out of the oven.

During the last couple of months I have been working hard to bring the next level of document store, enabling developers to use the power and flexibility of document stores (JSON structures) directly from their favourite dev language to the database, but without losing the power of transactions and consistency.

This new version comes with a new Web Console, easier to access and use than the shell. Just start your server and go to http://localhost:8090 to use it, this supports DQL as explained in the docs and therefore you will feel like in home if you are coming from any SQL Background.

New improvements in the index implementation and look up algorithm provides faster results, the cursors also had a lot of changes.

New drivers on the way, currently the usual PHP, java, Python, C# and C++ drivers are regularly updated but also I am adding drivers on demand, please drop me an email at info@djondb.com with your driver request and I will do my best to have it ready within a week.

What about the license? this is still the same, no changes here so you will be able to use it for free, in any commercial, private or open source solution. Do you need a special license? let me know at info@djondb.com and I will be sure to provide you with something that works for your organisation requirements.

In summary I’m very proud of this new version, and I want to hear from you, let me know in which application you are using djondb and I will be happy to add your logo to our customer list, also referencing your web site.


Posted in Uncategorized

Version 0.33

A couple weeks ago I released the new version 0.33, this new version included a new implementation of the indexes, server-side cursors to improve speed of results, internal memory managers that speed up the process, workers that allows to control concurrent operations, new Windows Service, new Linux server daemon scripts, etc.

I’m really happy with this new version, is working with a very acceptable performance in the projects I’m using it.

The only thing that I stopped doing was working on the social medias, I relay on you to spread the word.

Thanks for supporting djondb.


Posted in Uncategorized

What are we working on?

At this moment we are working on the following ideas:

1. Cursors: The idea of returning the full result is causing performance problems, because most of the time the users does not want to use all the results and the time spend to send the whole set to the client is a killer, the first idea was to send the results asynchronous, but at the end the idea of having server side cursors works best, this is almost finished at this moment and hopefully you will see the changes in the following days in the development branch.
2. Installers and compilation: The change to use CMake instead of autotools on linux/mac and Visual Studio on Windows is working really nice, but the change required to remove the compilation of some drivers from the common scripts, now we are working to include all of these drivers compilation on the cmake files.
3. Service Script: A service script to boot up the server using the normal services on linux.
4. GoLang Driver: A new driver for GoLang

If you have any suggestions please put them on the github project and we’ll enqueue them based on priority and complexity.


Posted in Uncategorized, djondb development

Compile djondb on Mac OSX

This is a step by step process about “Compiling djondb on Mac”, this will cover all the steps, if you already have installed one of the tools (for example home brew) then you can skip the step and move to the next one.

If you have any trouble, please let us know in the comment section, this will be updated with any new release or change to the process.


The first step is to install the xcode and the xcode command line tools, this will allow you to compile, debug, etc. It’s a prerequisite for home brew and djondb. XCode and XCode command line tools can be downloaded from apple’s developer page.

The next step will install home brew, if you already have brew installed in your system please do: “brew doctor” to check that everything is ok.

Home brew

Some of the tools required by compilation scrips will be installed using Home brew, this is a very good installation manager for OSX. The full installation instructions are in the home brew page, basically it will lead you to execute:

ruby -e "$(curl -fsSL https://raw.github.com/mxcl/homebrew/go)"

Once it’s installed please execute the command: brew doctor, which will check for anything that could be wrong on brew and fix the suggestions made by it.


Now with brew installed you could install some tools that will be used later:

brew install cmake
brew install git
brew install boost

Download the code

djondb source code is available at github, to download it just execute:

git clone git@github.com:djondb/djondb.git
cd djondb
git checkout devel

This compilation process is available at the current development branch, later this will be transferred to the master branch.

Make dependencies

This step will take around 15 minutes, it will compile the djondb dependencies. These dependencies will be sit at third_party/libs and third_party/includes folders.

cd <djondb dir>/db
make dependencies

Building djondb

The final step will build djondb.

cd <djondb dir>/db
mkdir build
cd build
cmake ..
sudo make install

Ready, one final step and everything will be set.

Make data dir

the data folder is where djondb files will be created, so you will need to create it.

cd <djondb dir>/db/third_party
sudo sh prepareDir.sh

Done, you are ready to test, run or change the code of djondb.

Posted in djondb development