Polyglot Persistent – Tools for the right job

After years of working with IT, as developers we tend to solve problems in similar ways, off course we are humans and we tend to think that the same hammer applies to all kind of problems, but the Internet of Things, where applications are not as straightforward as they used to be, are forcing us to use our brains.

Recently I have been tasked to create an ecommerce application allowing customers to pick from a range of configurations and enable them to provision their services without any other help than the web app, these services include Virtual machines, email, Active directory configuration, etc. Part of the requirement is to enable the platform to grow and be flexible to include new configurations and options based on the customers requirements.

This last requirement requires a system that adapts to changes, like adding new fields, or data that helps the system to achieve the wide range of parameters required for each service and mix of configurations. Also been an ecommerce app, that will charge money to customers, requires consistency and reliability.

How to achieve flexibility in terms of fields without having a deployment problem for every new change? Metadata programming to the rescue, instead of hard coding screens and options we created a set of tables, namespaces as they have been stored in djondb, defining the screen structures, fields, validations, etc. And code an engine that draws everything based on this metadata. This metadata programming is something really hard to archieve using RDBMS, as any new field requires alter tables and update a lot of data, meaning stopping services for new patches, but doing this with djondb is simple and doesn’t require breaking your head, and production at the same time.

On the other hand there’s data structure that is pretty much static and it does not require hard changes over time, things like customer information, orders (who, when, price, etc), etc. Most of these are perfect for a Relational Database, also it will be required to be reported, etc.

Metadata problem

An oversimplified version of the metadata looks like this:


Form: Server Order
Fields: [ RAM, CPU, OS, Data Center, Admin Password, IP Address ]

Off course, each field will have type, required flag, Min length, etc cialis gel. So our simple structure will really look like this:

{
  form: 1,
  description: "Server Order",
  fields: {
      description: { type: "string", caption: "Description", len: 100, required: true },
      RAM: { type: "dropdown", caption: "RAM", list: "ram", required: true },
      CPU: { type: "dropdown", caption: "CPU", list: "cputypes", required: true},
      AdminPassword: { type: "password", caption: "Password", required: true }
  }
}

Now, it’s really simple to create a routine in jquery rendering the above dynamically, and sending the data back to the server in the form of a JSON structure, both structures may grow without changing the code. The code will require changes if a new field type is supported, like “radio”, or others, but both server and client are flexible to render and store the information. Here I’m just showing one of the possible order type, now extend the concept to Email, where you will need to store the user name, password, size, etc. and a full range of other services that users will be able to provision using the platform. On the other hand adding new fields to djondb is not a big problem, just send the new JSON structure and djondb will take care of it.

Processing the order requires a similar approach, you will script some plans based on the above, so using a templating system at the server side that replaces values into script and executes the script is a no brainer. Here using a transactional system over this data makes sense as you need to update the order as soon as the provision is ready, but also store the provision information for later maintenance. No Problem with djondb you have both, you can store atomic multi document transactions and use JSON at the same time.

Relational Data goes to a RDBMS

NoSQL is not really a tool for replacing your full system, it is one of the many tools that you will use to build your application. So we kept relational in a relational database, their infrastructure is based on Microsoft, therefore SQL Server is their preferred choice, information like invoices, customer data, details, etc. where the data structure are almost fixed will live into a RDBMS, linking keys across environments is not really a problem. When dealing with dynamic information that will change over time and requires constant changes will go to djondb, whilst fixed structures are processed in SQL Server.

Another example on where to use the right tool for the right problem is text searching, nor SQL Server or NoSQL are really good for searching in a google style search, for this elastic search helps you to index your texts and provide results that will be linked to djondb and others.

Key decisions to support your architecture

– Fixed data, with no changing structure whatsoever, RDBMS (Pick your choice, SQL Server, Oracle, MySQL, etc)
– Unstructured data requiring transactions (Not too much choice in here, djondb)
– Text search capabilities (Elastic Search)

Now mix all of this in your application and you will use the right tool for the right job, implement your system without any hassle and make your customers happy, using multiple persistence storages into the same application is known as Polyglot persistence.

Hope you like this and let me know your comments, and your approaches to similar problems.

Version 3.5 Release

The production ready version 3.5 is now out of the oven.

During the last couple of months I have been working hard to bring the next level of document store, enabling developers to use the power and flexibility of document stores (JSON structures) directly from their favourite dev language to the database, but without losing the power of transactions and consistency.

This new version comes with a new Web Console, easier to access and use than the shell. Just start your server and go to http://localhost:8090 to use it, this supports DQL as explained in the docs and therefore you will feel like in home if you are coming from any SQL Background.

New improvements in the index implementation and look up algorithm provides faster results, the cursors also had a lot of changes.

New drivers on the way, currently the usual PHP, java, Python, C# and C++ drivers are regularly updated but also I am adding drivers on demand, please drop me an email at info@djondb cialis generika kaufen rezeptfrei bestellen.com with your driver request and I will do my best to have it ready within a week.

What about the license? this is still the same, no changes here so you will be able to use it for free, in any commercial, private or open source solution. Do you need a special license? let me know at info@djondb.com and I will be sure to provide you with something that works for your organisation requirements.

In summary I’m very proud of this new version, and I want to hear from you, let me know in which application you are using djondb and I will be happy to add your logo to our customer list, also referencing your web site.

Cross

Version 0.33

A couple weeks ago I released the new version 0.33, this new version included a new implementation of the indexes, server-side cursors to improve speed of results, internal memory managers that speed up the process, workers that allows to control concurrent operations, new Windows Service, new Linux server daemon scripts, etc cialis kaufen schweiz.

I’m really happy with this new version, is working with a very acceptable performance in the projects I’m using it.

The only thing that I stopped doing was working on the social medias, I relay on you to spread the word.

Thanks for supporting djondb.

Cross

What are we working on?

At this moment we are working on the following ideas:

1. Cursors: The idea of returning the full result is causing performance problems, because most of the time the users does not want to use all the results and the time spend to send the whole set to the client is a killer, the first idea was to send the results asynchronous, but at the end the idea of having server side cursors works best, this is almost finished at this moment and hopefully you will see the changes in the following days in the development branch.
2. Installers and compilation: The change to use CMake instead of autotools on linux/mac and Visual Studio on Windows is working really nice, but the change required to remove the compilation of some drivers from the common scripts, now we are working to include all of these drivers compilation on the cmake files.
3. Service Script: A service script to boot up the server using the normal services on linux.
4. GoLang Driver: A new driver for GoLang

If you have any suggestions please put them on the github project and we’ll enqueue them based on priority and complexity.

Cross