Polyglot Persistent – Tools for the right job

After years of working with IT, as developers we tend to solve problems in similar ways, off course we are humans and we tend to think that the same hammer applies to all kind of problems, but the Internet of Things, where applications are not as straightforward as they used to be, are forcing us to use our brains.

Recently I have been tasked to create an ecommerce application allowing customers to pick from a range of configurations and enable them to provision their services without any other help than the web app, these services include Virtual machines, email, Active directory configuration, etc. Part of the requirement is to enable the platform to grow and be flexible to include new configurations and options based on the customers requirements.

This last requirement requires a system that adapts to changes, like adding new fields, or data that helps the system to achieve the wide range of parameters required for each service and mix of configurations. Also been an ecommerce app, that will charge money to customers, requires consistency and reliability.

How to achieve flexibility in terms of fields without having a deployment problem for every new change? Metadata programming to the rescue, instead of hard coding screens and options we created a set of tables, namespaces as they have been stored in djondb, defining the screen structures, fields, validations, etc. And code an engine that draws everything based on this metadata. This metadata programming is something really hard to archieve using RDBMS, as any new field requires alter tables and update a lot of data, meaning stopping services for new patches, but doing this with djondb is simple and doesn’t require breaking your head, and production at the same time.

On the other hand there’s data structure that is pretty much static and it does not require hard changes over time, things like customer information, orders (who, when, price, etc), etc. Most of these are perfect for a Relational Database, also it will be required to be reported, etc.

Metadata problem

An oversimplified version of the metadata looks like this:

Form: Server Order
Fields: [ RAM, CPU, OS, Data Center, Admin Password, IP Address ]

Off course, each field will have type, required flag, Min length, etc cialis gel. So our simple structure will really look like this:

  form: 1,
  description: "Server Order",
  fields: {
      description: { type: "string", caption: "Description", len: 100, required: true },
      RAM: { type: "dropdown", caption: "RAM", list: "ram", required: true },
      CPU: { type: "dropdown", caption: "CPU", list: "cputypes", required: true},
      AdminPassword: { type: "password", caption: "Password", required: true }

Now, it’s really simple to create a routine in jquery rendering the above dynamically, and sending the data back to the server in the form of a JSON structure, both structures may grow without changing the code. The code will require changes if a new field type is supported, like “radio”, or others, but both server and client are flexible to render and store the information. Here I’m just showing one of the possible order type, now extend the concept to Email, where you will need to store the user name, password, size, etc. and a full range of other services that users will be able to provision using the platform. On the other hand adding new fields to djondb is not a big problem, just send the new JSON structure and djondb will take care of it.

Processing the order requires a similar approach, you will script some plans based on the above, so using a templating system at the server side that replaces values into script and executes the script is a no brainer. Here using a transactional system over this data makes sense as you need to update the order as soon as the provision is ready, but also store the provision information for later maintenance. No Problem with djondb you have both, you can store atomic multi document transactions and use JSON at the same time.

Relational Data goes to a RDBMS

NoSQL is not really a tool for replacing your full system, it is one of the many tools that you will use to build your application. So we kept relational in a relational database, their infrastructure is based on Microsoft, therefore SQL Server is their preferred choice, information like invoices, customer data, details, etc. where the data structure are almost fixed will live into a RDBMS, linking keys across environments is not really a problem. When dealing with dynamic information that will change over time and requires constant changes will go to djondb, whilst fixed structures are processed in SQL Server.

Another example on where to use the right tool for the right problem is text searching, nor SQL Server or NoSQL are really good for searching in a google style search, for this elastic search helps you to index your texts and provide results that will be linked to djondb and others.

Key decisions to support your architecture

– Fixed data, with no changing structure whatsoever, RDBMS (Pick your choice, SQL Server, Oracle, MySQL, etc)
– Unstructured data requiring transactions (Not too much choice in here, djondb)
– Text search capabilities (Elastic Search)

Now mix all of this in your application and you will use the right tool for the right job, implement your system without any hassle and make your customers happy, using multiple persistence storages into the same application is known as Polyglot persistence.

Hope you like this and let me know your comments, and your approaches to similar problems.

Posted in Uncategorized.
Loading Facebook Comments ...

Leave a Reply

Your email address will not be published. Required fields are marked *