Last year our team was quite busy assisting our customers on some varied technical assignments. Engagements ranged from consulting on highly scalable web-centric applications to designing Big Data solution to implementation of highly available database and web-infrastructure. During these engagements we saw that some database trends were quite dominant viz the adoption of Big Data, need to store unstructured data, scalable web facing application etc. The engagements and discussions we had last year left us pondering about where does it go from here. Will Big Data sustain in 2016? What will happen to NoSQL technologies this year? Well, we are not crystal gazing and so it is difficult to give definite answers to these questions. But from the discussions we have had with our customers we are sharing key findings here. We can see that not only Big Data and NoSQL concepts will sustain but they will form the foundation of trends in 2016 and years to come.

1. Real Time Big Data Analytics

Well, Big Data is not new neither is Analytics. Even Big Data Analytics is a popular term by now and we have seen a few use cases over the years on Big Data Analytics. But what the world will see in this year is more and more application of Big Data related technologies to do Real Time Big Data Analytics, i.e. crunching your data and acting on it pretty much online. Analytics would be used for making decision and someone might call it as emergence of Fully Automated Online Decision Making Systems. The use cases could be from Financial Industry to block a fraudulent transaction or from E-commerce to offer a deal to a customer who is about to walk-away from his online wishlist/cart. Emergence of Big Data technologies to store and process humongous and heterogeneous data has given rise to the belief that decision can be made based on data which we already have and data which is being generated in real time. Spark and Storm are the technologies which have been implemented by some of the early adopter of this technology. Cloudera made an announcement in September 2015 that it is investing heavily in developing enterprise grade Spark and in uniting Apache Spark and Hadoop. Ashnik, a partner with Cloudera enables our customers to realize the benefits of Big Data in mainstream IT projects.

2. More data will be stored in memory

Storing data in memory and caching has always been popular. The applications which has very stringent response time towards end-user e.g. authorization applications, trading platforms etc., are known to cache the whole data set which ranges from few MB to hundreds of GB. What we are observing as a gradual change isl, now people are storing transactional data and historical data too in memory. The motivation too has changed, people are storing data in memory so that they can process them faster to make decision online without delay. Mobile Advertisement, E-commerce are some of those industries where the early adopters have come from. Soon enough with Real Time Analytics becoming more popular and with squeezing time windows which are available to make a decision, we will see more and more data being stored in memory. This will also go with the fact that now it is possible to have servers with really large memory and they are affordable too. While the world of in-memory databases and data store has largely been dominated by proprietary solutions and appliances, there are some really good Open Source alternatives available e.g. MonetDB and VoltDB . Also with certain tweaks it is possible to use one of your ‘traditionally-on-disk’ databases for example MonogDB to be used as in Memory store.

3. Bridging of the gap between NoSQL databases and Relational Databases

At Ashnik, having an expertise on both PostgreSQL and MongoDB, we have often received a common question from many customers “which one and why?” And our explanation usually has been different from the explanation we would have offered yesterday to another customer for the very same question. This is simply because every customer environment is different and so are their requirements. While the enterprises can still be seen mulling over the decision if to adopt NoSQL or to continue with RDBMS, people from both the worlds (NoSQL and RDBMS) are seen arguing over the applicability of each other. What most have missed to notice is, smarter enterprises have chosen to implement hybrid setups which incorporates both NoSQL and RDBMS. It could be a simple setup with two different applications using NoSQL and RDBMS or a setup where-in the entire solution could not be achieved with a RDBMS or a NoSQL technology alone. The journey was also made easier with the rise of technologies like Foreign Data Wrapper from PostgreSQL community which allows you to query NoSQL stores like MongoDB from PostgreSQL and Data Integration tools like Pentaho Data Integration providing connector from various NoSQL stores. Ashnik team has worked on a few such projects last year where NoSQL and RDBMS technology was involved at the same time. We shared one such use-case via our Google Hangout. This year we expect this to be taken to whole new level with bridging the features set in both kind of technologies. We will see more NoSQL databases developing features which are already there in RDBMS or providing toolsets to achieve them. We have already heard and seen many NoSQL vendors talking, both subtly and openly about adding ACID compliance, data integrity checks or providing compatibility to process/translate SQL commands. We also saw how RDBMS vendors and development communities are adding some basic capabilities offered by NoSQL stores e.g. sharding, capability to store unstructured and heterogeneous data. In 2016 we will see more such features being exchanged between the two worlds that exists in data world. Whether one will be able to get past the other one is something that time will tell. But the concepts which NoSQL database have introduced will certainly survive and so will the long existing characteristics of RDBMS.
Stay tuned as we will bring more on-field reports from our engagements on how we are seeing these technologies and of course older trends progressing in 2016.