NoSQL - To be or Not To be, that is the question

By: Shrikant Navelkar, Director – Analytics and Big Data, Clover Infotech

Database technology has come a long way since IBM began its research project “System R” way back in 1974. This was followed by Oracle, who made Relational Database commercially available for the first time in 1979. For next 30 years, which is fairly a long period of time considering the pace of technological advancements, database technologies kept innovating and trying to reinvent itself but more or less in isolation. It failed to adopt to new changes that the programmer community was demanding in the early Internet era. Database technology vendors were happy with their dominating market share but were blind to emerging technology requirements of data driven applications in early 2000.

During the same period, companies such as Google, Yahoo and Amazon realized that they cannot depend on RDBMS technology for their vast data volumes and requirement of unique data modelling. These companies, then,started innovating by adopting open source culture and developed end-to-end ecosystem to manage their business requirements. The Open source movement made these innovations highly collaborative and at a blistering pace. And the gap between traditional database technology and these innovations widened further.

The emergence of NoSQL

NoSQL (originally referred as non-relational) is one such innovation. It is now providing competition to traditional RDBMS vendors and is posing a serious challenge to their three decade monopoly. Interestingly, NoSQL databases existed since late ’60s but its popularity surged in the first decade of twenty-first century due to adoption by Web 2.0 companies such as Facebook, Google, Amazon and many others for whom data was the single source of revenue.

NoSQL addresses needs of non-relational data model, distributed data stores, horizontal scalability and more importantly design agility demanded by programmers. MongoDB (earlier known as 10gen) is a leader who provides NoSQL in open source model along with subscription based technical support and other services.

Eliot Horowitz, MongoDB, CTO and co-founder, explains fundamentals of their design philosophy. He says “MongoDB wasn’t designed in a lab. We built it from our own experience building large scale, high availability robust system”. Interestingly, his team didn’t start from scratch. They considered what was broken and missing in another open source database, MySQL. They changed the data model from relational to document based. This paradigm change provided features such as high data retrieval speed, foundation for agile development, scalability and robustness required for Internet wide deployment.

It is interesting to study what is missing in traditional relational databases. For many years, relational databases provided underlying data modeling, data retrieving and application development platform for mission critical business applications. We can understand that these features were not enough for new generation companies such as Google and Facebook but aren’t they enough for our traditional business applications? The answer will depend on how you define scope of your business applications.

If you are happy with transaction processing applications and generating hundreds of reports, through it, then it is better to stick to RDBMS. However, if you are planning to add analytics flavor to your transactional applications, then you will be exposed to limitations of underlying RDBMS foundation. The real consumers of your applications, your business users, are never happy with transactional nature of reporting. They need intelligent reports generated by mashed up data. Companies tried to address this through business intelligence applications and data warehouse systems.

However, as businesses entered in matured Internet era (post 2010), business users started demanding need of incorporating external data to capture their customer behavior. Simultaneously, experts predicted that 80% of data generated by companies would be in unstructured format and would not be consumed by business applications. The combined effect of these two factors forced companies to relook at their business applications and RDBMS was well short of providing these capabilities.

Does that mean your traditional applications are becoming obsolete and you need to redevelop these applications from scratch? Fortunately, not. There is still some juice in your traditional applications and you can always develop new applications in NoSQL and make them co-existent. Your application and database architectures need to adopt to these new demands as fast as it possibly can.

Key drivers for NoSQL framework

So, what are the key drivers to consider NoSQL framework in your IT environment? I think, they can be listed as below:

a. You need to begin by being open about the fact that your new set of applications need to be business applications in the “true sense”. You need to capture processes, workflows, and more importantly external and unstructured data. It is not advisable to shy away by saying that your business environment and users don’t need external data. You will be surprised how fast and well your business users will consume it.

b. Analytics is not business intelligence or traditional data warehouse systems. The predictive nature of Analytics applications will add immense business value. Once you have decided on analytics as your priority, you need to invest enough bandwidth for NoSQL adoption.

c. There is an ever increasing demand to make your existing applications available to mobile users. The days of struggling to implement a newly deployed web browser based application on tiny mobile screens is long gone. Steve Jobs launched iPhone and changed (permanently) the way applications will run on smart devices. NoSQL (MongoDB) is an ideal platform to migrate some of the key workflows of your existing applications to smart devices. Needless to say, it provides a strong business case as a platform of choice to develop new mobile applications.

d. If the businesses are changing so fast, can your applications be away from social analytics, consumer behavior tracking, Internet of Things (IoT) for long? Why not to start preparing for a strong technology foundation before the business demands it?

e. Adoption of NoSQL or MongoDB in your IT enterprise is not as complex as adoption of PC based applications over Mainframes. It’s a logical shift. You need to leverage your investment in RDBMS applications, SQL, Linux and database clustering knowledge and open up to agile development method.

f. NoSQL and particularly MongoDB has made massive architectural improvements to bridge the gap between strong technology foundation of RDBMSs and nearly unlimited flexibility it offers to developers.

g. The ACID properties are guaranteed in traditional RDBMSs. MongoDB’s ACID compliance is a game changer which will unfold in the years to come.

I will conclude by saying that it is time for organizations to relook their application architecture and underlying technologies. I would recommend considering MongoDB as a platform for new application development. You can start by making a focused team with internal and external expertise, develop and test some useful pilots and slowly, you will realize that you are well poised to develop new generation applications on a NoSQL framework. The subsequent results would have addressed the perpetual “To be or not to be” question that you may have with regards to adoption of NoSQL technologies.