Sentiment Analysis/Opinion mining: The new breed of data analysis

By: Suvro Banerjee, Managing Director, Fuzzy Logix  

“What are various aspects of my store or website that visitors like and dislike? Why do a large percentage of visitors to my website abandon their shopping cart just before checkout? What kind of merchandise do my customers want to buy? How can I retain my most valuable customers?”- These are critical questions that marketing managers often aspire to get insights on. Unfortunately, however, their valiant efforts have not yielded results- till now!

With emerging technologies that enable analysis of Big Data at blazing speed, organizations are able to understand their customers intimately and align their business operations accordingly. Data analytics helps them analyse consumer data and develop deeper understanding of consumer patterns/preferences and general market trends. Two high-potential areas - opinion mining and sentiment analysis (used interchangeably in this discussion), are becoming the new favourites of marketers and brand managers.

Today, it will not be an exaggeration to say that everyone is directly and indirectly heavily dependent on the internet. Modern society uses the internet for a variety of reasons, from shopping to paying electricity bills, from watching movies online to enjoying an evolved gaming experience with multi-player gaming. Social media has enabled us to connect with family and friends located in far-flung areas instantly! With accessibility to such ease and comfort, what used to be called the social network of an individual has been redefined as a cross-channel hybrid engagement network. The impact of these networks on society is such that ‘social networking’ today is aptly translated as connecting with peers on social media, the online platforms. These online social networking channels leave massive breadcrumbs that a business organisation and even government can take advantage of to grasp public opinion. Sentiment Analysis is about mining data from social media to better understand the likes and dislikes of the target audience.

Sentiment analysis helps a company or an organisation specifically design products and services that are based on their target audience’s preferences. As a result of using such analysis, businesses and brand marketers receive warmer responses, earn loyalty, and simultaneously increase their market share. Today, the scope of social media is vast. Social channels host a total of 2.22 billion people worldwide. At present, Facebook has 1.65 billion monthly active users on its platform. The same figure for Instagram stood at 500 million in the first quarter of 2016. The number of active users on Twitter and LinkedIn are currently pegged at 313 million and 433 million respectively. Many regional social networks such as Qzone, Sina Weibo, Badoo, and Renren also cater to a significantly large population in their respective areas. This has become a compelling reason for business organizations to mine data available on these hotbeds of activities, sentiment and opinions in order to maintain coherence with present market demands.

It is beyond any doubt that the information which is constantly being generated and is readily available on these social networks is by and large Big Data- featuring volume, velocity, and variety in an unprecedented manner. The data, despite being readily available, is relatively tough to analyse given the exponential speed that it simultaneously increases with. To decode this greatly enriched high-quality data, advanced technologies such as NLP and text-mining are finding increasing use. These technologies are highly sophisticated and relatively hard to implement, because they have to understand the meaning as well as the general context of the vocabulary. Should the context of a sentence be wrongly processed, the analysis will yield wrong erroneous results. Additionally, given the massive velocity and volume of the data, it is desired that algorithm execution should maintain pace with the content generation speed to be relevant. In-Database Analytics becomes a compelling proposition here because of its sheer speed,efficiency and lower cost.

In-Database Analytics is a technology that enables data analysis to be executed within the database itself by bringing the analytics engine to the data rather than moving the data to the analytics server. This eliminates the time-consuming data exchange between the Data Warehouse and Analytics Platform. Eliminating the cumbersome activity of data management and manipulation, it provides data scientists the opportunity to focus on developing advanced models for greater effectiveness and accuracy. Organizations are able to run models on very large volumes of structured and un-structured data that was not possible with the erstwhile paradigm of having to pull data out before models are executed.

Sentiment analysis has emerged as a strategic weapon for many organizations to gain customer loyalty and market share. These insights are being used for predictive analytics, which forecasts the market acceptance of a product, service, or even a government reform before it is introduced. For example,, governments around the world are relying on such analysis to understand sentiments of its citizens, and in likelihood of civil unrest, take proactive measures to prevent protests and violence.

Revisiting the questions we presented at the beginning, opinion mining/sentiment analysis are yielding valuable insights like – we don’t carry the brands that my customers like, it’s too hard to find stuff on my website, our payment gateway often crashes, or our customer service is poor. These insights are so valuable that they can make a difference to the survival of a business!

Modern technology is opening new horizons for increasingly accurate and effective analysis, helping organizations cut marketing costs across industry verticals.. Sentiment Analysis, therefore, will play a critical role across all dimensions of the market, witnessing new players, new paradigms, and new ideas changing the world we live in.