Enterprise data management for the AI era

The average smartphone today has more than 80 apps, and we are estimated to use at least 40 of them regularly. Do we really need information or data from 40 apps? To complete our daily routine, we need about five good apps – one each for banking, news, travel, food, and maybe chatting. But most of us continue to consume tons of data generated from these apps, perhaps failing to gain important information from any of them by the end of the day.

The data consumption in enterprises today mirrors the above scenario.In the digital realm, enterprises gain access to information from external sources through emails, the web, and social channels. Internally, numerous word, excel, image and PPT files get created, ensuring that managing the information barrage becomes manually impossible. AI-driven processes, the expansion of IoT-connected edge devices, the development of digital twins of things, and the maturing of blockchain are creating new sources of data across environments. The entire data value processing pipeline—ingestion, inventory, blending and preparation, delivery, discovery, and visualization—must adapt to changes in data sources and formats, driving decision making.

Next-gen data operations encompass connecting data creators with consumers to derive the highest returnon investment (ROI). It goes beyond solutions and transforms the culture of the entire organization. Known as DataOps (data operations) methodology, it is an enhancement to the traditional DevOps framework and begins by identifying the business value that organizations are looking to achieve.

DataOps

DataOps is enterprise data management for the AI era. It is a methodology that involves transforming the technology and culture of an enterprise to garner improved data trust, protection, cycle times, insights, and cost-effectiveness. Although new to the IT world, DataOps is seeing an increase in adoption and evolution. Today, many of the prominent hi-tech companies with large DevOps teams have begun implementing DataOps initiatives. Others that have implemented initiatives to enhance data agility are steadilymoving toward DataOps.

Data integration, seamless collaboration amongst team members, and an AI-driven technology approach form the foundations of the DataOps methodology.

Integrate data

Most enterprises own data storage infrastructure with data and analytics software.While these technologies have transformed how data is stored and retrieved,the infrastructure can give mixed results and analyses when it comes to supporting the decision-making journey.

The genesis of this problem lies in data collection and usage—enterprises are trying to glean the right insights from the wrong data sources.Large data hubs (such as warehouses, master data management, data lakes, Hadoop, Salesforce, and ERP) are creating more data silos than seamless operations. Today, 80 percent of an analyst’stime is spent in discovering and preparing data, rather than in analyzing it.Lack of proper data frameworks restricts the usage of data and allocation of resources. New technology must help with integrating data across the four stages of the data journey – storing, enriching, activating, and monetizing. Today, this is even more relevant as machine learning and artificial intelligence pervade data management across enterprises. Enter DataOps. DataOps integrates data from the edge to the core, enhancing pipelines, automating governance, and minimizing friction.

A case in point is Uber’s Michelangelo platform. As an end-to-end DataOps platform, Michelangelo has helped to expand the use of machine learning (ML) models at the company. On a day-to-day basis, it helps to integrate the functioning of multiple machine learning models, deploy hundreds of models into production,drive millions of predictions every second, and allow large teams of data scientists, engineers, product managers, and researchers to collaborate across the company.

Catalyze collaboration

DataOps is part-data, part-tools, and part-people. People play a key role in data management, and they must also be encouraged to avoid data hoarding and to share data and input for realizing a common goal. Therefore, building a culture of trust, shared consciousness, and friendliness is key to greater success. Chief information officers must aim to remove hostility within teams and foster trust within and among teams. A ‘team of teams’ must brainstorm together to ensure that the enterprise has the big picture and that every stakeholder is invested in problem solving. The more the perspectives to solving a problem, the richer the solution.

Optimize data with and for AI

Artificial intelligence (AI) is pervading the daily operations of enterprises. It has an impact on both the internal and external ecosystems of a business. By setting new benchmarks in speed and innovation, it puts pressure on enterprises to enhance operational performance. According to more than 60 percent of the respondents of an IDC Survey, the top challenge for data or artificial intelligence-related projects is that they are too time consuming, especially in getting data. Thankfully, AIOps resolves the pressing challenges that enterprises face with data. As a subset of DataOps, it leverages AI to provide predictive analytics and intelligent automation. Enterprises can predict new requirements in data storage and budgets in advance. They can also accelerate root-cause analysis drastically and minimize errors that affect uptime, customer experience, and performance. As a catalyst to autonomous operations, AIOps also drives seamless integration of tools and automates many simple tasks to enhance speed and performance.

A recent research on DataOps by 451 Research revealed that enterprises that have adopted DataOps are more capable of executing digital transformation and cloud strategies, giving them a competitive advantage over rivals. It also finds that DataOps has the potential to not only enhance operational efficiencies, but also augment data security and catalyze transformation.⁶

The time to modernize IT infrastructure for the next wave of data is now. By embracing the new wave of DataOps in data management practices, CIOs and digital leaders will be able to enhance agility, speed, decision making, and performance of enterprises. To deliver the actual advantages of digital transformation, leaders will have to put data at the core of the enterprise. They must integrate data silos, managethe different data types, and create a culture of collaboration within the enterprise. The key is to allow software like AI and ML to intelligently automate simple tasks and enable IT teams to address more complex challenges.

• Sanjay Agrawal, Technology Head, Hitachi Vantara