Confluent in India: AI, Real-Time Streaming, and Data Security

In an insightful conversation with DataQuest, Will LaForest and Rohit Vyas from Confluent discuss their strategic focus on India’s rapidly expanding data streaming landscape. As the APAC region’s fastest-growing market, India plays a vital role in Confluent's global growth trajectory, powered by Kafka.

The executives shed light on how Confluent’s innovations are meeting diverse industry needs in India, from banking and retail to government services. They delve into Confluent's pioneering work in AI integration, real-time data processing, and multi-layered security, underscoring the company's commitment to building a complete data streaming ecosystem tailored to emerging industry requirements.

Excerpts:

DQ: Could you share your perspective on the Indian marketplace, particularly in data streaming?

Will: According to leading market research reports, APAC is the fastest growing region with India contributing the most to growth and consumption for APAC. From a business perspective, we're heavily investing in India, with about 15% of our workforce based here. This includes talent across engineering and services. Additionally, India is progressing faster than many other regions, sometimes even skipping over certain innovation cycles. The rate of growth here is unmatched because, in some cases, India is moving so fast that it jumps over traditional stages of technology. The potential here is enormous, and I expect India to become one of our most important markets globally.

Rohit:  Adding to that, as the Head of India for Solutions Consulting and Customer Success, I work closely with Will to strategize our go-to-market approach and ensure customer success. In terms of citizen-scale services, India is truly a global leader. The way the government incentivizes the use of data to offer services at this scale is remarkable. Confluent plays a critical role here, especially as a pioneer in data streaming technology through Kafka. With a significant presence in India, we're proud of the traction we have with leading companies across multiple sectors, including food delivery, digital retail, banking, and finance, and even within the public sector. The foundation we have built here is strong, and the opportunities are boundless.

DQ: With such rapid growth, have you collaborated with Indian startups or enterprises to support this expansion?

Rohit: Traditionally, we've run programs specifically for digital-native startups, especially in Bangalore and Mumbai. Last year, we held a Data in Motion Tour where we collaborated with startups to help them leverage Confluent's technology. Part of our team is focused on working closely with developers. The developer community here has a major role in tech adoption, and we have organized various events like the Kafka Summit in Bangalore. This summit, which was held for the first time in India earlier this year, helped strengthen our relationships with developers who play an influential role across APAC.

Will: From a product strategy standpoint, we recognize that it's essential to make our platform as accessible as possible for startups. Our investment focuses on ease of use, ensuring that even a small startup with limited resources can start with Confluent seamlessly. This approach is particularly beneficial in a high-growth market like India.

DQ: Each industry has its specific performance needs. How has Confluent adapted to meet these demands?

Will: When we started this cloud journey, we initially had only one product version. Now, we've diversified our offerings, with up to six versions, each with unique performance characteristics. For instance, we have a more cost-efficient option for businesses that don’t require real-time speed, and a high-performance, dedicated option for banking clients, where system uptime is critical. This adaptability means that each client, whether in finance, retail, or the public sector, can find the right solution for their needs. The acquisition of WorkStream is one of the latest examples of us tailoring our offerings.

DQ: With the rise of AI, how do you see data streaming evolving, and how is Confluent integrating AI operations?

Will: Data streaming has always been used in AI, but generative AI has expanded its role significantly. Previously, data streaming was primarily used to feed data into data lakes or warehouses for later analysis. However, with generative AI, real-time data engineering has become crucial. Instead of waiting for batch processing, all data processing now needs to happen in real-time. For instance, when you interact with a generative AI model, it must gather data from various sources, engineer the prompt, and deliver a response instantly. We’re investing heavily to make this integration seamless by enhancing our compatibility with popular AI ecosystems and enabling AI models to be used directly within streaming data pipelines.

Rohit: AI has evolved from traditional analytical AI, which has been around since 2014, to generative AI, which is more complex and requires real-time data processing. Many of our customers already use Confluent within their AI data pipelines. Generative AI systems need constant data ingestion for model training and inferencing. Our platform supports this by providing data governance, which is crucial in AI applications. We add value to the entire AI workflow, from data ingestion to prompt generation, and we partner with various technology providers to achieve this.

DQ: Data security and governance are essential, especially in a multi-cloud environment. How does Confluent address these concerns?

Will: Our approach to security includes multiple layers, from dedicated clusters and role-based access controls to detailed auditing and data lineage tracking. Security by layers is a gold standard, and Confluent adheres to this practice across all our offerings. Additionally, our governance tools enable companies to monitor data usage, track its flow, and ensure compliance with security protocols. This combination of layered security and robust governance helps us secure data across a variety of use cases.

DQ: What challenges did you face initially when introducing real-time data streaming to Indian enterprises, and how did Confluent address them?

Rohit: When we started in India, Kafka was already well-known in the open-source community, but many businesses weren’t familiar with the added benefits of our enterprise version. We had to convey the additional value we bring, like the enterprise support and the advanced intellectual property we offer, which includes enhanced security and scalability. India is a cost-sensitive market, so we had to emphasize our role as the creators of Kafka and explain how our roadmap aligns with their business needs. Now, our message resonates well, as evidenced by the strong turnout at events like this one.

DQ: What's next for Confluent, and how do you plan to continue your growth in the industry?

Will: Moving forward, we’ll keep investing in AI, particularly in stream processing with Flink. Our focus remains on the four key pillars—stream, connect, process, and govern. We are working to advance each of these pillars to address emerging industry needs. Stream processing is key, and with Flink, we’re adding new capabilities and APIs to improve data analysis in real-time. We see a lot of potential in expanding our stream processing capabilities to create even more value for our clients.

Rohit: To summarize, Confluent’s core focus is on building a comprehensive, complete data streaming platform. We are unique in that we address the full range of needs in data streaming, from integration to governance, and we will continue to invest in this completeness. The sky’s the limit in terms of where we can take this.