Elevating Enterprise Data Architectures with Aerospike

In today's data-driven world, businesses are constantly seeking ways to modernize their data architectures to stay competitive and efficient. Cloud and edge applications have become pivotal in this journey, demanding real-time data processing capabilities. To shed light on these evolving data architecture trends and the role of a real-time data platform, Minu Sirsalewala, Executive Editor – Special Projects chatted with Srini V Srinivasan, Founder and Chief Technology Officer of Aerospike, a leading player in the data management space. In this interview, Srini shares insights into the challenges organizations face in designing data architectures, the evolution of globally distributed databases, the impact of graph databases like Aerospike on Google Cloud, and how Indian businesses are leveraging real-time data platforms for global competitiveness.

Can you elaborate on how data architecture is evolving to meet the demands of both cloud and edge applications?

The way I think about data and the experience I had with Aerospike, it’s all about leveraging data in real-time. You talked about cloud and edge, and whether it's cloud-native or cross-cloud databases, the key is processing AI/ML tasks on recent data. Real-time data doesn't always mean sub-millisecond or sub-10 milliseconds latency; it may be near real-time. The more important thing is that whatever the application being used, there is a necessity to make the best possible decision based on the most recent events and the data gathered from there and that's kind of the main focus of Aerospike in that context.

There are a lot of systems we've seen over the years that we have replaced, so to speak, but the point is, there are traditional systems, which essentially implement certain ways of data processing, which are not necessarily capable of handling the heavy load of Read and Write capabilities that are required to make real-time decisions on the edge. These decisions also had to be made on the cloud, because that's a non-negotiable thing, virtually every customer of ours is either already on the cloud or moving applications to the cloud. Some of them, of course, keep mature applications running on their own data center on their private networks but that is only a small subset these days. Most new deployments are happening on the public cloud or even some other private clouds and so on, but it's all quite a dynamic area right now.

What are the key challenges organizations face when designing data architectures for these environments, and how can a real-time data platform address them?

One of the key challenges is handling high-scale transactions, especially in scenarios like real-time payments. Multiple parties must interact seamlessly with a broker in the middle, all meeting stringent SLAs. Aerospike specializes in high-performance systems capable of handling these heavy loads. Traditional systems often fail under such pressure.

For instance, let's consider the UPI payments revolution, which has propelled India to the forefront of financial transactions. India now leads in the number of transactions, surpassing countries like Brazil, China, and the United States. This remarkable achievement has been made possible by the introduction of open interfaces and the seamless interaction of various stakeholders. It's worth noting that a similar concept had previously revolutionized the Ed-tech industry about a decade ago. The crucial aspect here is the need for immediacy. For example, when making a UPI payment on a platform like Flipkart, the system cannot afford delays. It must confirm the payment promptly to enable the purchase. Other systems tend to face challenges, particularly at high scales, leading to failures, which is where our specialization lies.

Consumer-based activities in India often operate at an unparalleled scale globally. Whether we look at UPI payments integrated with giants like Jio, or platforms like Flipkart, the sheer magnitude of real-time consumer interactions is remarkable. What becomes evident is the pressing demand for technology solutions like ours to empower these applications in a manner that ensures the first customer enjoys the same seamless experience as the billionth one. Achieving this level of consistency and scalability is undeniably challenging, but it's a challenge we are dedicated to addressing.

Could you provide insights into the evolution of globally distributed databases and their significance in today's data landscape?

Classical databases like Oracle, Db2, and MySQL were not originally designed for distribution. As the internet's demands grew in the early 2000s, they struggled to handle the high read and write rates required for internet applications. This led to the emergence of NoSQL databases like Cassandra and Bigtable, pioneered by companies like Google, Facebook (formerly Meta), and Yahoo.

Aerospike, founded in 2009, improved upon Cassandra's distributed architecture. Other NoSQL databases like Yugabyte and CockroachDB also entered the scene, addressing the needs of internet and mobile applications.

Additionally, specialized databases like graph and time series databases evolved, and some databases became multi-model, supporting multiple data models on a single architecture. This evolution in globally distributed databases is ongoing and is expected to continue for the next five to ten years.

What do you see as the future trends and innovations in globally distributed databases, and how does Aerospike contribute to this evolution?

Aerospike's contribution to the evolution of distributed databases, particularly in handling high-performance at scale, is a tale of innovation and efficiency.

In the world of modern applications, where giants like PhonePe, Flipkart, Myntra, and international platforms like PayPal and Wayfair operate, the ability to handle immense workloads at lightning speed is paramount. Aerospike stepped up to the challenge with a focus on game-changing solutions.

At the heart of Aerospike's innovation lies the "Hybrid Memory Architecture." This groundbreaking approach harnesses the power of flash storage, enabling lightning-fast data access with sub-millisecond latencies. What's unique is that it makes all the data stored in flash equally accessible in memory, ensuring top-notch performance.

Imagine this: a single node, a standard Linux box, can accommodate a substantial amount of flash storage, ranging from 20 to 30 terabytes. This means smaller clusters with fewer nodes can now handle the same Herculean workloads. The implications are remarkable. Not only does this reduce the total cost of ownership, but it also streamlines power consumption and optimizes data center space usage.

What began as an efficient use of computing resources—processors, memory, network—morphed into something even more significant: sustainability. Aerospike's approach inadvertently led to the creation of more eco-friendly data centers.

One real-world example showcases this transformation. A customer of Aerospike's, having initially relied on hundreds of nodes to manage their operations, scaled down to just a fraction of that number, a mere 1/10th. The result? They were able to shut down two entire data centers. This unexpected outcome underscores the sustainability benefits of Aerospike's approach.

In essence, Aerospike's journey is a testament to its commitment to high-performance at scale. Their innovative Hybrid Memory Architecture not only supercharged data access but also revolutionized how organizations manage their data workloads. Smaller clusters, reduced costs, and sustainable data center practices—all made possible by Aerospike's pioneering spirit.

Aerospike Graph Database on Google Cloud has shown impressive benchmarks. What makes it stand out in terms of performance and scalability?

What truly sets Aerospike's graph database apart is the expertise and dedication of the team behind it. Comprising graph specialists who have an in-depth understanding of graph databases, this team has created specialized libraries to facilitate graph applications. These libraries are designed to work seamlessly with popular graph query languages like Gremlin, which are industry standards in the graph database space.

Here's what happens: while developers typically interact with the graph using these query languages, Aerospike's intermediate layer and graph libraries step in to translate these commands into the Aerospike API. This clever integration not only leverages the incredible speed and scalability of Aerospike but also ensures that the graph is efficiently modelled using Aerospike's data models.

What's truly remarkable is that this effort has produced results beyond expectations. While the team at Aerospike was well aware of the platform's exceptional capabilities in a vanilla environment, the addition of the graph library has exceeded their own expectations. With ongoing optimizations and improvements, Aerospike's graph database continues to evolve into a robust and high-performing solution. The journey is far from over, but the progress made thus far is undeniably impressive.

How can businesses benefit from incorporating a graph database like Aerospike into their data architectures, especially when dealing with large-scale data sets and real-time requirements?

Aerospike's graph database allows businesses to process data closer to the edge, enabling real-time analytics and machine learning. It ensures that decisions are made promptly and accurately, even with large-scale data sets, enhancing customer experiences.

In your experience, how do global technological innovations impact Indian businesses, particularly in the realm of data architectures?

In my view, India stands as an equal partner in this narrative, and I've consistently held this belief. When Aerospike embarked on its journey and secured its first round of funding in 2011, we made a deliberate choice to establish two core engineering teams. At that pivotal moment, with just 10 individuals on board, we appointed leaders for these teams—one based in India, specifically in Bangalore, and the other in the heart of Silicon Valley, USA.

In essence, you could think of it as a tale of two Silicon Valleys—one in India and the other in the United States. This approach has been integral to our identity right from the outset. I firmly assert that India has been an equal and indispensable partner in steering the Aerospike ship. When I alluded to "starting with things outside impacting India," I primarily referred to the realm of business, where external factors have a significant influence. However, on the technological front, our partnership has always been on equal footing.

Our dedicated teams, whether in India or the United States, have collaboratively contributed to building these systems. In today's interconnected world, exceptional technologists can be found across the globe, and we've harnessed this diversity to drive innovation and excellence.

Indian businesses across sectors like e-commerce, telecommunications, and banking have leveraged Aerospike to tackle internet-scale problems and compete globally.

Are there specific examples or use cases where Indian businesses have leveraged real-time data platforms to stay competitive on a global scale?

Examples like Flipkart, Airtel, and PhonePe demonstrate how Aerospike plays a pivotal role. These companies have harnessed Aerospike's capabilities to handle high-scale transactions, reduce operational costs, and deliver real-time services to customers, enhancing their global competitiveness.

Could you share some insights into Aerospike's expansion plans in India and how it aligns with the needs of the Indian market?

India has always held a pivotal position in our strategic outlook. As I mentioned earlier, our commitment to India extends beyond merely establishing a support team; we inaugurated a core engineering team right from the inception of our product development. This reflects our profound belief in the burgeoning Indian market, recognizing the incredible potential it holds for data-driven growth and transformation. India, with its vast population of over a billion and a half people, ranks among the world's most populous nations, second only to China.

Our engagement with the Indian market has proven immensely valuable to us in several ways. Firstly, it has pushed us to actively participate in India's transformative journey and leverage the opportunities it offers. Early on, Snapdeal, a prominent Indian e-commerce platform, adopted Aerospike, and this acted as a springboard. Talented engineers who had positive experiences with our technology spread across various large companies in India, contributing to our growing recognition in the region.

Secondly, our revenues have witnessed substantial growth in the Indian market and its extensions, such as Southeast Asia. We remain deeply committed to this growth trajectory, and our team in India continues to expand in tandem with our evolving business strategy. Looking ahead, we anticipate further expansion into new verticals. While we initially began in Ed tech, we have progressively ventured into e-commerce, telecom, payments, and banking, demonstrating our commitment to adapt and grow in response to market dynamics.