Power your Business with a Machine Learning Platform

The evolution of machine learning from a niche, emerging technology option to a mainstream technology offering has created a greater demand for ML solutions to solve business problems. In order to cater to this demand, organizations require a pool of data scientists who can develop and deploy ML models, more rapidly than ever before.

Data scientists, in order to architect better optimized ML models, require sizeable storage and compute resources at their disposal. This change in the scenario has led them to take their development environment from their laptops to cloud where they have access to on-demand compute and storage. To be more resourceful, they also need an ecosystem which provides them a controlled environment to develop their solutions.

Today, most companies who are on their journey of adopting machine learning in their core business are either using a machine learning platform easily available in the market or they have developed one of their own. A platform brings in efficiency, collaboration, maintainability, reusability and development discipline to the machine learning solutioning process.

Here is how a Machine Learning Platform can make your Data Science Team more Efficient

Data Access and Data Discovery

Data access and data discovery constitutes two of the critical reasons why large organizations prefer to build a platform of their own. Generally, large organization are characterised by multiple, disparate data sources on variety of platforms like Hadoop clusters, RDBMSs, No-SQL data stores, Blob/Object Stores and unstructured data files (photos, text).  Organizations follow data governance and access control processes to be compliant to security, regulatory and data audit requirements.

All these requirements are implemented at a machine learning platform level so that Data Scientists are confident about security and compliance requirements. This not only relieves them from the burden of ensuring they are compliant, but also protects an organization from inadvertent misuse of data.

Data Scientists can also greatly benefit from easy access to organization metadata having data lineage, business descriptions and data quality metrics through this platform to help them with data discovery and data analysis.

Additionally, the platform provides data visualization tools, data transformation/cleaning functions, visibility into resource utilization, access to application logs that allows a developer to build solutions without going through multiple tools/interfaces. As each organization is unique in how they deal with their data assets, a custom build solution is mandatory to make best use of the data in an easy, secure and reliable way.

On Demand Resource Provisioning

Even today, most data scientists begin their journeys on their laptop in-spite of its limitations.

It’s because of a simple reason that they are in complete control of their environment without depending on internal IT or the corporate network. On one hand, it provides great flexibility but on the other it forces them to make compromises due to limited compute and storage capabilities. All these limitations can be easily overcome by machine learning platform that provides them a way to provision on-demand compute (CPU/GPU/Memory) and storage, which they can increase/decrease.

This helps them create a development environment with pre-loaded commonly used machine learning libraries and allows them to make use of the enterprise data. It is a must that all code be version controlled and protected. Machine learning platform should not aim to replace how data scientists work, but, should complement their work so that they can me more efficient and better organised.

Enabling Hassle-free Development

Data Scientists should be able to quickly start with their work without an elaborate set-up process. They should be able to choose their IDEs such as RStudio, Zeppelin and Jupyter, and have access to libraries to get started. Those data scientists who prefer to work on their laptop should be able to do so using their favourite IDE (PyCharm, Eclipse and the likes) with the flexibility to sync-up their workspace with platform at a later stage.

To cater to this, the machine learning platform provides multiple flavours of ready-to-use containers that are pre-loaded with commonly used libraries. It also provides flexibility to users to use their own libraries or different version of the same library, depending on the use case.

It’s a given that the platform ensures fault tolerance and high availability to provide interruption free, reliable environment to work.  All of this is enabled through a single interface that provides a development-friendly environment to build complete machine learning workflow right from data ingestion, model training, model tuning to deployment.

Collaboration and Team Work

Data science is not an individual game anymore. A typical data science team includes data engineers, data scientists and software developers, each one of them having responsibilities carved to build a complete data science solution.

Team members should be able to work on and see each other’s artefacts based on their role in the team. They should be able to share their work for others to see. A machine learning platform is built to promote team work and collaboration with and across data science teams.

A common project set-up for the team that is controlled by the project administrator is required to reduce individual set-up overhead. While the platform promotes team work, it should also be easy for the project team to publish their work for other teams in the organization.

The platform promotes sharing of project artefacts like data, notebook, models etc. for others to learn or re-use. A well-crafted solution gallery goes a long way to adopt best practices and to build better optimised solutions.

Workflow for Machine Learning Pipelines

A data science solution is not made up of a monolithic code. More often than not, it is a multi-step process represented as directed acyclic graph. For example, a machine learning workflow may consist of data ingestion, model training, hyper parameter tuning and batch predictions steps having certain inter-dependencies on each other. Instead of writing code to invoke these modules, the platform has an in-built workflow engine which helps create such workflows which then can either be called from external applications or can be scheduled.

Productionise Models

Finally, for making predictions all models need to be deployed in production environment. The models need to be deployed in a run time environment which is quite similar to development environment. Many frameworks like TensorFlow, H2O etc. provide their own run time environment which should be used for the models that were built using those.

A machine learning platform makes it easy for data scientists to deploy these models in production without any dependencies on software engineers. The platform takes care of providing run time environment, latency, concurrency and fault tolerance. Data scientists can deploy their model for batch, interactive or streaming predictions. The platform provides all the plumbing required to expose the model as a service for interactive use or tying it to message queue for streaming predictions.

Implement Engineering Best Practices

A good platform is a critical requisite to implement engineering best practices. The machine learning platform brings engineering discipline to data science and helps implement version control, code review, testing and production roll-out processes. The platform provides High Availability and Fault Tolerance, thereby, guaranteeing the users that their work and data is always protected. It also ensures zero downtime for its customers even during upgrades and platform releases.

Monitoring and alerting is an integral part of a platform which promises that the support team can proactively monitor for performance, resource utilization, service failure, thereby, taking timely action.

A machine learning platform not only governs the infrastructure requirements of a data scientist, it also provides easy-to-use, efficient and integrated environment to build end-to-end data science solutions within the platform.

Therefore, a machine learning platform is an absolute necessity for any organization that intends to leverage data science in their business operations in an impactful way.

By Mr Ajay Sharma, Senior Director, Engineering at Walmart Labs