Autonomous Data Platforms: Converging AI, MLOps, and Cloud Engineering for Digital

Abstract

An Autonomous Data Platform integrates data engineering, MLOps, and AI services into a single platform. The convergence is important because many business problems that require data analysis, predictive modeling, and monitoring can be realized as autonomous data pipelines. Organizations are struggling to establish best practices and standards for autonomous data platforms. The research considers the data management principles, conceptual models, and architectural patterns of data platforms from a product perspective. Emphasis is placed on the convergence with MLOps and cloud engineering. The use cases and evaluations of autonomous data platforms enabled by the convergence are examined.

With the proliferation of data and new generation artificial intelligence (AI) technologies, organizations are exploring new roles, processes, and technology products to groom the data and build data models for predictive analytics and forecasting. The autonomy of data pipelines is becoming popular as organizations increasingly require learners and predictors to be created, deployed, and monitored automatically. The concept of an Autonomous Data Platform describes a converged product combining data engineering, MLOps, and AI services within an organization. Autonomous Data Platforms can be realized and realized as intelligent data pipelines that groom data and support organizations in various business functions such as customer relationship management and risk analytics systems.