Data Platforms are crucial for enabling true digital transformation. They continuously evolve by adopting new, cutting-edge technologies, often via cloud virtualization. These rapid changes require a shift in focus to applications and developers and an increased pace in innovation and deployment. New digital user-machine agents, “Internet of Things” devices or Application Program Interfaces (APIs), that once were only applied to research initiatives are now required in order to remain competitive.
DevOps is a well-known approach to software development. It accelerates the lifecycle of a product by using automation. DevOps mainly focuses on continuous integration and continuous delivery. These two objectives are achieved by leveraging on-demand IT resources often referred as “infrastructure as code“.
If we look at today’s data-focused development, we might apply the same principles. Data analytics also requires highly optimized code builds and delivery. Here is where the concept DataOps emerges. But DataOps is not just “DevOps applied to data analytics”, which is a very common misconception.
DataOps requires cutting-edge innovation and integration of different ecosystems. The result of these two requirements is the Data Platforms, which are built upon on-demand IT resources (e.g. AWS S3, AWS EMR) and the integration of different APIs and applications (e.g. Spark, Jupyter, Tableau, TensorFlow).
Given the quantity and complexity of the IT operations required in a Data Platform, automation is crucial. Given the tools available for data analytics in that platform, using machine learning to analyse big data collected from IT tools and devices is a feasible methodology. Therefore, the main goal of AIOps is to spot and react to issues in real time using predictive analytics.
As big data and machine learning are the core components of AIOps, monitoring systems, logging scripts and alert schedules are necessary. Predictive algorithms can be pivotal in reducing MTTR (Mean Time To Repair) by automatically finding the exact cause of the problem and launching a proactive solution to it.
AIOps might seem like a futuristic, idealised idea of automating IT operations or even a vague collection of buzzwords. However, do yesterday’s tools seem apt to tackle today’s IT evolving ecosystem? It is a reality today that platforms that were static and predictable systems in the past are moving to software defined pools of resources that change dynamically. Data platforms are a necessary technological shift to enable AI and they require new tools based on AI to be feasible.
Moogsoft AIOps
Amazon EC2 Predictive Scaling
Azure VM resiliency
Amazon S3 Intelligent Tiering
