Agility has been a mantra in software development for many years, primarily practiced as a project management methodology to navigate project with loose or evolving requirements. Now, being agile is applied to other activities beyond software development, including research and development. Collaboration and iteration is especially important in the data analytics practice, including incorporating shorter feedback loops between the business teams and the data analysis teams allows for faster alignment and adaptation to business requirements.
Agility is a critical skill in data analytics. Data analytics aims to either optimise an existing process, or examine a totally new way to accomplish a business goal, and in doing so, probably disrupting existing processes. how effective the different models will be. Different levels of performance of the models may lead to improvements and iterations of the business processes. For instance, in a tower air traffic control environment, a real-time data analytics process may provide an a likelihood of particular traffic using a runway exit. This information may then be used to optimise the time between arrival and departure operations. In this instance, operating procedures do not have to change for the algorithm performance to have an impact on effectiveness. Considering final performance cannot be known before different methodologies are tested, it is important to maintain business and technical agile adaptation to incorporate key analytical insights.
Agile methodologies are not an abstract concept. The organisational structure of working on data analytics work need to equip themselves with a set of tools and methods to ensure collaboration. There needs to be “bridges” between the operationally-focus experts and the data analytics team members. Often times, this bridge is the Product Owner, as the critical figure of the agile methodology. The Product Owner has to understand and capture the business requirements and then be able to communicate these requirements to operational experts in a way so they can ideate new procedures and methods that perform the function in a much more effective way; furthermore utilising insights from the analytics team. This Product Owner simultaneously challenges operational experts, to imagine a disruption of business process, as well as the analytics team, in generating methods and tools capable of providing the right predictive analytics is a challenging function.
Agile is also an effective practice in data management and data processing functions. Even before business experts have defined the operational challenges, the data management team needs to be capable of acquiring, extracting, transforming and loading the data in the infrastructure. Addressing data management challenges, including data provenance and data quality, will surely be a challenging task when data protection requirements are essential. Agility can also help in this context, by providing a methodology in which to approach the challenge. The agile methodology encourages first understanding requirements and limitations, ensuring the different data entities that are relevant are involved as stakeholders, and finally collaborating through iterative cycles with operational experts.
Incorporating the agile methodology pays off at every step of the process. Although the waterfall project management is a well established methodology in most organisations, there are inherent difficulties to break the barrier towards a new project methodology. The waterfall approach risks further development without frequent testing and lacks opportunity to incorporate key data analytics throughout the lifespan. Communication and collaboration through the right set of tools is also important, and further ensures the bridge between business, operations, and analytics. With this the implementation of an agile practice is strengthened and becomes the first step to be successful in data analytics.
