What's Happening?
NAB is undertaking a significant modernization of its data pipelines for the Ada platform, aiming to standardize and streamline operations. The bank is implementing Spark Declarative Pipelines (SDP) across over 1600 data pipelines, as revealed by Dheeraj
Puli, Head of Data Reliability Engineering, at the Databricks Data+AI Summit in San Francisco. This initiative is part of NAB's ongoing data journey, which began with the introduction of Ada in 2022. The platform, which utilizes Databricks on AWS, replaced several older data systems. The modernization effort focuses on transforming data for the silver and gold layers of the platform, moving away from custom-built Spark pipelines to a more standardized SDP approach. This shift is expected to reduce complexity and operational costs, with anticipated savings of 15% once fully implemented.
Why It's Important?
The modernization of NAB's data pipelines is crucial for enhancing the bank's operational efficiency and reducing costs. By adopting a standardized approach with SDP, NAB aims to streamline its data processing, which is vital for supporting advanced analytics and machine learning models. This move not only positions NAB as a leader in data management within the financial sector but also sets a precedent for other institutions looking to optimize their data operations. The anticipated cost savings and improved data reliability could lead to better financial performance and customer service, providing NAB with a competitive edge in the banking industry.
What's Next?
NAB plans to extend the use of SDP to the gold layer of its data platform, with the long-term goal of achieving a fully declarative, end-to-end streaming architecture. This would make NAB the first bank to run all its data pipelines on SDP, setting a new industry standard. The bank is also exploring new features like 'rewind and replay' to enhance data recovery capabilities. As NAB continues to modernize its data infrastructure, it is likely to influence other financial institutions to adopt similar strategies, potentially leading to widespread changes in data management practices across the industry.
