Metadata Driven Optimization of Distributed ETL Pipelines in Cloud Native Data Warehouses
DOI:
https://doi.org/10.22399/ijcesen.4742Keywords:
AI-driven optimization, metadata-driven ETL, cloud-native data warehouses, distributed data pipelines, data governanceAbstract
The data proliferation of distributed and cloud-native systems has altered organizational approach to Extract, Transform, and Load (ETL) pipelines to support analytics and decision-making. Conventional ETL frameworks based on fixed and script-based processes become scalable, maintainable, and real-time flexible in multi-cloud environments. Our review explores the development of metadata-based ETL systems. It could separate the logic and implementation of pipelines by externalizing transformation policies, lineage, and governance policies into structured metadata. These architectures enable dynamic reconfiguration, automation, and optimization of ETL processes. It helps to promote the agility and scalability in cloud-native data warehouses such as Snowflake, Databricks, and BigQuery. The paper summarises the existing developments in distributed ETL optimization, such as metadata-aware orchestration, AI-based performance tuning, and predictive workload balancing. It discusses metadata lifecycle management, lineage tracking, and security compliance governance mechanisms through such frameworks as Apache Atlas and Azure Purview. Moreover, we mention the new trends connected with Generative AI to modernize ETL, self-healing cognitive pipelines, and sustainable metadata management with the principles of green computing. This paper will show that metadata-based design can turn ETL systems into self-optimizing data pipes. Those were adaptive and autonomous, by collating the results of scholarly and industry research. The combination of artificial intelligence and metadata governance creates the basis of the future generation of intelligent, interoperable, and sustainable cloud data ecosystems.
References
[1] Kondylakis, Haridimos, Varvara Kalokyri, Stelios Sfakianakis, Kostas Marias, Manolis Tsiknakis, Ana Jimenez-Pastor, Eduardo Camacho-Ramos, et al. 2023. “Data Infrastructures for AI in Medical Imaging: A Report on the Experiences of Five EU Projects.” European Radiology Experimental 7 (1): 20.
[2] Foidl, Harald, Valentina Golendukhina, Rudolf Ramler, and Michael Felderer. 2024. “Data Pipeline Quality: Influencing Factors, Root Causes of Data-Related Issues, and Processing Problem Areas for Developers.” The Journal of Systems and Software 207 (111855): 111855.
[3] Debauche, Olivier, Saïd Mahmoudi, Pierre Manneback, and Frédéric Lebeau. 2022. “Cloud and Distributed Architectures for Data Management in Agriculture 4.0 : Review and Future Trends.” Journal of King Saud University - Computer and Information Sciences 34 (9): 7494–7514.
[4] Isah, Haruna, Tariq Abughofa, Sazia Mahfuz, Dharmitha Ajerla, Farhana Zulkernine, and Shahzad Khan. 2019. “A Survey of Distributed Data Stream Processing Frameworks.” IEEE Access: Practical Innovations, Open Solutions 7: 154300–316.
[5] Malti, Arslan Nedhir, Mourad Hakem, and Badr Benmammar. 2024. “A New Hybrid Multi-Objective Optimization Algorithm for Task Scheduling in Cloud Systems.” Cluster Computing 27 (3): 2525–48.
[6] Wen, Lei, Hengshun Qian, and Wenpan Liu. 2022. “Research on Intelligent Cloud Native Architecture and Key Technologies Based on DevOps Concept.” Procedia Computer Science 208: 590–97.
[7] Dean, J., & Ghemawat, S. (2008). MapReduce: Simplified data processing on large clusters. Communications of the ACM, 51(1), 107–113.
[8] Abadi, D. J. (2018). Data management in the cloud: Limitations and opportunities. IEEE Data Engineering Bulletin, 41(1), 3–9.
[9] Amorim, Ricardo Carvalho, João Aguiar Castro, João Rocha da Silva, and Cristina Ribeiro. 2017. “A Comparison of Research Data Management Platforms: Architecture, Flexible Metadata and Interoperability.” Universal Access in the Information Society 16 (4): 851–62.
[10] Oyighan, Diseiye, Ejiro Sandra Ukubeyinje, Boma T. David -West, and Bolaji David Oladokun. 2024. “The Role of AI in Transforming Metadata Management: Insights on Challenges, Opportunities, and Emerging Trends.” Asian Journal of Information Science and Technology 14 (2): 20–26.
[11] Rozony, F. Z., Aktar M. N. A., M. Ashrafuzzaman, and A. Islam. 2024. “A Systematic Review of Big Data Integration Challenges and Solutions for Heterogeneous Data Sources.” Academic Journal on Business Administration, Innovation & Sustainability 4 (04): 1–18.
[12] Alonso, Juncal, Leire Orue-Echevarria, Valentina Casola, Ana Isabel Torre, Maider Huarte, Eneko Osaba, and Jesus L. Lobo. 2023. “Understanding the Challenges and Novel Architectural Models of Multi-Cloud Native Applications – a Systematic Literature Review.” Journal of Cloud Computing Advances Systems and Applications 12 (1).
[13] Alonso, Juncal, Leire Orue-Echevarria, Valentina Casola, Ana Isabel Torre, Maider Huarte, Eneko Osaba, and Jesus L. Lobo. 2023. “Understanding the Challenges and Novel Architectural Models of Multi-Cloud Native Applications – a Systematic Literature Review.” Journal of Cloud Computing Advances Systems and Applications 12 (1).
[14] Munson, Jacob, Thomas Cuezze, Siddat Nesar, and Dominique Zosso. 2025. “A Review of Large Language Models and the Recommendation Task.” Discover Artificial Intelligence 5 (1).
[15] Singh, Gaurav, and Adarsh Maurya. 2025. “The Role of Metadata in Data Curation for Enhancing Discoverability in Large Datasets.” International Journal of Web of Multidisciplinary Studies 2 (1): 31–37.
[16] Balabanov, O. S., and Institute of Software Systems NAS of Ukraine. 2019. “Big Data Analytics: Principles, Trends and Tasks (a Survey).” Problemy Programmirovaniya. Problems in Programming, no. 2: 047–068.
[17] Xu, Xi, Jianqiang Li, Zhichao Zhu, Linna Zhao, Huina Wang, Changwei Song, Yining Chen, Qing Zhao, Jijiang Yang, and Yan Pei. 2024. “A Comprehensive Review on Synergy of Multi-Modal Data and AI Technologies in Medical Diagnosis.” Bioengineering (Basel, Switzerland) 11 (3): 219.
[18] Eweje, Adeoluwa, and Francis Ohaegbu. 2021. “Advances in Modern Data Stack Architectures for Scalable Data Integration and Business Intelligence.” International Journal of Multidisciplinary Research and Growth Evaluation 2 (5): 538–50.
[19] Jahanshad, Neda, Petra Lenzini, and Janine Bijsterbosch. 2024. “Current Best Practices and Future Opportunities for Reproducible Findings Using Large-Scale Neuroimaging in Psychiatry.” Neuropsychopharmacology: Official Publication of the American College of Neuropsychopharmacology 50 (1): 37–51.
[20] Chanda, D. (2024). Automated ETL Pipelines for Modern Data Warehousing: Architectures, Challenges, and Emerging Solutions. The Eastasouth Journal of Information System and Computer Science, 1(03), 209–212.
[21] Eweje, Adeoluwa, and Francis Ohaegbu. 2021. “Advances in Modern Data Stack Architectures for Scalable Data Integration and Business Intelligence.” International Journal of Multidisciplinary Research and Growth Evaluation 2 (5): 538–50.
[22] Munappy, Aiswarya Raj, Jan Bosch, and Helena Homström Olsson. 2020. “Data Pipeline Management in Practice: Challenges and Opportunities.” In Product-Focused Software Process Improvement, 168–84.
[23] Ragazou, Konstantina, Ioannis Passas, Alexandros Garefalakis, and Constantin Zopounidis. 2023. “Business Intelligence Model Empowering SMEs to Make Better Decisions and Enhance Their Competitive Advantage.” Discover Analytics 1 (1).
[24] Mantri, A. (2023). Advanced ML Techniques for Optimizing ETL Workflows with Apache Spark and Snowflake. Journal of Artificial Intelligence & Cloud Computing, 2(3), 339– 347.
[25] National University Bangladesh, Gazipur, Bangladesh, Hosne Ara Mohna, Tonmoy Barua, Manager, Facilities and Administration, MetLife, Bangladesh, Mohammad Mohiuddin, Data Engineer, NCC Bank PLC, Dhaka, Bangladesh, Md Mostafizur Rahman, and Assistant Manager, Teletalk Bangladesh Ltd, Dhaka, Bangladesh. 2022. “Ai-Ready Data Engineering Pipelines: A Review of Medallion Architecture and Cloud-Based Integration Models.” American Journal of Scholarly Research and Innovation 01 (01): 319–50.
[26] Vishwanadham Mandala. 2018. “Meta-Orchestrated Data Engineering: A Cloud-Native Framework for Cross-Platform Semantic Integration.” Global Research and Development Journals 3 (12).
[27] Seenivasan, D. 2024. “AI Driven Enhancement of ETL Workflows for Scalable and Efficient Cloud Data Engineering.” International Journal of Engineering and Computer Science 13 (06): 10–18535.
[28] Bhatlawande, S., Rajandekar, R., & Shilaskar, S. (2024). Implementing Middleware Architecture for Automated Data Pipeline over Cloud Technologies. IEEE 13th International Conference on Communication Systems and Network Technologies (CSNT), 13(1), 506–513.
[29] Machireddy, Jeshwanth Reddy. 2023. “Data Quality Management and Performance Optimization for Enterprise-Scale ETL Pipelines in Modern Analytical Ecosystems.” Journal of Data Science, Predictive Analytics, and Big Data Applications 8 (7): 1–26.
[30] Shivaramakrishna, D., and M. Nagaratna. 2023. “A Novel Hybrid Cryptographic Framework for Secure Data Storage in Cloud Computing: Integrating AES-OTP and RSA with Adaptive Key Management and Time-Limited Access Control.” Alexandria Engineering Journal 84 (December): 275–84.
[31] Vattumilli, P. K. (2024). Metadata-Driven ETL Pipelines: A Framework for Scalable Data Integration Architecture. International Journal of Scientific Research in Computer Science, Engineering and Information Technology, 6(6), Article 61224.
[32] Ananthakrishnan Kunju, A. (2024). Autonomous GenAI Agents for Legacy-to-Cloud ETL Modernization. Journal of Artificial Intelligence General Science (JAIGS), 1(1), 55–72.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2025 International Journal of Computational and Experimental Science and Engineering
This work is licensed under a Creative Commons Attribution 4.0 International License.
