Optimization Strategies for Cloud-Native Lakehouse Platforms Supporting Large-Scale Data Processing and Machine Learning Workloads
DOI:
https://doi.org/10.22399/ijcesen.5560Keywords:
Cloud-native lakehouse, Data processing, Machine learning, Query optimization, Resource elasticity, Cost optimisationAbstract
Cloud-native lakehouses are commonly used to handle large scale data processing and machine learning workloads. Nonetheless, increasing data size, varying computation needs, complexities of queries, costs associated with infrastructure and issues related to governance pose various optimization challenges. This paper seeks to understand how optimization can be done in cloud-native lakehouses in order to improve processing performance, scalability, machine learning efficiency, resource utilization, efficiency and security. The qualitative research approach was employed through secondary data from academic journals, technical articles, conference proceedings, reports and reliable technical sources. Inductive approach was used to detect various optimization trends from the secondary data. It is evident that separation of compute-storage, elasticity, workload scheduling, partitioning, columnar storage, caching, compression and optimization of queries result in processing efficiency. Machine learning workloads get benefits from resource elasticity, data locality, workload-based scheduling and optimization of GPUs/CPU. The results further demonstrate that automation of provision and dynamic scaling can lead to minimisation of underutilisation of infrastructure and operational efficiency improvements. In addition, metadata management, access control, encryption, audit trails, and data lineage add up to governance and security aspects in a distributed lakehouse architecture. From the study it is clear that optimization of systems should not only consider speed of execution of queries or infrastructure costs. Rather, performance, scalability, resource utilization, security, governance, and complexity of operations should all be taken into account. Thus, a workload-aware approach to system optimization can offer more sustainable architecture for large scale analytics and machine learning.
References
1. Adams, F. (2023). Step-by-step breakdown of real-time ML operational deployment pipeline. Medium. Retrieved at https://medium.com/@FrankAdams7/step-by-step-breakdown-of-real-time-ml-operational-deployment-pipeline-848f290eb5d9
2. Alibaba Cloud. (2022, January 14). Data warehouse and lake house: Continuous evolution of the cloud-native big data platform. Alibaba Cloud
3. Armbrust, M., Das, T., Sun, L., Yavuz, B., Zhu, S., Murthy, M., ... & Zaharia, M. (2020). Delta lake: high-performance ACID table storage over cloud object stores. Proceedings of the VLDB Endowment, 13(12), 3411-3424. https://www.eecs.umich.edu/courses/cse584/static_files/papers/p975-armbrust.pdf DOI: https://doi.org/10.14778/3415478.3415560
4. Berlato, S., Carbone, R., Lee, A. J., & Ranise, S. (2020, October). Exploring architectures for cryptographic access control enforcement in the cloud for fun and optimization. In Proceedings of the 15th ACM Asia Conference on Computer and Communications Security (pp. 208-221). https://dl.acm.org/doi/abs/10.1145/3320269.3384767 DOI: https://doi.org/10.1145/3320269.3384767
5. Cheong, H. I., Lyons, A., Houghton, R., & Majumdar, A. (2023). Secondary qualitative research methodology using online data within the context of social sciences. International Journal of Qualitative Methods, 22, 16094069231180160. https://journals.sagepub.com/doi/abs/10.1177/16094069231180160 DOI: https://doi.org/10.1177/16094069231180160
6. Gao, W., Hu, Q., Ye, Z., Sun, P., Wang, X., Luo, Y., ... & Wen, Y. (2022). Deep learning workload scheduling in gpu datacenters: Taxonomy, challenges and vision. arXiv preprint arXiv:2205.11913. https://arxiv.org/abs/2205.11913
7. Hambardzumyan, S., Tuli, A., Ghukasyan, L., Rahman, F., Topchyan, H., Isayan, D., ... & Buniatyan, D. (2022). Deep lake: A lakehouse for deep learning. arXiv preprint arXiv:2209.10785. https://arxiv.org/abs/2209.10785
8. Hambardzumyan, S., Tuli, A., Ghukasyan, L., Rahman, F., Topchyan, H., Isayan, D., ... & Buniatyan, D. (2022). Deep lake: A lakehouse for deep learning. arXiv preprint arXiv:2209.10785. https://arxiv.org/abs/2209.10785
9. Han, Y. S. (2021). High-Performance Data Management Architectures for Scalable Machine Learning Pipelines in Cloud Ecosystems. American International Journal of Computer Science and Technology, 3(2), 11-20. http://aijcst.org/index.php/aijcst/article/view/29 DOI: https://doi.org/10.63282/3117-5481/AIJCST-V3I2P102
10. Kianian, B., Kurdve, M., & Andersson, C. (2019). Comparing life cycle costing and performance part costing in assessing acquisition and operational cost of new manufacturing technologies. Procedia Cirp, 80, 428-433. https://www.sciencedirect.com/science/article/pii/S2212827119300277 DOI: https://doi.org/10.1016/j.procir.2019.01.025
11. Lehmann, C., Goren Huber, L., Horisberger, T., Scheiba, G., Sima, A. C., & Stockinger, K. (2020). Big Data architecture for intelligent maintenance: a focus on query processing and machine learning algorithms. Journal of Big Data, 7(1), 61. https://link.springer.com/article/10.1186/s40537-020-00340-7 DOI: https://doi.org/10.1186/s40537-020-00340-7
12. Li, S., & Avestimehr, S. (2020). Coded computing: Mitigating fundamental bottlenecks in large-scale distributed computing and machine learning. Foundations and Trends in Communications and Information Theory, 17(1), 1-148. https://www.emerald.com/ftcit/article/17/1/1/1332643 DOI: https://doi.org/10.1561/0100000103
13. Margherita, E. G., & Braccini, A. M. (2021). Managing industry 4.0 automation for fair ethical business development: A single case study. Technological Forecasting and Social Change, 172, 121048. https://www.sciencedirect.com/science/article/pii/S0040162521004807 DOI: https://doi.org/10.1016/j.techfore.2021.121048
14. Moreno-Vozmediano, R., Montero, R. S., Huedo, E., & Llorente, I. M. (2019). Efficient resource provisioning for elastic cloud services based on machine learning techniques. Journal of Cloud Computing, 8(1), 1-18. https://link.springer.com/article/10.1186/s13677-019-0128-9 DOI: https://doi.org/10.1186/s13677-019-0128-9
15. Roberts, K., Dowell, A., & Nie, J. B. (2019). Attempting rigour and replicability in thematic analysis of qualitative research data; a case study of codebook development. BMC medical research methodology, 19(1), 66. https://link.springer.com/chapter/10.1007/978-3-030-22113-3_9 DOI: https://doi.org/10.1186/s12874-019-0707-y
16. Rzadca, K., Findeisen, P., Swiderski, J., Zych, P., Broniek, P., Kusmierek, J., ... & Wilkes, J. (2020, April). Autopilot: workload autoscaling at google. In proceedings of the fifteenth european conference on computer systems (pp. 1-16). https://dl.acm.org/doi/abs/10.1145/3342195.3387524 DOI: https://doi.org/10.1145/3342195.3387524
17. Sebaa, A., & Tari, A. (2019). Query optimization in cloud environments: Challenges, taxonomy, and techniques. The Journal of Supercomputing, 75(6), 5420–5450. https://doi.org/10.1007/s11227-019-02806-9 DOI: https://doi.org/10.1007/s11227-019-02806-9
18. Shah, M., & Gogineni, A. (2022). Distributed Query Optimization for Petabyte-Scale Databases. Int. J. Recent Innov. Trends Comput. Commun, 10(10), 223-231. https://openreview.net/pdf?id=NkGdG15swy
19. Wang, H., Wu, Y., Min, G., Xu, J., & Tang, P. (2019). Data-driven dynamic resource scheduling for network slicing: A deep reinforcement learning approach. Information Sciences, 498, 106-116. https://www.sciencedirect.com/science/article/pii/S0020025519303986 DOI: https://doi.org/10.1016/j.ins.2019.05.012
20. Weng, Q., Xiao, W., Yu, Y., Wang, W., Wang, C., He, J., ... & Ding, Y. (2022). {MLaaS} in the wild: Workload analysis and scheduling in {Large-Scale} heterogeneous {GPU} clusters. In 19th USENIX symposium on networked systems design and implementation (NSDI 22) (pp. 945-960). https://www.usenix.org/conference/nsdi22/presentation/weng
21. Yang, P., Xiong, N., & Ren, J. (2020). Data security and privacy protection for cloud storage: A survey. Ieee Access, 8, 131723-131740. https://ieeexplore.ieee.org/abstract/document/9142202/ DOI: https://doi.org/10.1109/ACCESS.2020.3009876
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2025 International Journal of Computational and Experimental Science and Engineering
This work is licensed under a Creative Commons Attribution 4.0 International License.
