Distributed Big Data Frameworks and High-Scale Service Design: Proven Engineering Patterns Across Cloud-Native Deployments
DOI:
https://doi.org/10.22399/ijcesen.5086Keywords:
Big Data, Cloud Platform Engineering, Distributed Systems, Event Streaming, Fault Tolerance, MicroservicesAbstract
Cloud platform engineering is a merger of distributed systems design, data-intensive computing, and reliability engineering to provide resilient services on a global scale. This article explores historical architectural patterns, including batch analytics, streaming backbones, globally replicated databases, and container orchestration, as viewed through well-documented deployment experiences and broadly reusable platform practices. The co-designing of the data plane and the control plane is related to reliability governance in terms of service-level objectives (SLOs) and error budgets, which are extrapolated to fintech, e-commerce, media, and IoT. Throughout these verticals, structural principles are recurrent: consistency trade-offs are intentional, long-lasting log abstractions, orchestrating via reconciliation, and replicating based on the workload. The article ends with a vendor-neutral composable blueprint that unifies these principles into a layered reference architecture that can be applied in deployments over cloud-native platforms.
References
[1] Juncal Alonso et al., "Understanding the challenges and novel architectural models of multi-cloud native applications: a systematic literature review," J. Cloud Comput., Springer Nature, vol. 12, no. 6, pp. 1-35, Jan. 12, 2023. https://link.springer.com/article/10.1186/s13677-022-00367-6
[2] Chiara Rucco, A. Longo, and M. Saad, "Enhancing Data Ingestion Efficiency in Cloud-Based Systems: A Design Pattern Approach," Data Sci. Eng., Springer Nature, Jul. 15, 2025. https://link.springer.com/article/10.1007/s41019-025-00300-2
[3] Mohammed Bergui, S. Najah, and N. S. Nikolov, "A survey on bandwidth-aware geo-distributed frameworks for big-data analytics," J. Big Data, Springer Nature, vol. 8, no. 40, Mar. 2021. https://link.springer.com/article/10.1186/s40537-021-00427-9
[4] Lukas Harzenetter et al., "Automated detection of design patterns in declarative deployment models," in Proc. 14th IEEE/ACM Int. Conf. Utility Cloud Comput. (UCC '21), Article no. 4, pp. 1-10, Dec. 17, 2021. https://dl.acm.org/doi/10.1145/3468737.3494085
[5] Gullapalli Sathar et al., "Cloud Computing for Big Data Analytics: Scalable Solutions for Data-Intensive Applications," J. Inf. Syst. Eng. Manage., vol. 10, no. 4, Jan. 16, 2025. https://jisem-journal.com/index.php/journal/article/view/10181/4685
[6] Sreedhar Pasupuleti et al., "Modernizing Legacy ETL Frameworks: A Scalable Approach to Cloud-Native Data Engineering," Sarcouncil J. Eng. Comput. Sci., vol. 4, no. 11, Nov. 19, 2025. https://sarcouncil.com/download-article/SJECS-561-2025-158-167.pdf
[7] Sailesh Oduri, "Engineering Resilience: Cloud-Native Design Patterns for Fault-Tolerant Systems," Commun. Appl. Nonlinear Anal., vol. 32, no. 2, Feb. 20, 2025. https://internationalpubls.com/index.php/cana/article/view/5958/3361
[8] Preetham Vemasani and S. Modi, "Building Resilient Distributed Systems: Fault-Tolerant Design Patterns for Stateful Workflows," Int. J. Comput. Eng. Technol. (IJCET), IAEME Publication, vol. 15, no. 3, May-Jun. 2024.
https://iaeme.com/MasterAdmin/Journal_uploads/IJCET/VOLUME_15_ISSUE_3/IJCET_15_03_016.pdf
[9] Alessandro Tundo et al., "Monitoring Probe Deployment Patterns for Cloud-Native Applications: Definition and Empirical Assessment," IEEE Trans. Serv. Comput., vol. 17, no. 4, Jul./Aug. 2024. https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=10380718
[10] Shubham Gupta, M. Sundararamaiah, and G. Geeta, "Leveraging Cloud-Native Data Engineering for Big Data Analytics," in Proc. 2025 3rd Int. Conf. Advancement Comput. Comput. Technol. (InCACCT), IEEE, Apr. 17-18, 2025. https://ieeexplore.ieee.org/document/11011292
[11] Saulo Ferreira et al., "Benchmarking Consistency Levels of Cloud-Distributed NoSQL Databases Using YCSB," IEEE Access, IEEE Xplore, Apr. 17, 2025. https://ieeexplore.ieee.org/document/10955378
[12] Robson A. Campêlo et al., "A brief survey on replica consistency in cloud environments," J. Internet Serv. Appl., Springer Nature, vol. 11, no. 1, Feb. 21, 2020. https://link.springer.com/article/10.1186/s13174-020-0122-y
[13] Gopalakrishnan Venkatasubbu, "A Cloud-Native Event-Driven Reactive Architecture for Real-Time Retail Transaction Processing," Int. J. Softw. Eng. (IJSE), CSC-OpenAccess Library, vol. 12, no. 5, pp. 78-89, Dec. 01, 2025. https://cscjournals.org/library/manuscriptinfo.php?mc=IJSE-195
[14] Shivareddy Devarapalli et al., "Cloud-Native LLMOps Meets DataOps: A Unified Framework for High-Volume Analytical Systems," in Proc. 2025 Int. Conf. Comput. Technol. Data Commun. (ICCTDC), IEEE, Jul. 04-05, 2025. https://ieeexplore.ieee.org/document/11158069
[15] Shubham Gupta, M. Sundararamaiah, and G. Geeta, "Leveraging Cloud-Native Data Engineering for Big Data Analytics," in Proc. 2025 3rd Int. Conf. Advancement Comput. Comput. Technol. (InCACCT), IEEE, Apr. 17-18, 2025. https://ieeexplore.ieee.org/document/11011292
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2025 International Journal of Computational and Experimental Science and Engineering
This work is licensed under a Creative Commons Attribution 4.0 International License.
