Optimizing Financial Data Reconciliation Pipelines Using Serverless AWS Architectures
DOI:
https://doi.org/10.22399/ijcesen.5543Keywords:
Financial Data Reconciliation, Serverless Computing, AWS Architecture, Data Pipelines, Cloud OptimizationAbstract
The financial data reconciliation process establishes a fundamental procedure that ensures that different data sources maintain their uniformity. The traditional reconciliation system uses batch pipelines as its main operational system which cannot support process expansion. The financial data processing needs have reached a point where better processing systems and faster data processing speeds become necessary. The study investigates how serverless AWS architectures improve financial data reconciliation processes according to its current state. Serverless computing enables businesses to scale their operations while maintaining cost efficiency through its event-driven system which uses Function-as-a-Service technology. The present paper evaluates AWS Lambda S3 Kinesis and Step Functions as the main architectural components which create reconciliation workflows through their interconnections. The optimization methods improve performance while reducing costs and enhancing system reliability. The system uses three techniques which include parallel processing memory tuning and event oriented orchestration. The evaluation process examined cold start latency stateless execution security concern and vendor lock-in issues. The future pathways combine AI-driven pipelines with edge computing technology and multi-cloud systems. The new advancements will improve system resilience while increasing processing efficiency and automation capabilities. The serverless system offers financial reconciliation systems an efficient route to modernization. The system will enable financial institutions to process data in real time while they manage their evolving operational requirements.
References
[1] P. Vassiliadis, “A survey of extract–transform–load technology,” International Journal of Data Warehousing and Mining, vol. 5, no. 3, pp. 1–27, 2009. DOI: https://doi.org/10.4018/jdwm.2009070101
[2] M. Stonebraker et al., “The end of an architectural era (it’s time for a complete rewrite),” Proceedings of the VLDB Endowment, vol. 1, no. 2, pp. 1150–1160, 2008.
[3] E. Jonas et al., “Cloud programming simplified: A Berkeley view on serverless computing,” arXiv preprint arXiv:1902.03383, 2019.
[4] G. Adzic and R. Chatley, “Serverless computing: Economic and architectural impact,” in Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering, pp. 884–889, 2017. DOI: https://doi.org/10.1145/3106237.3117767
[5] I. Baldini et al., “Serverless computing: Current trends and open problems,” in Research Advances in Cloud Computing, Springer, 2017, pp. 1–20.
[6] A. G. Labrinidis and H. V. Jagadish, “Challenges and opportunities with big data,” Proceedings of the VLDB Endowment, vol. 5, no. 12, pp. 2032–2033, 2012. DOI: https://doi.org/10.14778/2367502.2367572
[7] Poojara, S., Dehury, C.K., Jakovits, P., Srirama, S.N. (2023). Serverless Data Pipelines for IoT Data Analytics: A Cloud Vendors Perspective and Solutions. In: Thakkar, H.K., Dehury, C.K., Sahoo, P.K., Veeravalli, B. (eds) Predictive Analytics in Cloud, Fog, and Edge Computing. Springer, Cham. https://doi.org/10.1007/978-3-031-18034-7_7 DOI: https://doi.org/10.1007/978-3-031-18034-7_7
[8] R. A. P. Rajan, "Serverless Architecture - A Revolution in Cloud Computing," 2018 Tenth International Conference on Advanced Computing (ICoAC), Chennai, India, 2018, pp. 88-93 DOI: https://doi.org/10.1109/ICoAC44903.2018.8939081
[9] P. Castro, V. Ishakian, V. Muthusamy, and A. Slominski, “The rise of serverless computing,” Communications of the ACM, vol. 62, no. 12, pp. 44–54, 2019. DOI: https://doi.org/10.1145/3368454
[10] E. van Eyk et al., “Serverless is more: From PaaS to present cloud computing,” IEEE Internet Computing, vol. 22, no. 5, pp. 8–17, 2018. DOI: https://doi.org/10.1109/MIC.2018.053681358
[11] J. Spillner, S. Mateos, and D. Montero, “Faaster, better, cheaper: The prospect of serverless scientific computing and HPC,” in 2017 IEEE International Conference on Cloud Computing Technology and Science (CloudCom), pp. 154–161. DOI: https://doi.org/10.1007/978-3-319-73353-1_11
[12] Tuli, S., Gill, S. S., Xu, M., Garraghan, P., Bahsoon, R., Dustdar, S., Sakellariou, R., Rana, O., Buyya, R., Casale, G., & Jennings, N. R. (2022). HUNTER: AI based holistic resource management for sustainable cloud computing. Journal of Systems and Software, 184, 111124. https://doi.org/10.1016/j.jss.2021.111124 DOI: https://doi.org/10.1016/j.jss.2021.111124
[13] Gupta S, Singh N. Toward intelligent resource management in dynamic Fog Computing-based Internet of Things environment with Deep Reinforcement Learning: A survey. Int J Commun Syst. 2023;36(4):e5411. DOI: https://doi.org/10.1002/dac.5411
[14] T. Akidau et al., “The dataflow model: A practical approach to balancing correctness, latency, and cost in massive-scale, unbounded, out-of-order data processing,” Proceedings of the VLDB Endowment, vol. 8, no. 12, pp. 1792–1803, 2015. DOI: https://doi.org/10.14778/2824032.2824076
[15] J. Dean and S. Ghemawat, “MapReduce: Simplified data processing on large clusters,” Communications of the ACM, vol. 51, no. 1, pp. 107–113, 2008. DOI: https://doi.org/10.1145/1327452.1327492
[16] R. Cordingly, W. Shu and W. J. Lloyd, "Predicting Performance and Cost of Serverless Computing Functions with SAAF," 2020 IEEE Intl Conf on Dependable, Autonomic and Secure Computing, Intl Conf on Pervasive Intelligence and Computing, Intl Conf on Cloud and Big Data Computing, Intl Conf on Cyber Science and Technology Congress (DASC/PiCom/CBDCom/CyberSciTech), Calgary, AB, Canada, 2020, pp. 640-649 DOI: https://doi.org/10.1109/DASC-PICom-CBDCom-CyberSciTech49142.2020.00111
[17] G. McGrath and P. R. Brenner, "Serverless Computing: Design, Implementation, and Performance," 2017 IEEE 37th International Conference on Distributed Computing Systems Workshops (ICDCSW), Atlanta, GA, USA, 2017, pp. 405-410
[18] A. Eivy, “Be wary of the economics of ‘serverless’ cloud computing,” IEEE Cloud Computing, vol. 4, no. 2, pp. 6–12, 2017. DOI: https://doi.org/10.1109/MCC.2017.32
[19] Javed, H., Toosi, A.N., Aslanpour, M.S. (2022). Serverless Platforms on the Edge: A Performance Analysis. In: Buyya, R., Garg, L., Fortino, G., Misra, S. (eds) New Frontiers in Cloud Computing and Internet of Things. Internet of Things. Springer, Cham. DOI: https://doi.org/10.1007/978-3-031-05528-7_6
[20] E. Jonas, Q. Pu, S. Venkataraman, I. Stoica, and B. Recht, “Occupy the cloud: Distributed computing for the 99%,” in Proceedings of the 2017 Symposium on Cloud Computing, pp. 445–451. DOI: https://doi.org/10.1145/3127479.3128601
[21] Chetabi, F.A., Ashtiani, M. & Saeedizade, E. A Package-Aware Approach for Function Scheduling in Serverless Computing Environments. J Grid Computing 21, 23 (2023). DOI: https://doi.org/10.1007/s10723-023-09657-y
[22] L. Wang, M. Kunze, J. Tao, and G. von Laszewski, “Towards building a cloud for scientific applications,” Advances in Engineering Software, vol. 42, no. 9, pp. 714–722, 2011. DOI: https://doi.org/10.1016/j.advengsoft.2011.05.007
[23] B. Varghese and R. Buyya, “Next generation cloud computing: New trends and research directions,” Future Generation Computer Systems, vol. 79, pp. 849–861, 2018. DOI: https://doi.org/10.1016/j.future.2017.09.020
[24] P. Jamshidi, A. Ahmad, and C. Pahl, “Cloud migration research: A systematic review,” IEEE Transactions on Cloud Computing, vol. 1, no. 2, pp. 142–157, 2013. DOI: https://doi.org/10.1109/TCC.2013.10
[25] M. Villamizar et al., “Evaluating the monolithic and the microservice architecture pattern to deploy web applications in the cloud,” in 2015 10th Computing Colombian Conference, pp. 583–590. DOI: https://doi.org/10.1109/ColumbianCC.2015.7333476
[26] J. Spillner, “Serverless computing and cloud functions,” IEEE Cloud Computing, vol. 4, no. 1, pp. 6–10, 2017. DOI: https://doi.org/10.1145/3154847.3154854
[27] G. McGrath and P. R. Brenner, “Serverless computing: Design, implementation, and performance,” in 2017 IEEE 37th International Conference on Distributed Computing Systems Workshops, pp. 405–410. DOI: https://doi.org/10.1109/ICDCSW.2017.36
[28] M. Shahrad, J. Balkind, and D. Wentzlaff, “Architectural implications of function-as-a-service computing,” in Proceedings of the 52nd Annual IEEE/ACM International Symposium on Microarchitecture, pp. 1063–1075, 2019. DOI: https://doi.org/10.1145/3352460.3358296
[29] A. Baldini et al., “Serverless computing: Current trends and open problems,” in Research Advances in Cloud Computing, Springer, 2017.
[30] R. Buyya, C. S. Yeo, and S. Venugopal, “Market-oriented cloud computing: Vision, hype, and reality,” Future Generation Computer Systems, vol. 25, no. 6, pp. 599–616, 2009. DOI: https://doi.org/10.1016/j.future.2008.12.001
[31] J. Huang, M. Wang, Y. Wu, Y. Chen and X. Shen, "Distributed Offloading in Overlapping Areas of Mobile-Edge Computing for Internet of Things," in IEEE Internet of Things Journal, vol. 9, no. 15, pp. 13837-13847. DOI: https://doi.org/10.1109/JIOT.2022.3143539
[32] T. Lynn, P. Rosati, A. Lejeune, and V. Emeakaroha, “A preliminary review of enterprise serverless cloud computing (function-as-a-service) platforms,” in 2017 IEEE International Conference on Cloud Computing Technology and Science, pp. 162–169. DOI: https://doi.org/10.1109/CloudCom.2017.15
[33] V. Velepucha and P. Flores, "A Survey on Microservices Architecture: Principles, Patterns and Migration Challenges," in IEEE Access, vol. 11, pp. 88339-88358, 2023. DOI: https://doi.org/10.1109/ACCESS.2023.3305687
[34] I. Baldini, P. Castro, K. S. Chang, P. Cheng, S. Fink, V. Ishakian, N. Mitchell, V. Muthusamy, R. Rabbah, A. Slominski, and P. Suter, “Serverless computing: Current trends and open problems,” in Research Advances in Cloud Computing, Springer, 2017, pp. 1–20. DOI: https://doi.org/10.1007/978-981-10-5026-8_1
[35] R. Buyya, V. Srirama, and S. N. Srirama, “Fog and edge computing: Principles and paradigms,” Wiley, 2019. DOI: https://doi.org/10.1002/9781119525080
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2024 International Journal of Computational and Experimental Science and Engineering
This work is licensed under a Creative Commons Attribution 4.0 International License.
