Distributed Reinforcement Learning Based Efficient Radio Resource Allocation For Heterogeneous IoT Wireless Network

Authors

  • Abhishek Kumar Verma NA
  • Vinay Kumar Singh
  • M. R. Khan

DOI:

https://doi.org/10.22399/ijcesen.2852

Keywords:

Internet of Things (IoT), Power Allocation, Heterogeneous Network , Reinforcement Learning, Quality of Service (QoS)

Abstract

The heterogeneous cellular network has emerged as a critical infrastructure in supporting diverse Internet of Things (IoT) based services. As next-generation technologies continue to evolve, they will provide a unified framework capable of seamlessly connecting huge number of IoT devices. This integration will support the complex requirements of modern IoT-driven business processes. To achieve maximum capacity for IoT applications there is a need to integrate multiple wireless network technologies which makes the environment dense. However, this network densification also raises interference levels from neighbouring devices which can negatively impact the Quality of Service (QoS). In a dense IoT environment with limited radio resources there is need of efficient radio resource allocation strategy to maintain QoS across various connected devices. This work presents a distributed reinforcement learning based power allocation algorithm for heterogeneous IoT networks. We also propose a reward function that accounts for the QoS needs of multiple IoT users, promotes fairness, and ensures reliable connectivity. We carry out complexity analysis and convergence analysis of our proposed algorithm and also we explore different learning frameworks to evaluate the performance of the algorithm. Results demonstrated that the proposed method is effective in improving network capacity and other performance measures in dense heterogeneous environment.

References

[1] Zhou, I., Li, X., Chen, Y., & Zhang, H. (2021). Internet of Things 2.0: Concepts, applications, and future directions. IEEE Access, 9, 70961–71012. https://doi.org/10.1109/ACCESS.2021.3078549 DOI: https://doi.org/10.1109/ACCESS.2021.3078549

[2] Palattella, M. R., Dohler, M., Grieco, L. A., Boggia, G., & Mendes, P. (2016). Internet of Things in the 5G era: Enablers, architecture, and business models. IEEE Journal on Selected Areas in Communications, 34(3), 510–527. https://doi.org/10.1109/JSAC.2016.2525418 DOI: https://doi.org/10.1109/JSAC.2016.2525418

[3] Peng, M., Zhang, K., Jiang, J., Wang, J., & Wang, W. (2015). Energy-efficient resource assignment and power allocation in heterogeneous cloud radio access networks. IEEE Transactions on Vehicular Technology, 64(11), 5275–5287. https://doi.org/10.1109/TVT.2014.2379922 DOI: https://doi.org/10.1109/TVT.2014.2379922

[4] Zhang, H., Huang, S., Jiang, C., Long, K., Leung, V. C. M., & Poor, H. V. (2017). Energy-efficient user association and power allocation in millimeter-wave-based ultra-dense networks with energy harvesting base stations. IEEE Journal on Selected Areas in Communications, 35(9), 1936–1947. https://doi.org/10.1109/JSAC.2017.2720898 DOI: https://doi.org/10.1109/JSAC.2017.2720898

[5] Bin Farooq, U., Sajid Hashmi, U., Qadir, J., Imran, A., & Mian, A. N. (2018). User transmit power minimization through uplink resource allocation and user association in HetNets. Proceedings of the IEEE Global Communications Conference (GLOBECOM), 1–6. https://doi.org/10.1109/GLOCOM.2018.8647409 DOI: https://doi.org/10.1109/GLOCOM.2018.8647409

[6] Song, L., Chai, K. K., Chen, Y., Schormans, J., Loo, J., & Vinel, A. (2017). QoS-aware energy-efficient cooperative scheme for cluster-based IoT systems. IEEE Systems Journal, 11(3), 1447–1455. https://doi.org/10.1109/JSYST.2015.2465292 DOI: https://doi.org/10.1109/JSYST.2015.2465292

[7] Wang, J., Jiang, C., Wei, Z., Pan, C., Zhang, H., & Ren, Y. (2019). Joint UAV hovering altitude and power control for space-air-ground IoT networks. IEEE Internet of Things Journal, 6(2), 1741–1753. https://doi.org/10.1109/JIOT.2018.2875493 DOI: https://doi.org/10.1109/JIOT.2018.2875493

[8] Bakht, K., Ahmed, R., Ijaz, S., Ahmed, A., Jabeen, F., & Khwaja, A. S. (2019). Power allocation and user assignment scheme for beyond 5G heterogeneous networks. Wireless Communications and Mobile Computing, 1–9. https://doi.org/10.1155/2019/2472783 DOI: https://doi.org/10.1155/2019/2472783

[9] Ha, V. N., & Le, L. B. (2014). Fair resource allocation for OFDMA femtocell networks with macrocell protection. IEEE Transactions on Vehicular Technology, 63(3), 1388–1401. https://doi.org/10.1109/TVT.2013.2284572 DOI: https://doi.org/10.1109/TVT.2013.2284572

[10] Kai, C., Li, H., Xu, L., Li, Y., & Jiang, T. (2019). Joint subcarrier assignment with power allocation for sum rate maximization of D2D communications in wireless cellular networks. IEEE Transactions on Vehicular Technology, 68(5), 4748–4759. https://doi.org/10.1109/TVT.2019.2903815 DOI: https://doi.org/10.1109/TVT.2019.2903815

[11] Saad, H., Mohamed, A., & Elbatt, T. (2012). Distributed cooperative Q-learning for power allocation in cognitive femtocell networks. Proceedings of the IEEE Vehicular Technology Conference (VTC-Fall), 1–5. https://doi.org/10.1109/VTCFall.2012.6399230 DOI: https://doi.org/10.1109/VTCFall.2012.6399230

[12] Tefft, J. R., & Kirsch, N. J. (2013). A proximity-based Q-learning reward function for femtocell networks. Proceedings of the IEEE Vehicular Technology Conference (VTC-Fall), 1–5. https://doi.org/10.1109/VTCFall.2013.6692057 DOI: https://doi.org/10.1109/VTCFall.2013.6692057

[13] Amiri, R., Mehrpouyan, H., Fridman, L., Mallik, R. K., Nallanathan, A., & Matolak, D. (2018). A machine learning approach for power allocation in HetNets considering QoS. Proceedings of the IEEE International Conference on Communications (ICC), 1–6. https://doi.org/10.1109/ICC.2018.8422864 DOI: https://doi.org/10.1109/ICC.2018.8422864

[14] Zhang, G. A., Gu, J. Y., Bao, Z. H., Xu, C., & Zhang, S. B. (2014). Joint routing and channel assignment algorithms in cognitive wireless mesh networks. Transactions on Emerging Telecommunications Technologies, 25(3), 294–307. https://doi.org/10.1002/ett DOI: https://doi.org/10.1002/ett.2560

[15] Zhang, Y., Kang, C., Ma, T., Teng, Y., & Guo, D. (2018). Power allocation in multi-cell networks using deep reinforcement learning. Proceedings of the IEEE Vehicular Technology Conference (VTC-Fall), 1–6. https://doi.org/10.1109/VTCFall.2018.8690757 DOI: https://doi.org/10.1109/VTCFall.2018.8690757

[16] Meng, F., Chen, P., Wu, L., & Cheng, J. (2020). Power allocation in multi-user cellular networks: Deep reinforcement learning approaches. IEEE Transactions on Wireless Communications, 19(10), 6255–6267. https://doi.org/10.1109/TWC.2020.3001736 DOI: https://doi.org/10.1109/TWC.2020.3001736

[17] Ding, H., Zhao, F., Tian, J., Li, D., & Zhang, H. (2020). A deep reinforcement learning for user association and power control in heterogeneous networks. Ad Hoc Networks, 102, 102069. https://doi.org/10.1016/j.adhoc.2019.102069 DOI: https://doi.org/10.1016/j.adhoc.2019.102069

[18] Giannopoulos, A., Oikonomou, K., Koutsopoulos, I., & Antoniou, E. (2021). Demand-driven power allocation in wireless networks with deep Q-learning. Proceedings of the IEEE International Symposium on a World of Wireless, Mobile and Multimedia Networks (WoWMoM), 248–251. https://doi.org/10.1109/WoWMoM51794.2021.00045 DOI: https://doi.org/10.1109/WoWMoM51794.2021.00045

[19] Wang, H., Fan, Y., & Yang, L. (2021). A power allocation algorithm for underwater acoustic communication networks based on reinforcement learning. Proceedings of the IEEE International Conference on Civil Aviation Safety and Information Technology (ICCASIT), 986–990. https://doi.org/10.1109/ICCASIT53235.2021.9633599 DOI: https://doi.org/10.1109/ICCASIT53235.2021.9633599

[20] Jiang, S., & Zheng, J. (2021). A Q-learning based dynamic power control algorithm for D2D communication underlaying cellular networks. Proceedings of the IEEE Wireless Communications and Signal Processing (WCSP), 1–5. https://doi.org/10.1109/WCSP52459.2021.9613167 DOI: https://doi.org/10.1109/WCSP52459.2021.9613167

[21] Anzaldo, A., & Andrade, A. G. (2022). Deep reinforcement learning for power control in multi-tasks wireless cellular networks. Proceedings of the IEEE Mediterranean Communication and Networking Conference (MeditCom), 61–65. https://doi.org/10.1109/MeditCom55741.2022.9928617 DOI: https://doi.org/10.1109/MeditCom55741.2022.9928617

[22] Huangi, R., Si, J., Shi, J., & Li, Z. (2021). Deep-reinforcement-learning-based resource allocation in ultra-dense networks. Proceedings of the IEEE Wireless Communications and Signal Processing (WCSP), 1–5. https://doi.org/10.1109/WCSP52459.2021.9613186 DOI: https://doi.org/10.1109/WCSP52459.2021.9613186

[23] Zhao, Y., Peng, T., Guo, Y., & Wang, W. (2021). Energy-efficient uplink power allocation in ultra-dense networks through multi-agent reinforcement learning. Proceedings of the IEEE Vehicular Technology Conference (VTC-Fall), 1–7. https://doi.org/10.1109/VTC2021-Fall52928.2021.9625554 DOI: https://doi.org/10.1109/VTC2021-Fall52928.2021.9625554

[24] Sun, M., Mei, E., Wang, S., & Jin, Y. (2023). Joint DDPG and unsupervised learning for channel allocation and power control in centralized wireless cellular networks. IEEE Access, 11, 42191–42203. https://doi.org/10.1109/ACCESS.2023.3270316 DOI: https://doi.org/10.1109/ACCESS.2023.3270316

[25] Valcarce, A., & Zhang, J. (2010). Empirical indoor-to-outdoor propagation model. IEEE Access, 9, 682–685. https://doi.org/10.1109/ACCESS.2010.5643660 DOI: https://doi.org/10.1109/LAWP.2010.2058085

[26] Bin Sediq, A., Gohary, R. H., Schoenen, R., & Yanikomeroglu, H. (2013). Optimal and low-complexity power allocation for OFDMA-based relay systems. IEEE Transactions on Wireless Communications, 12(1), 171–184. https://doi.org/10.1109/TWC.2012.112112.110796 DOI: https://doi.org/10.1109/TWC.2013.061413.121703

[27] Khenak, F. (2010). V-learning. Proceedings of the 2010 International Conference on Computer Information Systems and Industrial Management Applications (CISIM), 228–232. https://doi.org/10.1109/CISIM.2010.5643660 DOI: https://doi.org/10.1109/CISIM.2010.5643660

[28] Smith, J. Q., & White, D. J. (1994). Markov decision processes. Mathematics of Operations Research, 157(1), 1–25.

[29] Hierons, R. (1999). Machine learning (T. M. Mitchell). McGraw-Hill.

[30] Sutton, R. S., & Barto, A. G. (2018). Reinforcement learning: An introduction. MIT Press.

Downloads

Published

2025-06-26

How to Cite

Abhishek Kumar Verma, Vinay Kumar Singh, & M. R. Khan. (2025). Distributed Reinforcement Learning Based Efficient Radio Resource Allocation For Heterogeneous IoT Wireless Network. International Journal of Computational and Experimental Science and Engineering, 11(3). https://doi.org/10.22399/ijcesen.2852

Issue

Section

Research Article