Task Migration Strategy in Vehicular Networks Based on Reinforcement Learning

DOI: https://doi.org/10.62517/jbdc.202401418

Author(s)

Zou Jing, Gong Qishuai, Wang Zhe

Affiliation(s)

Guangxi Minzu University School of Artificial Intelligence, Nanning, China

Abstract

With the research and application of edge computing in vehicular networks, computing tasks can be offloaded from vehicles to roadside edge servers to reduce system service latency. However, as vehicles move, the computing tasks need to be migrated from one edge server to another. Predicting the vehicles movement trajectory and formulating a reasonable task migration plan for this is a key challenge that needs to be addressed. Traditional computing offloading methods cannot be directly applied in vehicular networks. Therefore, this paper constructs a vehicular task offloading system based on a multi-layered computing network, introduces a Markov mobility model to describe the vehicle movement trajectory, and solves the optimal migration path problem. Since this problem is NP-hard, a solution method based on a constrained Markov model is proposed, along with an Actor-Network Primal-Dual Deep Deterministic Policy Gradient (ANPD-DDPG) algorithm based on reinforcement learning to achieve the optimal solution. Finally, in simulation experiments, the proposed method is compared with existing research, showing about a 33% reduction in system delay and migration cost. The characteristics of the ANPD-DDPG algorithm in terms of convergence speed and system delay are also analyze.

Keywords

Internet of Vehicles; Edge Computing; Task Offloading; Task Migration

References

[1]DAI Y, Xyu D, MAHARJAN S, et al. Joint load balancing and offloading in vehicular edge computing and networks[J]. IEEE Internet of Things Journal, 2018, 6(3): 4377-4387. [2]Tamani N, Brik B, Lagraa N, et al. On link stability metric and fuzzy quantification for service selection in mobile vehicular cloud[J]. IEEE Transactions on Intelligent Transportation Systems, 2019, 21(5): 2050-2062. [3]Yang C, Liu Y, Chen X, et al. Efficient mobility-aware task offloading for vehicular edge computing networks[J]. IEEE Access, 2019, 7: 26652-26664. [4]Sorkhoh I, Ebrahimi D, Atallah R, et al. Workload scheduling in vehicular networks with edge cloud capabilities[J]. IEEE Transactions on Vehicular Technology, 2019, 68(9): 8472-8486. [5]Wang L, Jiao L, Li J, et al. MOERA: Mobility-agnostic online resource allocation for edge computing[J]. IEEE Transactions on Mobile Computing, 2018, 18(8): 1843-1856. [6]Zhang Yilin, Liang Yuzhu, Yin Mujun, et al. Survey on the Methods of Computation Offloading in Mobile Edge Computing[J]. Journal of Computer Science, 2021, 44(12): 2406-2430.) [7]Liu J, Mao Y, Zhang J, et al. Delay-optimal computation task scheduling for mobile-edge computing systems[C]//2016 IEEE international symposium on information theory (ISIT). IEEE, 2016: 1451-1455. [8]You C., Huang K. "Multiuser resource allocation for mobile edge computation offloading." Proceedings of the 2016 IEEE Global Communications Conference (GLOBECOM), Washington, USA, 2016, pp. 1-6. [9]Zhao P, Tian H, Qin C, et al. Energy-saving offloading by jointly allocating radio and computational resources for mobile edge computing[J]. IEEE access, 2017, 5: 11255-11268. [10]Wang S, Urgaonkar R, Zafer M, et al. Dynamic service migration in mobile edge computing based on Markov decision process[J]. IEEE/ACM Transactions on Networking, 2019, 27(3): 1272-1288. [11]Wang J, Hu J, Min G, et al. Online service migration in edge computing with incomplete information: A deep recurrent actor-critic method[J]. arXiv preprint arXiv:2012.08679, 2020: 41. [12]Wang S, Guo Y, Zhang N, et al. Delay-aware microservice coordination in mobile edge computing: A reinforcement learning approach[J]. IEEE Transactions on Mobile Computing, 2019, 20(3): 939-951. [13]Labriji I, Meneghello F, Cecchinato D, et al. Mobility aware and dynamic migration of MEC services for the Internet of Vehicles[J]. IEEE Transactions on Network and Service Management, 2021, 18(1): 570-584. [14]Ouyang T, Zhou Z, Chen X. Follow me at the edge: Mobility-aware dynamic service placement for mobile edge computing[J]. IEEE Journal on Selected Areas in Communications, 2018, 36(10): 2333-2345. [15]Gebrie H, Farooq H, Imran A. What machine learning predictor performs best for mobility prediction in cellular networks?[C]//2019 IEEE International Conference on Communications Workshops (ICC Workshops). IEEE, 2019: 1-6. [16]Al-Shaery A M, Ahmed S G, Aljassmi H, et al. Open Dataset for Predicting Pilgrim Activities for Crowd Management During Hajj Using Wearable Sensors[J]. IEEE Access, 2024. [17]Kim T, Sathyanarayana S D, Chen S, et al. Modems: Optimizing edge computing migrations for user mobility[J]. IEEE Journal on Selected Areas in Communications, 2022, 41(3): 675-689. [18]Kim T, Sathyanarayana S D, Chen S, et al. Modems: Optimizing edge computing migrations for user mobility[J]. IEEE Journal on Selected Areas in Communications, 2022, 41(3): 675-689. [19]IMANE A. Mobile Edge Computing for the Internet of Things[J]. 2019. [20]Rejiba Z, Masip-Bruin X, Marín-Tordera E. A survey on mobility-induced service migration in the fog, edge, and related computing paradigms[J]. ACM Computing Surveys (CSUR), 2019, 52(5): 1-33. [21]Liang Z, Liu Y, Lok T M, et al. Multi-cell mobile edge computing: Joint service migration and resource allocation[J]. IEEE Transactions on Wireless Communications, 2021, 20(9): 5898-5912. [22]Ngo M V, Luo T, Hoang H T, et al. Coordinated container migration and base station handover in mobile edge computing[C]//GLOBECOM 2020-2020 IEEE Global Communications Conference. IEEE, 2020: 1-6. [23]Ma L, Yi S, Carter N, et al. Efficient live migration of edge services leveraging container layered storage[J]. IEEE Transactions on Mobile Computing, 2018, 18(9): 2020-2033. [24]Xu Xiaobin, Wang Qi, Fan Cunqun, et al. An Aggregated Edge Computing Resource Management Method forSpace-Air-Ground Integrated Information Networks[J]. Journal of Computer Science, 2023, 46(04): 690-710.) [25]Kuang Zhufang, Chen Qinglin, Li Linfeng, et al. Multi-user Edge Computing Task offloading Scheduling and ResourceAllocation Based on Deep Reinforcement Learning[J]. Journal of Computer Science, 2022, 45(04): 812-824.) [26]Wang S, Urgaonkar R, Zafer M, et al. Dynamic service migration in mobile edge computing based on Markov decision process[J]. IEEE/ACM Transactions on Networking, 2019, 27(3): 1272-1288. [27] Zeng Yaoping, Jiang Weiwei, Liu Yueqiang, et al. Dynamic Offloading Algorithm for Tasks in Vehicle Edge Networks [J]. Computer Engineering and Applications, 2024, 60(14): 267-274.) [28]Labriji I, Meneghello F, Cecchinato D, et al. Mobility aware and dynamic migration of MEC services for the Internet of Vehicles[J]. IEEE Transactions on Network and Service Management, 2021, 18(1): 570-584. [29]Ouyang T, Zhou Z, Chen X. Follow me at the edge: Mobility-aware dynamic service placement for mobile edge computing[J]. IEEE Journal on Selected Areas in Communications, 2018, 36(10): 2333-2345. [30]Cao T, Qian Z, Wu K, et al. Service placement and bandwidth allocation for MEC-enabled mobile cloud gaming[C]//2021 IEEE 22nd International Symposium on a World of Wireless, Mobile and Multimedia Networks (WoWMoM). IEEE, 2021: 179-188. [31]Yang B, Chai W K, Xu Z, et al. Cost-efficient NFV-enabled mobile edge-cloud for low latency mobile applications[J]. IEEE Transactions on Network nd Service Management, 2018, 15(1): 475-488. [32]Fallah S N, Ganjkhani M, Shamshirband S, et al. Computational intelligence on short-term load forecasting: A methodological overview[J]. Energies, 2019, 12(3): 393. [33]Hu B, Hu W. Linkshare: Device-centric control for concurrent and continuous mobile-cloud interactions[C]//Proceedings of the 4th ACM/IEEE Symposium on Edge Computing. 2019: 15-29. [34] Xu Xiaolong, Fang Zijie, Qi Lianyong, et al. A Deep Reinforcement Learning-Based Distributed Service Offloading Method for Edge Computing Empowered Internet of Vehicles[J]. Journal of Computer Science, 2021, 44(12): 2382-2405.) [35]Ottenwälder B, Koldehofe B, Rothermel K, et al. MCEP: A mobility-aware complex event processing system[J]. ACM Transactions on internet technology (TOIT), 2014, 14(1): 1-24. [36]Jia Y, Wu C, Li Z, et al. Online scaling of NFV service chains across geo-distributed datacenters[J]. IEEE/ACM Transactions on Networking, 2018, 26(2): 699-710. [37]Bari F, Chowdhury S R, Ahmed R, et al. Orchestrating virtualized network functions[J]. IEEE Transactions on Network and Service Management, 2016, 13(4): 725-739. [38]Pande S K, Panda S K, Das S. Dynamic service migration and resource management for vehicular clouds[J]. Journal of Ambient Intelligence and Humanized Computing, 2021, 12: 1227-1247