2024 03 v.29 1-11
面向无人机辅助WSN的改进DDPG算法
基金项目(Foundation):
国家自然科学基金项目（62271391）;
陕西省教育厅服务地方专项科研项目（21JC032）
邮箱(Email):
DOI:
10.13682/j.issn.2095-6533.2024.03.001
中文作者单位:
西安邮电大学通信与信息工程学院;陕西省信息通信网络及安全重点实验室;
摘要(Abstract):
为了减小无人机辅助无线传感器网络(Unmanned Aerial Vehicle Assisted Wireless Sensor Network, UAV-WSN)数据收集的信息新鲜度(the Age of Information, AoI),提出一种改进的深度确定性策略梯度(Deep Deterministic Policy Gradient, DDPG)算法。构建最小AoI的马尔可夫决策过程(Markov Decision Process, MDP)模型，通过经验回放矩阵和双层网络结构提高算法的收敛速度。将玻尔兹曼策略引入搜索策略中，解决UAV-WSN系统在选择最优动作时局部最优的问题，采用多层长短期记忆神经网络模型，以控制经验池中信息的记忆和遗忘程度，避免算法训练时回合间相互影响。将所提算法与演员-评论家(Actor-Critic, AC)算法、深度Q网络(Deep Q-Network, DQN)算法、DDPG算法及random算法对比，结果表明，改进的DDPG算法具有较好的收敛性和稳定性，能够最小化AoI。
关键词(KeyWords):
无人机;无线传感器网络;深度确定性策略梯度;信息新鲜度;玻尔兹曼策略;长短记忆神经网络
| 157 | 2 | 240 |
| 下载次数 | 被引频次 | 阅读次数 |
参考文献
[1] UWAECHIA A N,MAHYUDDI N M.A comprehensive survey on millimeter wave communications for fifth-generation wireless networks:Feasibility and challenges[J].IEEE Access,2020,8:62367-62414.
[2] 赵鹏越，全齐全，邓宗全.旋翼式火星无人机技术发展综述[J].宇航学报，2018,39(2):121-130.ZHAO P Y,QUAN Q Q,DENG Z Q.Overview of research on Rotarywing mars unmanned aerial vehicles[J].Astronautical Journal,2018,39(2):121-130.(in Chinese)
[3] MAO K,NING B,ZHU Q.ML-based delay-angle-joint path loss prediction for UAV mmWave channels[EB/OL].[2023-10-11].https://springer.longhoe.net/article/10.1007/s11276-021-02817-6#google_vignette.
[4] 陈新颖，盛敏，李博.面向6G的无人机通信综述[J].电子与信息学报，2022,44(3):781-789.CHEN X Y,CHENG M,LI B.Survey on unmanned aerial vehicle communications for 6G[J].Journal of Electronics and Information,2022,44(3):781-789.(in Chinese)
[5] ARAFAT M Y,HABIB M A,MOH S.Routing protocols for UAV-aided wireless sensor networks[J].Applied Sciences,2020,10(12):1-23.
[6] WU T,LIU J.A novel AI-based framework for AoI-optimal trajectory planning in UAV-assisted wireless sensor networks[J].IEEE Transactions on Wireless Communications,2021,21(4):2462-2475.
[7] TRIPATHI V,TALAK R,MODIANO E.Age optimal information gatheringand dissemination on graphs[J].IEEE Transactions on Mobile Computing,2021,22(1):54-68.
[8] LIU J,WANG X,BAI B.Age-optimal trajectory planning for UAV-assisted data collection[C]//Proceedings of the IEEE INFOCOM 2018-IEEE Conference on Computer Communications Workshops.New York:IEEE,2018:553-558.
[9] SUN C,WEI D.An age-based data collection and path planning algorithm in UAV-assisted wireless sensor networks[C]//Proceedings of the 2022 5th International Conference on Artificial Intelligence and Pattern Recognition.New York:ACM,2022:995-1000.
[10] WU T,LIU J.A novel AI-based framework for AoI-optimal traje ctory planning in UAV-assisted wirelesssensor networks[J].IEEE Transactions on Wireless Communications,2021,21(4):2462-2475.
[11] 刘伯阳，郭天润，宋佳佳，等.认知边缘计算网络中的计算能效优化方案[J].西安邮电大学报，2022,27(5):1-10.LIU B Y,GUO T R,SONG J J.Computation efficiency maximization for cognitive radio based mobile edge computing networks[J].Journal of Xi’an University of Posts and Telecommunications,2022,27(5):1-10.(in Chinese)
[12] XIONG J,GUO H.Task offloading in UAV-aided edge computing:Bit allocation and trajectory optimization[J].IEEE Communications Letters,2019,23(3):538-541.
[13] WANGY,FANG W,DING Y.Computation offloading optimization for UAV-assisted mobile edge computing:A deep deterministic policy gradient approach[J].Wireless Networks,2021,27(4):2991-3006.
[14] SALEEM U,LIU Y,JANGSHER S.Performance guaranteed partial offloading for mobile edge computing[C]//Proceedings of the 2018 IEEE Global Communications Conference.New York:IEEE,2018:1-6.
[15] HU Q,CAI Y,YU G.Joint offloading and trajectory design for UAV-enabled mobile edge computing systems[J].IEEE Internet of Things Journal,2018,6(2):1879-1892.
[16] CARRILHO E,CARDOSO M,MARQUES F M.10-MDP based dentaladhesives:Adhesive interface characterization and adhesive stabilitya systematic review[J].Materials,2019,12(5):790.
[17] 鲁韵，王姣.一种基于改进贝尔曼方程的最短路径规划算法[J].武汉理工大学学报(交通科学与工程版),2022,46(6):1003-1007.LU Y,WANG J.A shortest path planning algorithm based on improved Bellman equation [J].Journal of Wuhan University of Technology(Transportation Science and Engineering Edition),2022,46(6):1003-1007.(in Chinese)
[18] 李新华.基于多智能体强化学习的露天矿车辆调度方法[J].西安邮电大学学报，2023,28(1):104-110.LI X H.Vehicle scheduling algorithm in open pit mine based on multi-agent reinforcement learning[J].Journal of Xi’an University of Posts and Telecommunications,2023,28(1):104-110.(in Chinese)
[19] 汪友明，程琳.改进的CNN-LSTM轴承故障诊断方法[J].西安邮电大学学报，2021,26(1):97-103.WANG Y M,CHENG L.Improved CNN-LSTM rolling bearing fault diagnosis method[J].Journal of Xi’an University of Posts and Telecommunications,2021,26(1):97-103.(in Chinese)
[2] 赵鹏越，全齐全，邓宗全.旋翼式火星无人机技术发展综述[J].宇航学报，2018,39(2):121-130.ZHAO P Y,QUAN Q Q,DENG Z Q.Overview of research on Rotarywing mars unmanned aerial vehicles[J].Astronautical Journal,2018,39(2):121-130.(in Chinese)
[3] MAO K,NING B,ZHU Q.ML-based delay-angle-joint path loss prediction for UAV mmWave channels[EB/OL].[2023-10-11].https://springer.longhoe.net/article/10.1007/s11276-021-02817-6#google_vignette.
[4] 陈新颖，盛敏，李博.面向6G的无人机通信综述[J].电子与信息学报，2022,44(3):781-789.CHEN X Y,CHENG M,LI B.Survey on unmanned aerial vehicle communications for 6G[J].Journal of Electronics and Information,2022,44(3):781-789.(in Chinese)
[5] ARAFAT M Y,HABIB M A,MOH S.Routing protocols for UAV-aided wireless sensor networks[J].Applied Sciences,2020,10(12):1-23.
[6] WU T,LIU J.A novel AI-based framework for AoI-optimal trajectory planning in UAV-assisted wireless sensor networks[J].IEEE Transactions on Wireless Communications,2021,21(4):2462-2475.
[7] TRIPATHI V,TALAK R,MODIANO E.Age optimal information gatheringand dissemination on graphs[J].IEEE Transactions on Mobile Computing,2021,22(1):54-68.
[8] LIU J,WANG X,BAI B.Age-optimal trajectory planning for UAV-assisted data collection[C]//Proceedings of the IEEE INFOCOM 2018-IEEE Conference on Computer Communications Workshops.New York:IEEE,2018:553-558.
[9] SUN C,WEI D.An age-based data collection and path planning algorithm in UAV-assisted wireless sensor networks[C]//Proceedings of the 2022 5th International Conference on Artificial Intelligence and Pattern Recognition.New York:ACM,2022:995-1000.
[10] WU T,LIU J.A novel AI-based framework for AoI-optimal traje ctory planning in UAV-assisted wirelesssensor networks[J].IEEE Transactions on Wireless Communications,2021,21(4):2462-2475.
[11] 刘伯阳，郭天润，宋佳佳，等.认知边缘计算网络中的计算能效优化方案[J].西安邮电大学报，2022,27(5):1-10.LIU B Y,GUO T R,SONG J J.Computation efficiency maximization for cognitive radio based mobile edge computing networks[J].Journal of Xi’an University of Posts and Telecommunications,2022,27(5):1-10.(in Chinese)
[12] XIONG J,GUO H.Task offloading in UAV-aided edge computing:Bit allocation and trajectory optimization[J].IEEE Communications Letters,2019,23(3):538-541.
[13] WANGY,FANG W,DING Y.Computation offloading optimization for UAV-assisted mobile edge computing:A deep deterministic policy gradient approach[J].Wireless Networks,2021,27(4):2991-3006.
[14] SALEEM U,LIU Y,JANGSHER S.Performance guaranteed partial offloading for mobile edge computing[C]//Proceedings of the 2018 IEEE Global Communications Conference.New York:IEEE,2018:1-6.
[15] HU Q,CAI Y,YU G.Joint offloading and trajectory design for UAV-enabled mobile edge computing systems[J].IEEE Internet of Things Journal,2018,6(2):1879-1892.
[16] CARRILHO E,CARDOSO M,MARQUES F M.10-MDP based dentaladhesives:Adhesive interface characterization and adhesive stabilitya systematic review[J].Materials,2019,12(5):790.
[17] 鲁韵，王姣.一种基于改进贝尔曼方程的最短路径规划算法[J].武汉理工大学学报(交通科学与工程版),2022,46(6):1003-1007.LU Y,WANG J.A shortest path planning algorithm based on improved Bellman equation [J].Journal of Wuhan University of Technology(Transportation Science and Engineering Edition),2022,46(6):1003-1007.(in Chinese)
[18] 李新华.基于多智能体强化学习的露天矿车辆调度方法[J].西安邮电大学学报，2023,28(1):104-110.LI X H.Vehicle scheduling algorithm in open pit mine based on multi-agent reinforcement learning[J].Journal of Xi’an University of Posts and Telecommunications,2023,28(1):104-110.(in Chinese)
[19] 汪友明，程琳.改进的CNN-LSTM轴承故障诊断方法[J].西安邮电大学学报，2021,26(1):97-103.WANG Y M,CHENG L.Improved CNN-LSTM rolling bearing fault diagnosis method[J].Journal of Xi’an University of Posts and Telecommunications,2021,26(1):97-103.(in Chinese)
基本信息:
DOI：10.13682/j.issn.2095-6533.2024.03.001
中图分类号:TP212.9;TN929.5;V279
引用信息:
[1]孙爱晶,魏德,孙驰.面向无人机辅助WSN的改进DDPG算法[J].西安邮电大学学报,2024,29(03):1-11.DOI:10.13682/j.issn.2095-6533.2024.03.001.
基金信息:
国家自然科学基金项目（62271391）; 陕西省教育厅服务地方专项科研项目（21JC032）