| 170 | 3 | 378 |
| 下载次数 | 被引频次 | 阅读次数 |
为了减小无人机辅助无线传感器网络(Unmanned Aerial Vehicle Assisted Wireless Sensor Network, UAV-WSN)数据收集的信息新鲜度(the Age of Information, AoI),提出一种改进的深度确定性策略梯度(Deep Deterministic Policy Gradient, DDPG)算法。构建最小AoI的马尔可夫决策过程(Markov Decision Process, MDP)模型，通过经验回放矩阵和双层网络结构提高算法的收敛速度。将玻尔兹曼策略引入搜索策略中，解决UAV-WSN系统在选择最优动作时局部最优的问题，采用多层长短期记忆神经网络模型，以控制经验池中信息的记忆和遗忘程度，避免算法训练时回合间相互影响。将所提算法与演员-评论家(Actor-Critic, AC)算法、深度Q网络(Deep Q-Network, DQN)算法、DDPG算法及random算法对比，结果表明，改进的DDPG算法具有较好的收敛性和稳定性，能够最小化AoI。
Abstract：In order to reduce the age of information(AoI) of data collection in unmanned aerial vehicle assisted wireless sensor network(UAV-WSN), an improved deep deterministic policy gradient(DDPG) algorithm is proposed. The Markov decision process(MDP) model with the minimum AoI is constructed. The convergence speed of the algorithm is improved by the experience playback matrix and the two-layer network structure. The Boltzmann strategy is introduced into the search strategy to solve the UAV-WSN system. The problem of local optimum when selecting the optimal action is introduced into the multi-layer long-term and short-term memory neural network model to control the memory and forgetting degree of information in the experience pool, and avoid the mutual influence between rounds during algorithm training. The proposed algorithm is compared with the actor-critic(AC) algorithm, the deep Q-network(DQN) algorithm, the DDPG algorithm, and the random algorithm. The results show that the improved DDPG algorithm has better convergence and stability, and can minimize the AoI.
[1] UWAECHIA A N,MAHYUDDI N M.A comprehensive survey on millimeter wave communications for fifth-generation wireless networks:Feasibility and challenges[J].IEEE Access,2020,8:62367-62414.
[2] 赵鹏越，全齐全，邓宗全.旋翼式火星无人机技术发展综述[J].宇航学报，2018,39(2):121-130.ZHAO P Y,QUAN Q Q,DENG Z Q.Overview of research on Rotarywing mars unmanned aerial vehicles[J].Astronautical Journal,2018,39(2):121-130.(in Chinese)
[3] MAO K,NING B,ZHU Q.ML-based delay-angle-joint path loss prediction for UAV mmWave channels[EB/OL].[2023-10-11].https://springer.longhoe.net/article/10.1007/s11276-021-02817-6#google_vignette.
[4] 陈新颖，盛敏，李博.面向6G的无人机通信综述[J].电子与信息学报，2022,44(3):781-789.CHEN X Y,CHENG M,LI B.Survey on unmanned aerial vehicle communications for 6G[J].Journal of Electronics and Information,2022,44(3):781-789.(in Chinese)
[5] ARAFAT M Y,HABIB M A,MOH S.Routing protocols for UAV-aided wireless sensor networks[J].Applied Sciences,2020,10(12):1-23.
[6] WU T,LIU J.A novel AI-based framework for AoI-optimal trajectory planning in UAV-assisted wireless sensor networks[J].IEEE Transactions on Wireless Communications,2021,21(4):2462-2475.
[7] TRIPATHI V,TALAK R,MODIANO E.Age optimal information gatheringand dissemination on graphs[J].IEEE Transactions on Mobile Computing,2021,22(1):54-68.
[8] LIU J,WANG X,BAI B.Age-optimal trajectory planning for UAV-assisted data collection[C]//Proceedings of the IEEE INFOCOM 2018-IEEE Conference on Computer Communications Workshops.New York:IEEE,2018:553-558.
[9] SUN C,WEI D.An age-based data collection and path planning algorithm in UAV-assisted wireless sensor networks[C]//Proceedings of the 2022 5th International Conference on Artificial Intelligence and Pattern Recognition.New York:ACM,2022:995-1000.
[10] WU T,LIU J.A novel AI-based framework for AoI-optimal traje ctory planning in UAV-assisted wirelesssensor networks[J].IEEE Transactions on Wireless Communications,2021,21(4):2462-2475.
[11] 刘伯阳，郭天润，宋佳佳，等.认知边缘计算网络中的计算能效优化方案[J].西安邮电大学报，2022,27(5):1-10.LIU B Y,GUO T R,SONG J J.Computation efficiency maximization for cognitive radio based mobile edge computing networks[J].Journal of Xi’an University of Posts and Telecommunications,2022,27(5):1-10.(in Chinese)
[12] XIONG J,GUO H.Task offloading in UAV-aided edge computing:Bit allocation and trajectory optimization[J].IEEE Communications Letters,2019,23(3):538-541.
[13] WANGY,FANG W,DING Y.Computation offloading optimization for UAV-assisted mobile edge computing:A deep deterministic policy gradient approach[J].Wireless Networks,2021,27(4):2991-3006.
[14] SALEEM U,LIU Y,JANGSHER S.Performance guaranteed partial offloading for mobile edge computing[C]//Proceedings of the 2018 IEEE Global Communications Conference.New York:IEEE,2018:1-6.
[15] HU Q,CAI Y,YU G.Joint offloading and trajectory design for UAV-enabled mobile edge computing systems[J].IEEE Internet of Things Journal,2018,6(2):1879-1892.
[16] CARRILHO E,CARDOSO M,MARQUES F M.10-MDP based dentaladhesives:Adhesive interface characterization and adhesive stabilitya systematic review[J].Materials,2019,12(5):790.
[17] 鲁韵，王姣.一种基于改进贝尔曼方程的最短路径规划算法[J].武汉理工大学学报(交通科学与工程版),2022,46(6):1003-1007.LU Y,WANG J.A shortest path planning algorithm based on improved Bellman equation [J].Journal of Wuhan University of Technology(Transportation Science and Engineering Edition),2022,46(6):1003-1007.(in Chinese)
[18] 李新华.基于多智能体强化学习的露天矿车辆调度方法[J].西安邮电大学学报，2023,28(1):104-110.LI X H.Vehicle scheduling algorithm in open pit mine based on multi-agent reinforcement learning[J].Journal of Xi’an University of Posts and Telecommunications,2023,28(1):104-110.(in Chinese)
[19] 汪友明，程琳.改进的CNN-LSTM轴承故障诊断方法[J].西安邮电大学学报，2021,26(1):97-103.WANG Y M,CHENG L.Improved CNN-LSTM rolling bearing fault diagnosis method[J].Journal of Xi’an University of Posts and Telecommunications,2021,26(1):97-103.(in Chinese)
基本信息:
DOI：10.13682/j.issn.2095-6533.2024.03.001
中图分类号:TP212.9;TN929.5;V279
引用信息:
[1]孙爱晶,魏德,孙驰.面向无人机辅助WSN的改进DDPG算法[J].西安邮电大学学报,2024,29(03):1-11.DOI:10.13682/j.issn.2095-6533.2024.03.001.
基金信息:
国家自然科学基金项目（62271391）; 陕西省教育厅服务地方专项科研项目（21JC032）