基于加权平均随机递归梯度下降算法Stochastic Recursive Gradient Descent Algorithm with Weighted Average
费经泰;郝庆一;程一元;孙钊;
摘要(Abstract)：
对传统的随机递归梯度下降算法(SARAH)采用梯度加权平均技术，在强凸条件下提出了一种加权的SARAH算法—WA-SARAH算法。然后理论上证明了该算法具有线性收敛速率，并且给出了相应的收敛阶。通过合理地选取加权系数，发现WA-SARAH算法的收敛阶要优于SARAH算法。最后通过数值实验，验证了WA-SARAH算法的合理性。
关键词(KeyWords)： 机器学习;随机递归梯度下降算法;加权平均;加权系数;收敛阶
基金项目(Foundation): 安徽省自然科学基金项目“视野受限情形下的行人流及相关复杂系统的建模与实验研究”(1908085MA22);; 安徽高校自然科学研究项目“分布式机器学习的算法设计与理论研究”(KJ2021A1033);; 巢湖学院校级科研项目“加速随机方差缩减梯度下降算法研究”(XLY-202105,XLY-202103)资助
作者(Authors): 费经泰;郝庆一;程一元;孙钊;
参考文献(References)：
- [1] Jorge N.Optimization Methods for Large-Scale Machine Learning[J].Siam Review,2016,60(2):223-311.
- [2] Nesterov Y.Introductory Lectures on Convex Optimization:A Basic Course[M].Boston:Springer Science Business Media,2013:51-110.
- [3] Robbins H,Monro S.A Stochastic Approximation Method[J].Annals of Mathematical Statistics,1951,22(3):400-407.
- [4] Shamir O,Zhang T.Stochastic Gradient Descent for Non-smooth Optimization:Convergence Results and Optimal Averaging Schemes[C]//Dasgupta S,McAllester D,et al.International Conference on Machine Learning.Georgia:PMLR,2013:71-79.
- [5] Bottou L.Large-scale Machine Learning with Stochastic Gradient Descent[C]// Lechevallier Y,Saporta G,et al.International Conference on Computational Statistics.Paris:Physica Verlag HD,2010:177-186.
- [6] Shalev-Shwartz S,Singer Y,Srebro N,et al.Pegasos:Primal Estimated Sub-Gradient Solver for SVM[J].Mathematical programming,2011,127(1):3-30.
- [7] Johnson R,Zhang T.Accelerating Stochastic Gradient Descent Using Predictive Variance Reduction[J].News in physiological sciences,2013,1(3):315-323.
- [8]Schmidt M,Roux N L,Bach F.Minimizing Finite Sums With the Stochastic Average Gradient[J].Mathematical Programming,2017,162(1-2):83-112.
- [9] Defazio A,Bach F,Lacoste-Julien S.SAGA:A Fast Incremental Gradient Method With Support for Non-Strongly Convex Composite Objectives[C]//Corinna C,Neil D L,Daniel D L,et al.Advances in Neural Information Processing Systems 28.Canada:MIT Press,2014:1646-1654.
- [10]Nguyen L M,Liu J,Scheinberg K,et al.SARAH:A Novel Method for Machine Learning Problems Using Stochastic Recursive Gradient[C]//Precup D,Teh Y W.International Conference on Machine Learning.Australia:PMLR,2017:2613-2621.