Ph.D. Defense: Qianqian Tong

Title: Parallel and Federated Algorithms for Large-scale Machine Learning Problems

Major Advisor: Dr. Jinbo Bi

Associate Advisors: Dr. Sanguthevar Rajasekaran, Dr. Dongjin Song

Date/Time: Friday, Nov 11, 2022, 1:00 pm

Location: WebEx Online

Meeting link:  https://uconn-cmr.webex.com/uconn-cmr/j.php?MTID=mff573eb8db5afdf743745e8a9b4dbe40

Meeting number:  ‪2623 968 4658

Join by phone:  +1-415-655-0002 US Toll

Access code:  242 768 63

Abstract:

Stochastic optimization algorithms have been the main force for scalable machine learning owing to efficient per-iteration complexity. Besides stochastic gradients, second-order (Hessian matrix) derivatives can be used to capture more curvature information.  However, the high computational cost and expensive memory storage of matrix render second-order methods prohibitive for large-scale problems. In this work, we examine two approaches to efficiently employ the second-order information.

The first approach is to use quasi-Newton methods that approximate the Hessian matrix. We propose an asynchronous parallel algorithm for stochastic quasi-Newton (AsySQN) method, which has a full parallelism with a convergence guarantee, different from existing methods that only parallelize a single step of the method. Empirical evaluations demonstrate the speedup in comparison with the non-parallel SQN, and the effectiveness in solving ill-conditioned problems.

Another approach is to use adaptive gradient methods (AGMs), which approximate Hessian information by second-order momentum.  AGMs have been widely used to optimize nonconvex problems in the deep learning area. We improve AGMs from two current limitations: (1) adaptive learning rate (A-LR) varies significantly across the dimensions of the optimization problem and over epochs; (2) no convergence analysis of AGMs discusses their hyper-parameters.  We propose new AGMs that calibrate the A-LR with an activation function and show that the proposed methods outperform existing AGMs and generalize better in multiple deep learning tasks.

The AGMs are further extended in the federated learning setting, which allows loads of edge computing devices to collaboratively learn a global model without data sharing. We propose a family of effective federated AGMs via calibrated learning rate, to alleviate generalization performance deterioration caused by dissimilarity of data population among devices.  Our analysis shows that the proposed methods converge to a first-order stationary point under non-IID and unbalanced data settings for nonconvex optimization. To further improve test performance, we compare different calibration schemes for the adaptive learning rate with the most advanced federated algorithms and evaluate the benefits of AGMs over the current federated learning methods.