Abstract

Big data is one of the major challenges in machine learning, which leads to slow training and scalability issues of models. In this work, we have identified problem formulation, problem solvers, optimization strategies and platform/framework utilization, as major areas to tackle the challenge. But out of these potential areas, recently, researchers have focused on stochastic approximation algorithms, coordinate descent algorithms, proximal algorithms and parallel & distributed algorithms to tackle the challenge. We have utilized the best of stochastic approximation and coordinate descent approaches to propose a batch block optimization framework (BBOF), which has been used with first and second order methods to solve the large-scale learning problems. But it has been observed that the stochastic approximation and coordinate descent, do not work well when combined together because the advantage is lost in extra overhead to implement BBOF due to double sampling, i.e., sampling of data points and that of features. We have proposed stochastic average adjusted gradient (SAAG-I, II, III and IV) methods, as variance reduction techniques to solve the large-scale problems. We have also proposed stochastic trust region Newton (STRON) method, which solves the Newton system inexactly to handle the large-scale problems. Moreover, simple sampling techniques have been proposed to improve the training time of models by reducing the data access time. We have provided theoretical analysis and empirical results which have proved efficacy of proposed methods against the existing techniques.