Unified Analysis of Algorithms for Dictionary Learning

The advantages of data sparse representation have generated a lot of interest in recent years. The goal of dictionary learning is to seek an over-complete dictionary adaptively by data, from which every signal can be represented approximately by a linear combination of only a few columns of the dictionary. This problem is often solved by solving two optimization processes iteratively: sparse approximation and dictionary update. We analyze these two processes respectively. The drawback of the greedy algorithm for sparse approximation is discussed in this paper, and we give a new forward-backward index updating strategy for sparse approximation, which makes use of previous sparse patterns in iterations. A unified analysis of the framework for dictionary update is given, and we conclude that the dictionary update of existing algorithms all can be regarded as the projected gradient algorithm on Euclidean space or Oblique manifold with different choices of step size. Based on our analysis, a new algorithm is proposed for dictionary learning with a convergent cost function sequence. Numerical simulations show that our algorithm can be applied to refine the solutions produced by existing dictionary learning algorithms.