Last Page Update: 23/07/2015

Latest Library Version: 1.0.6 (see Release Notes for more info)

LRSLibrary

Low-Rank and Sparse tools for Background Modeling and Subtraction in Videos.

The LRSLibrary provides a collection of low-rank and sparse decomposition algorithms in MATLAB. The library was designed for motion segmentation in videos, but it can be also used or adapted for other computer vision problems (for more information, please see this page). Currently the LRSLibrary contains a total of 93 matrix-based and tensor-based algorithms. The LRSLibrary was tested successfully in MATLAB R2013 and R2014 both x86 and x64 versions.

See also:

Presentation about Matrix and Tensor Tools for Computer Vision 
http://www.slideshare.net/andrewssobral/matrix-and-tensor-tools-for-computer-vision

MTT: Matlab Tensor Tools for Computer Vision
https://github.com/andrewssobral/mtt

IMTSL: Incremental and Multi-feature Tensor Subspace Learning
https://github.com/andrewssobral/imtsl

Citation

If you use this library for your publications, please cite it as:

@incollection{lrslibrary2015,
author    = {Sobral, Andrews and Bouwmans, Thierry and Zahzah, El-hadi},
title     = {LRSLibrary: Low-Rank and Sparse tools for Background Modeling and Subtraction in Videos},
booktitle = {Robust Low-Rank and Sparse Matrix Decomposition: Applications in Image and Video Processing},
publisher = {CRC Press, Taylor and Francis Group.}
year      = {2015},
}

GUI

The LRSLibrary provides an easy-to-use graphical user interface (GUI) for background modeling and subtraction in videos. First, run the setup script lrs_setup (or run('C:/lrslibrary/lrs_setup')), then run lrs_gui, and enjoy it!

(Click in the image to see the video)

Each algorithm is classified by its cpu time consumption with the following icons:

The algorithms were grouped in eight categories: RPCA for Robust PCA, ST for Subspace Tracking, MC for Matrix Completion, TTD for Three-Term Decomposition, LRR for Low-Rank Representation, NMF for Non-negative Matrix Factorization, NTF for Non-negative Tensor Factorization, or TD for standard Tensor Decomposition.

List of the algorithms available in LRSLibrary

• RPCA: Robust PCA (44)

• • RPCA: Robust Principal Component Analysis (De la Torre and Black, 2001) website
• • PCP: Principal Component Pursuit (Candes et al. 2009)
• • FPCP: Fast PCP (Rodriguez and Wohlberg, 2013)
• • R2PCP: Riemannian Robust Principal Component Pursuit (Hintermüller and Wu, 2014)
• • AS-RPCA: Active Subspace: Towards Scalable Low-Rank Learning (Liu and Yan, 2012)
• • ALM: Augmented Lagrange Multiplier (Tang and Nehorai 2011)
• • EALM: Exact ALM (Lin et al. 2009) website
• • IALM: Inexact ALM (Lin et al. 2009) website
• • IALM_LMSVDS: IALM with LMSVDS (Liu et al. 2012)
• • IALM_BLWS: IALM with BLWS (Lin and Wei, 2010)
• • APG_PARTIAL: Partial Accelerated Proximal Gradient (Lin et al. 2009) website
• • APG: Accelerated Proximal Gradient (Lin et al. 2009) website
• • DUAL: Dual RPCA (Lin et al. 2009) website
• • SVT: Singular Value Thresholding (Cai et al. 2008) website
• • ADM: Alternating Direction Method (Yuan and Yang, 2009)
• • LSADM: LSADM (Goldfarb et al. 2010)
• • L1F: L1 Filtering (Liu et al. 2011)
• • DECOLOR: Contiguous Outliers in the Low-Rank Representation (Zhou et al. 2011) website1 website2
• • RegL1-ALM: Low-Rank Matrix Approximation under Robust L1-Norm (Zheng et al. 2012) website
• • GA: Grassmann Average (Hauberg et al. 2014) website
• • GM: Grassmann Median (Hauberg et al. 2014) website
• • TGA: Trimmed Grassmann Average (Hauberg et al. 2014) website
• • STOC-RPCA: Online Robust PCA via Stochastic Optimization (Feng et al. 2013) website
• • MoG-RPCA: Mixture of Gaussians RPCA (Zhao et al. 2014) website
• • OP-RPCA: Robust PCA via Outlier Pursuit (Xu et al. 2012) website
• • NSA1: Non-Smooth Augmented Lagrangian v1 (Aybat et al. 2011)
• • NSA2: Non-Smooth Augmented Lagrangian v2 (Aybat et al. 2011)
• • PSPG: Partially Smooth Proximal Gradient (Aybat et al. 2012)
• • flip-SPCP-sum-SPG: Flip-Flop version of Stable PCP-sum solved by Spectral Projected Gradient (Aravkin et al. 2014)
• • flip-SPCP-max-QN: Flip-Flop version of Stable PCP-max solved by Quasi-Newton (Aravkin et al. 2014)
• • Lag-SPCP-SPG: Lagrangian SPCP solved by Spectral Projected Gradient (Aravkin et al. 2014)
• • Lag-SPCP-QN: Lagrangian SPCP solved by Quasi-Newton (Aravkin et al. 2014)
• • FW-T: SPCP solved by Frank-Wolfe method (Mu et al. 2014) website
• • BRPCA-MD: Bayesian Robust PCA with Markov Dependency (Ding et al. 2011) website
• • BRPCA-MD-NSS: BRPCA-MD with Non-Stationary Noise (Ding et al. 2011) website
• • VBRPCA: Variational Bayesian RPCA (Babacan et al. 2011)
• • PRMF: Probabilistic Robust Matrix Factorization (Wang et al. 2012) website
• • OPRMF: Online PRMF (Wang et al. 2012) website
• • MBRMF: Markov BRMF (Wang and Yeung, 2013) website
• • TFOCS-EC: TFOCS with equality constraints (Becker et al. 2011) website
• • TFOCS-IC: TFOCS with inequality constraints (Becker et al. 2011) website
• • GoDec: Go Decomposition (Zhou and Tao, 2011) website
• • SSGoDec: Semi-Soft GoDec (Zhou and Tao, 2011) website
• • GreGoDec: Greedy Semi-Soft GoDec Algotithm (Zhou and Tao, 2013) website

• ST: Subspace Tracking (3)

• • GRASTA: Grassmannian Robust Adaptive Subspace Tracking Algorithm (He et al. 2012) website
• • GOSUS: Grassmannian Online Subspace Updates with Structured-sparsity (Xu et al. 2013) website
• • pROST: Robust PCA and subspace tracking from incomplete observations using L0-surrogates (Hage and Kleinsteuber, 2013) website

• MC: Matrix Completion (5)

• • LRGeomCG: Low-rank matrix completion by Riemannian optimization (Bart Vandereycken, 2013) website1 website2
• • GROUSE: Grassmannian Rank-One Update Subspace Estimation (Balzano et al. 2010) website
• • OptSpace: Matrix Completion from Noisy Entries (Keshavan et al. 2009) website
• • FPC: Fixed point and Bregman iterative methods for matrix rank minimization (Ma et al. 2008) website
• • SVT: A singular value thresholding algorithm for matrix completion (Cai et al. 2008) website

• LRR: Low Rank Recovery (6)

• • EALM: Exact ALM (Lin et al. 2009)
• • IALM: Inexact ALM (Lin et al. 2009)
• • ADM: Alternating Direction Method (Lin et al. 2011)
• • LADMAP: Linearized ADM with Adaptive Penalty (Lin et al. 2011)
• • FastLADMAP: Fast LADMAP (Lin et al. 2011)
• • ROSL: Robust Orthonormal Subspace Learning (Shu et al. 2014) website

• TTD: Three-Term Decomposition (4)

• • 3WD: 3-Way-Decomposition (Oreifej et al. 2012) website
• • MAMR: Motion-Assisted Matrix Restoration (Ye et al. 2015) website
• • RMAMR: Robust Motion-Assisted Matrix Restoration (Ye et al. 2015) website
• • ADMM: Alternating Direction Method of Multipliers (Parikh and Boyd, 2014) website1 website2

• NMF: Non-Negative Matrix Factorization (14)

• • NMF-MU: NMF solved by Multiplicative Updates
• • NMF-PG: NMF solved by Projected Gradient
• • NMF-ALS: NMF solved by Alternating Least Squares
• • NMF-ALS-OBS: NMF solved by Alternating Least Squares with Optimal Brain Surgeon
• • PNMF: Probabilistic Non-negative Matrix Factorization
• • ManhNMF: Manhattan NMF (Guan et al. 2013) website
• • NeNMF: NMF via Nesterovs Optimal Gradient Method (Guan et al. 2012) website
• • LNMF: Spatially Localized NMF (Li et al. 2001)
• • ENMF: Exact NMF (Gillis and Glineur, 2012) website
• • nmfLS2: Non-negative Matrix Factorization with sparse matrix (Ji and Eisenstein, 2013) website
• • Semi-NMF: Semi Non-negative Matrix Factorization
• • Deep-Semi-NMF: Deep Semi Non-negative Matrix Factorization (Trigeorgis et al. 2014) website
• • iNMF: Incremental Subspace Learning via NMF (Bucak and Gunsel, 2009) website
• • DRMF: Direct Robust Matrix Factorization (Xiong et al. 2011) website

• NTF: Non-Negative Tensor Factorization (6)

• • betaNTF: Simple beta-NTF implementation (Antoine Liutkus, 2012)
• • bcuNTD: Non-negative Tucker Decomposition by block-coordinate update (Xu and Yin, 2012) website
• • bcuNCP: Non-negative CP Decomposition by block-coordinate update (Xu and Yin, 2012) website
• • NTD-MU: Non-negative Tucker Decomposition solved by Multiplicative Updates (Zhou et al. 2012)
• • NTD-APG: Non-negative Tucker Decomposition solved by Accelerated Proximal Gradient (Zhou et al. 2012)
• • NTD-HALS: Non-negative Tucker Decomposition solved by Hierarchical ALS (Zhou et al. 2012)

• TD: Tensor Decomposition (11)

• • HoSVD: Higher-order Singular Value Decomposition (Tucker Decomposition)
• • HoRPCA-IALM: HoRPCA solved by IALM (Goldfarb and Qin, 2013) website
• • HoRPCA-S: HoRPCA with Singleton model solved by ADAL (Goldfarb and Qin, 2013) website
• • HoRPCA-S-NCX: HoRPCA with Singleton model solved by ADAL (non-convex) (Goldfarb and Qin, 2013) website
• • Tucker-ADAL: Tucker Decomposition solved by ADAL (Goldfarb and Qin, 2013) website
• • Tucker-ALS: Tucker Decomposition solved by ALS
• • CP-ALS: PARAFAC/CP decomposition solved by ALS
• • CP-APR: PARAFAC/CP decomposition solved by Alternating Poisson Regression (Chi et al. 2011)
• • CP2: PARAFAC2 decomposition solved by ALS (Bro et al. 1999)
• • RSTD: Rank Sparsity Tensor Decomposition (Yin Li, 2010) website
• • t-SVD: Tensor SVD in Fourrier Domain (Zhang et al. 2013)

• Some remarks:

• • The FW-T algorithm of Mu et al. (2014) works only with CVX library. Download and install it in: lrslibrary/libs/cvx/.
• • The DECOLOR algorithm of Zhou et al. (2011) don't works in MATLAB R2014a(x64), but works successfully in MATLAB R2013b(x64) and both R2014a(x86) and R2013b(x86).

Usage example

For complete details and examples, please see the demo.m file.

%% First run the setup script
lrs_setup; % or run('C:/lrslibrary/lrs_setup')

%% Load configuration
lrs_load_conf;

%% Load video
input_avi = fullfile(lrs_conf.lrs_dir,'dataset','demo.avi');
output_avi = fullfile(lrs_conf.lrs_dir,'output','output.avi');

%% Processing videos
%
% Robust PCA
process_video('RPCA', 'FPCP', input_avi, output_avi);
% Subspace Tracking
process_video('ST', 'GRASTA', input_avi, output_avi);
% Matrix Completion
process_video('MC', 'GROUSE', input_avi, output_avi);
% Low Rank Recovery
process_video('LRR', 'FastLADMAP', input_avi, output_avi);
% Three-Term Decomposition
process_video('TTD', '3WD', input_avi, output_avi);
% Non-Negative Matrix Factorization
process_video('NMF', 'ManhNMF', input_avi, output_avi);
% Non-Negative Tensor Factorization
process_video('NTF', 'bcuNCP', input_avi, output_avi);
% Tensor Decomposition
process_video('TD', 'Tucker-ALS', input_avi, output_avi);

%% Processing matrices and tensors
%
load('dataset/trafficdb/traffic_patches.mat');
V = im2double(imgdb{100});
show_3dvideo(V);

%% Matrix-based algorithms
%
[M,m,n,p] = convert_video3d_to_2d(V);
show_2dvideo(M,m,n);

% Robust PCA
out = process_matrix('RPCA', 'FPCP', M, []);
% Subspace Tracking
out = process_matrix('ST', 'GRASTA', M, []);
% Matrix Completion
out = process_matrix('MC', 'GROUSE', M, []);
% Low Rank Recovery
out = process_matrix('LRR', 'FastLADMAP', M, []);
% Three-Term Decomposition
out = process_matrix('TTD', '3WD', M, []);
% Non-Negative Matrix Factorization
out = process_matrix('NMF', 'ManhNMF', M, []);

% Show results
show_results(M,out.L,out.S,out.O,p,m,n);

%% Tensor-based algorithms
T = tensor(V);

% Non-Negative Tensor Factorization
out = process_tensor('NTF', 'bcuNCP', T);
% Tensor Decomposition
out = process_tensor('TD', 'Tucker-ALS', T);

% Show results
show_3dtensors(T,out.L,out.S,out.O);

CPU time consumption

The figure below shows the average CPU time consumption and the speed classification of each algorithm to decompose a 2304x51 matrix or 48x48x51 tensor data. Both matrix and tensor data were built from dataset/demo.avi file. The experiments were performed in a Intel Core i7-3740QM CPU 2.70GHz with 16Gb of RAM running MATLAB R2013b and Windows 7 Professional SP1 64 bits.

A complete review evaluating the algorithms in many specific criterias will be published in a paper journal soon

About LRSLibrary

The LRSLibrary has been developed by Andrews Sobral thanks especially to Thierry Bouwmans for his continued support and for collaborating on this important initiative. I'm grateful to all authors who have contributed in some way to the success of the LRSLibrary.

How to contribute with LRSLibrary project

Everyone is invited to cooperate with the LRSLibrary project by sending to us any implementation of low-rank and sparse decomposition algorithms.

Option 1: email it to me (andrewssobral at gmail dot com).

Option 2: fork the library on GitHub, push your changes, then send me a pull request.

License

The LRSLibrary is free and open source for academic/research purposes (non-commercial)¹.

¹ Some algorithms of the LRSLibrary are free for commercial purposes and others not. First you need to contact the authors of your desired algorithm and check with them the appropriate license.

Problems or Questions

If you have any problems or questions, please contact the author: Andrews Sobral (andrewssobral at gmail dot com)

Release Notes:

• Version 1.0.6: Added three new algorithms: STOC-RPCA: Online Robust PCA via Stochastic Optimization of Feng et al. (2013), MoG-RPCA: Mixture of Gaussians RPCA of Zhao et al. (2014), and OP-RPCA: Robust PCA via Outlier Pursuit of Xu et al. (2012).

• Version 1.0.5: Added three new method categories ST (Subspace Tracking), MC (Matrix Completion), and TTD (Three-Term Decomposition). Added fifteen new algorithms: 3WD - 3-Way-Decomposition of Oreifej et al. (2012), MAMR and Robust MAMR - Motion-Assisted Matrix Restoration of Ye et al. (2015), ADMM - Alternating Direction Method of Multipliers of Parikh and Boyd (2014), GreGoDec - Greedy Semi-Soft GoDec Algotithm of Zhou and Tao (2013), GRASTA (Grassmannian Robust Adaptive Subspace Tracking Algorithm) of He et al. (2012), GOSUS (Grassmannian Rank-One Update Subspace Estimation) of Balzano et al. (2010), OptSpace - A Matrix Completion Algorithm of Keshavan et al. (2009), FPC - Fixed point and Bregman iterative methods for matrix rank minimization of Ma et al. (2008), SVT - A singular value thresholding algorithm for matrix completion of Cai et al. (2008), LRGeomCG - Low-rank matrix completion by Riemannian optimization of Bart Vandereycken (2013), RPCA - Robust Principal Component Analysis of De la Torre and Black (2001), GA - Grassmann Average, GM - Grassmann Median, and TGA - Trimmed Grassmann Average of Hauberg et al. (2014). Also fixed some errors.

• Version 1.0.4: Added a setup script, and configuration file. Also fixed some errors.

• Version 1.0.3: Added three new algorithms: FW-T (Mu et al. 2014), iNMF (Bucak and Gunsel, 2009) and DRMF (Xiong et al. 2011).

• Version 1.0.2: Added four new algorithms: GOSUS (Xu et al. 2013), pROST (Hage and Kleinsteuber, 2013), RegL1-ALM (Zheng et al. 2012) and ROSL (Shu et al. 2014).

• Version 1.0.1: Added RPCA-SPCP algorithms of Aravkin et al. (2014), thanks to Professor Stephen Becker.

• Version 1.0.0: First version with 64 algorithms.