[R] Regularization Methods

1. Formular of Regularization Methods

$$ Q_{\lambda}(\beta_0, \beta) = -l(\beta_0, \beta) + p_{\lambda}(\beta)$$

 

2. The negative log-likelihood function

• Quantitative outcome: least square loss function
• Binary outcome: logistic likelihood
• Matched case-control outcome: conditional logistic likelihood
• Count outcome: Poisson likelihood
• Qualitative outcome: Multinomial likelihood
• Survival outcome: Cox partial likelihood

 

3. Types of Penalty Functions

• Convex penalty functions
  • Lasso (Tibshirani, JRSS, 1996)
  • Fused lasso (Tibshirani et al. JRSS, 2005)
  • Adaptive lasso (Zou, JASA, 2006)
  • Elastic-net (Zou and Hastie, JRSS, 2005)
• Non-convex penalty functions
  • lq-norm penalty with 0 < q < 1
  • Smoothly clipped absolute deviation (SCAD) (Fan and Li, JASA, 2005)
  • Minimax concave penalty (MCP) (Zhang, AOS, 2010)
•  Group structure penalty functions
  • Group lasso (Yuan and Lin, JRSS, 2006)
  • Graph-constrained regularization (Li and Li, AOAS 2010)
  • Sparse group lasso (Simon et al. JGCS 2013)