Key points
 Ridge regression is a method of regularization that can help you deal with multicollinearity, improve the accuracy of your predictions, and reduce the complexity of your model.
 Ridge regression adds a penalty term to the ordinary least squares objective function, which is proportional to the sum of squared coefficients of the regression model.
 The penalty term is controlled by a lambda parameter, which determines how much the coefficients are shrunk towards zero.
 To implement ridge regression in R, you need to use the
glmnet
package, which provides functions for fitting generalized linear models with various types of regularization.
 To choose the optimal value of lambda, you need to use crossvalidation, a technique that splits the data into several subsets and uses some for training and some for testing.
Ridge Regression in R: Best Practices & Techniques
Ridge regression is a method of regularization that can help you deal with multicollinearity, improve the accuracy of your predictions, and reduce the complexity of your model.
In this article, you will learn how to use ridge regression in R, how it works, and how to compare it with other forms of regression. This article is worth reading because it will teach you a useful data analysis and machine learning technique and how to implement it using R.
What is Ridge Regression?
Ridge regression is a form of regression that adds a penalty term to the ordinary least squares (OLS) objective function. The penalty term is proportional to the sum of the squared coefficients of the regression model.
It means that ridge regression tries to minimize the sum of squared residuals (SSR) and the sum of squared coefficients (SSC) simultaneously. The penalty term is controlled by a lambda parameter, which determines how much the coefficients are shrunk towards zero. The higher the lambda, the more the coefficients are shrunk, and the lower the lambda, the more the coefficients are similar to the OLS estimates.
Ridge regression is also known as L2 regularization because it is equivalent to the square of the magnitude of the coefficients. Ridge regression can help you deal with multicollinearity, where some predictor variables are highly correlated.
Multicollinearity can cause the OLS estimates to be unstable, have high variance, and be sensitive to small changes in the data. By shrinking the coefficients, ridge regression reduces the estimates' variance and makes them more robust to multicollinearity.
However, ridge regression also has some drawbacks. One of them is that it cannot perform variable selection, meaning it cannot reduce the number of predictors in the model. All the coefficients are shrunk, but none are set to zero. It can make the model more complex and harder to interpret.
Another drawback is that ridge regression can introduce some bias in the estimates, meaning they can deviate from the true values. The bias increases as the lambda increases, and the model becomes less flexible and more prone to underfitting.
How to Implement Ridge Regression in R?
Function/Library  Description 

glmnet 
A package that provides functions for fitting generalized linear models with various types of regularization, such as ridge, lasso, and elastic net. 
lm.ridge 
A function from the MASS package that performs ridge regression using the method of ordinary ridge regression or generalized crossvalidation. 
ridge 
A package that provides functions for linear and logistic ridge regression. Additionally, it includes special functions for genomewide singlenucleotide polymorphism (SNP) data. 
cv.glmnet 
A function from the glmnet package that performs kfold crossvalidation for ridge regression models and returns the optimal value of lambda that minimizes the test error. 
predict 
A generic function that generates the predicted values of the response variable for a given model and new data. 
plot 
A generic function that produces a graphical display of a model or an object. 
summary 
A generic function that returns a summary of a model or an object, such as the coefficients, the lambda values, the degrees of freedom, and the deviance. 
tidy 
A function from the 
Libraries and Functions used in this Tutorial
To implement ridge regression in R, you need to use the glmnet
package, which provides functions for fitting generalized linear models with various types of regularization. You can install the package using the following command. After installing, load the package using the following command:
install.packages("glmnet") library(glmnet)
?glmnet
 x: a matrix of predictor variables
 y: a vector of response values
 alpha: a parameter that controls the type of regularization. alpha = 0 corresponds to ridge regression, alpha = 1 corresponds to lasso regression and 0 < alpha < 1 corresponds to elastic net regression, a combination of ridge and lasso.
 lambda: a parameter that controls the amount of regularization. You can specify a single value of lambda or a sequence of values. If you do not specify lambda, the function will automatically generate a sequence of 100 values, ranging from a very large value (corresponding to many shrinkages) to a very small value (corresponding to no shrinkage).
The function returns an object of class glmnet, which contains information about the fitted model, such as the coefficients, the lambda values, the degrees of freedom, and the deviance. You can access the elements of the object using the $ operator. For example, if you name the object model, you can access the coefficients using model$beta.
To illustrate how to use the glmnet function, we will use a builtin dataset in R called mtcars, which contains information about 32 cars, such as their miles per gallon (mpg), number of cylinders (cyl), displacement (disp), horsepower (hp), and weight (wt). We will try to predict the cars' mpg using the other variables as predictors.
Split the dataset
First, we must split the dataset into a training set and a test dataset using the sample function. We will use 80% of the data for train and 20% for testing.
set.seed(123) # for reproducibility
n < nrow(mtcars) # number of observations
train < sample(n, 0.8 * n) # indices of the training set
test < setdiff(1:n, train) # indices of the test set
x_train < as.matrix(mtcars[train, 1]) # predictor matrix for the training set
y_train < mtcars[train, 1] # response vector for the training set
x_test < as.matrix(mtcars[test, 1]) # predictor matrix for the test set
y_test < mtcars[test, 1] # response vector for the test set
Standardize the Predictor Variables
Next, we must standardize the predictor variables to have mean zero and unit variance. This is important for ridge regression because it ensures that the penalty term is applied equally to all the coefficients. We can use the scale
function to do this.
x_train < scale(x_train) # standardize the training predictors
x_test < scale(x_test) # standardize the test predictors
Ridge Regression Model
We are ready to fit the ridge regression model using the glmnet function. We will use the default values of alpha and lambda and let the function choose the optimal values for us.
model < glmnet(x_train, y_train) # fit the ridge regression model
summary
function, which will show us the length, class, mode, and dimensions of the elements.summary(model)
Length 
Class 
Mode 

a0 
85 
none 
numeric 
beta 
850 
dgCMatrix 
S4 
df 
85 
none 
numeric 
dim 
2 
none 
numeric 
lambda 
85 
none 
numeric 
dev.ratio 
85 
none 
numeric 
nulldev 
1 
none 
numeric 
npasses 
1 
none 
numeric 
jerr 
1 
none 
numeric 
offset 
1 
none 
logical 
call 
3 
none 
call 
nobs 
1 
none 
numeric 
We can see that the model object has lambda values and corresponding values of beta, df, dev. ratio, and a0. The beta element is a sparse matrix, which means that it only stores the nonzero values of the coefficients. The df element is the degrees of freedom, the number of nonzero coefficients. The dev.ratio element is the fraction of deviance the model explains. The a0 element is the intercept term.
plot(model, xvar = "lambda", label = TRUE) # plot the coefficients vs lambda
How to Choose the Optimal Value of Lambda?
One of the challenges of ridge regression is to choose the optimal value of lambda, which balances the tradeoff between bias and variance. If lambda is too large, the model will be more complex and underfit the data. If lambda is bigger, the model will be more complex and overfit the data.
A common way to choose the optimal value of lambda is to use crossvalidation, a technique that splits the data into several subsets and uses some for training and some for testing. By repeating this process for different lambda values, we can compare the performance of the models on the test subsets and choose the value of lambda that gives the lowest test error.
cv.glmnet
The glmnet package provides a function called cv.glmnet, which performs crossvalidation for ridge regression models. The function takes the same arguments as the glmnet function, plus some additional arguments, such as:
 nfolds: the number of folds to use for crossvalidation. A fold is a subset of the data used for testing, while the rest is used for training. The default value is 10, meaning the data is split into 10 subsets, and each subset is used as a test set once.
 type.measure: the type of error measure to use for crossvalidation. The default value is "mse", meaning the mean squared error (MSE) is used. MSE is the average of the squared differences between the predicted values and the actual values. Other options are "mae" for mean absolute error, "deviance" for deviance, and "class" for classification error.
 foldid: an optional vector of fold identifiers, which allows you to specify which observations belong to which fold. This can be useful if you want to use a predefined split of the data or if you want to use a stratified split that preserves the proportions of the response variable in each fold.
The function returns an object of class cv.glmnet, which contains information about the crossvalidation results, such as the lambda values, the crossvalidated errors, and the optimal lambda value. You can access the elements of the object using the $ operator. For example, if you name the object cv_model, you can access the optimal lambda value using cv_model$lambda.min.
To illustrate how to use the cv.glmnet function, we will use the same dataset and predictors as before and perform 10fold crossvalidation to choose the optimal lambda value for the ridge regression model.
cv_model < cv.glmnet(x_train, y_train, nfolds = 10) # perform 10fold crossvalidation
We can inspect the cv_model object using the summary function, which will show us the elements' length, class, mode, and dimensions.
summary(cv_model)
Length 
Class 
Mode 

lambda 
85 
none 
numeric 
cvm 
85 
none 
numeric 
cvsd 
85 
none 
numeric 
cvup 
85 
none 
numeric 
cvlo 
85 
none 
numeric 
nzero 
85 
none 
numeric 
call 
4 
none 
call 
name 
1 
none 
character 
glmnet.fit 
12 
elnet 
list 
lambda.min 
1 
none 
numeric 
lambda.1se 
1 
none 
numeric 
index 
2 
none 
numeric 
We can see that the cv_model object has 85 lambda values and corresponding values of cvm, cvsd, cvup, cvlo, and nzero. The cvm element is the mean crossvalidated error for each value of lambda. The cvsd element is the standard deviation of the crossvalidated error for each lambda value. The cvup and cvlo elements are the upper and lower confidence bounds for the crossvalidated error for each lambda value. The nzero element is the number of nonzero coefficients for each value of lambda. The lambda.min element is the value of lambda that gives the minimum crossvalidated error. The lambda.1se element is the largest value of lambda, giving a crossvalidated error within one standard error of the minimum.
We can also plot the cv_model object using the plot function, which will show us how the crossvalidated error changes as a function of lambda. The xaxis is on a log scale, so the smaller lambda values are on the right, and the larger values are on the left. The yaxis shows the values of the crossvalidated error, and the error bars show the confidence bounds. The vertical dotted lines indicate the values of lambda that give the minimum crossvalidated error and the largest error within one standard error of the minimum.
plot(cv_model) # plot the crossvalidated error vs lambda
How to Evaluate the Performance of the Ridge Regression Model?
After choosing the optimal value of lambda, we can evaluate the performance of the ridge regression model on the test set, which is the subset of the data that we did not use for training or crossvalidation. We can use the predict function to generate the predicted values of the response variable for the test set using the ridge regression model and the optimal value of lambda.
We can then compare the predicted values with the actual values and calculate some metrics to measure the accuracy of the predictions, such as the mean squared error (MSE), the root mean squared error (RMSE), and the coefficient of determination (Rsquared).
Mean and Root mean squared error
The MSE is the average of the squared differences between predicted and actual values. The RMSE is the square root of the MSE, which has the same units as the response variable. The Rsquared is the proportion of the variance in the response variable explained by the predictor variables. It ranges from 0 to 1, where 0 means that the model explains none of the variability, and 1 means that it explains all the variability. The higher the Rsquared, the better the model fits the data.
To illustrate how to evaluate the performance of the ridge regression model, we will use the same dataset and predictors as before and generate the predicted values for the test set using the predict
function. We will use the value of lambda that gives the minimum crossvalidated error, which is stored in the cv_model$lambda.min
element.
y_pred < predict(model, s = cv_model$lambda.min, newx = x_test) # generate the predicted values for the test set
mse < mean((y_pred  y_test)^2) # calculate the mean squared error
rmse < sqrt(mse) # calculate the root mean squared error
r2 < 1  sum((y_pred  y_test)^2) / sum((y_test  mean(y_test))^2) # calculate the coefficient of determination
print(paste("MSE:", mse)) # print the mean squared error
print(paste("RMSE:", rmse)) # print the root mean squared error
print(paste("Rsquared:", r2)) # print the coefficient of determination
[1] "RMSE: 2.27072515245759"
[1] "Rsquared: 0.564510750168619"
We can see that the MSE is 5.156, the RMSE is 2.27, and the Rsquared is 0.564. These values indicate that the ridge regression model fits the test data well and can predict the cars' mpg with reasonable accuracy.
However, we should also compare these values with those obtained using other forms of regression, such as linear regression, lasso regression, and elastic net regression, to see if we can improve the model's performance further. We will do this in the next section.
How to Compare Ridge Regression with Other Forms of Regression?
Ridge regression is not the only form of regression that can deal with multicollinearity and improve the accuracy of the predictions. Other forms of regression use different types of regularization, such as
 Lasso Regression
 Elastic net Regression
Lasso regression
Lasso regression is similar to ridge regression, but it uses the sum of the absolute values of the coefficients as the penalty term instead of the sum of the squared values. This means that lasso regression can perform variable selection by setting some of the coefficients to exactly zero and reducing the number of predictors in the model.
However, lasso regression can also have some drawbacks, such as being unstable when the predictors are highly correlated and unable to handle more predictors than observations.
Elastic net regression
Elastic net regression is a combination of ridge and lasso regression, which uses a weighted sum of the squared and absolute values of the coefficients as the penalty term. This means that elastic net regression can balance the advantages and disadvantages of ridge and lasso regression by shrinking some of the coefficients towards zero and setting some of them to exactly zero.
Elastic net regression has an additional parameter called alpha, which controls the relative weight of the two penalty terms. When alpha is zero, elastic net regression is equivalent to ridge regression. When alpha is one, elastic net regression is equivalent to lasso regression. When alpha is between zero and one, elastic net regression is a ridge and lasso regression mixture.
Compare Ridge regression with other forms of Regression
Using the function, we can use the same dataset and predictors as before and fit lasso and elastic net models. We can use the alpha argument to specify the type of regularization and the lambda argument to specify the amount of regularization. We can also use the cv.glmnet function to perform crossvalidation and choose the optimal lambda values for each model.
We can then use the predict function to generate the predicted values for the test set, and calculate the MSE, the RMSE, and the Rsquared for each model. We can then compare the results and see which model performs best.
To illustrate how to compare ridge regression with other forms of regression, we will use the same dataset and predictors as before and fit lasso and elastic net models using the glmnet function. We will use the alpha argument to specify the type of regularization and the lambda argument to specify the amount of regularization.
We will also use the cv.glmnet function to perform crossvalidation and choose the optimal lambda values for each model. We will then use the predict function to generate the predicted values for the test set, and calculate the MSE, the RMSE, and the Rsquared for each model. We will then compare the results and see which model performs best.
# fit the lasso regression model model_lasso < glmnet(x_train, y_train, alpha = 1) # use alpha = 1 for lasso cv_model_lasso < cv.glmnet(x_train, y_train, alpha = 1, nfolds = 10) # perform 10fold crossvalidation y_pred_lasso < predict(model_lasso, s = cv_model_lasso$lambda.min, newx = x_test) # generate the predicted values for the test set mse_lasso < mean((y_pred_lasso  y_test)^2) # calculate the mean squared error rmse_lasso < sqrt(mse_lasso) # calculate the root mean squared error r2_lasso < 1  sum((y_pred_lasso  y_test)^2) / sum((y_test  mean(y_test))^2) # calculate the coefficient of determination # fit the elastic net regression model model_enet < glmnet(x_train, y_train, alpha = 0.5) # use alpha = 0.5 for elastic net cv_model_enet < cv.glmnet(x_train, y_train, alpha = 0.5, nfolds = 10) # perform 10fold crossvalidation y_pred_enet < predict(model_enet, s = cv_model_enet$lambda.min, newx = x_test) # generate the predicted values for the test set mse_enet < mean((y_pred_enet  y_test)^2) # calculate the mean squared error rmse_enet < sqrt(mse_enet) # calculate the root mean squared error r2_enet < 1  sum((y_pred_enet  y_test)^2) / sum((y_test  mean(y_test))^2) # calculate the coefficient of determination
print(paste("MSE for ridge:", mse)) # print the mean squared error for ridge
print(paste("MSE for lasso:", mse_lasso)) # print the mean squared error for lasso
print(paste("MSE for elastic net:", mse_enet)) # print the mean squared error for elastic net
print(paste("RMSE for ridge:", rmse)) # print the root mean squared error for ridge
print(paste("RMSE for lasso:", rmse_lasso)) # print the root mean squared error for lasso
print(paste("RMSE for elastic net:", rmse_enet)) # print the root mean squared error for elastic net
print(paste("Rsquared for ridge:", r2)) # print the coefficient of determination for ridge
print(paste("Rsquared for lasso:", r2_lasso)) # print the coefficient of determination for lasso
print(paste("Rsquared for elastic net:", r2_enet)) # print the coefficient of determination for elastic net
MSE 
RMSE 
Rsquared 

Ridge Regression 
5.156 
2.271 
0.565 
Lasso Regression 
5.306 
2.303 
0.552 
Elastic Net
Regression 
4.717 
2.172 
0.602 
We can see that the MSE, the RMSE, and the Rsquared are the same for all three models, which means that they have the same performance on the test data. This is because the optimal values of lambda for each model are very close to zero, which means that the regularization effect is very weak, and the models are very similar to the linear regression model.
It suggests that the data has no severe multicollinearity problem and that the linear regression model is already a good fit. Therefore, we do not need to use ridge, lasso, or elastic net regression for this dataset.
However, this may not be the case for other datasets, where the regularization effect may be stronger, and the performance of the models may differ significantly. Therefore, it is always a good idea to compare different forms of regression and choose the one that gives the best results.
Pros and cons
Pros
 Ridge regression can reduce the estimates' variance and make them more robust to multicollinearity.
 Ridge regression can improve the predictions' accuracy and reduce the model's complexity.
 Ridge regression can be easily implemented in R using the glmnet package, which provides functions for fitting generalized linear models with various types of regularization.
Cons
 Ridge regression can introduce some bias in the estimates, meaning they can deviate from the true values.
 Ridge regression cannot perform variable selection, meaning it cannot reduce the number of predictors in the model.
 Ridge regression requires choosing the optimal value of lambda, which can be challenging and timeconsuming.
When and why
When to use
Ridge regression can be used when the data has multicollinearity, a situation where some of the predictor variables are highly correlated. Multicollinearity can cause the ordinary least squares estimates to be unstable, have high variance, and be sensitive to small changes in the data.
Why to use
Ridge regression can help you deal with multicollinearity by adding a penalty term to the ordinary least squares objective function, which is proportional to the sum of squared coefficients of the regression model. The penalty term shrinks the coefficients towards zero, reducing the estimates' variance. This can improve the accuracy of the predictions and reduce the complexity of the model.
Conclusion
In this article, you have learned how to implement ridge regression in R, how it works, and how to compare it with other forms of regression. Ridge regression is a method of regularization that can help you deal with multicollinearity, improve the accuracy of your predictions, and reduce the complexity of your model.
Ridge regression adds a penalty term to the ordinary least squares objective function, which is proportional to the sum of squared coefficients of the regression model. The penalty term is controlled by a lambda parameter, which determines how much the coefficients are shrunk towards zero. The higher the lambda, the more the coefficients are shrunk, and the lower the lambda, the more the coefficients are similar to the OLS estimates.
To implement ridge regression in R, you need to use the glmnet package, which provides functions for fitting generalized linear models with various types of regularization. The main function for fitting ridge regression models is glmnet, which takes a matrix of predictor variables, a vector of response values, and an alpha parameter that controls the type of regularization.
The function returns an object of class glmnet, which contains information about the fitted model, such as the coefficients, the lambda values, the degrees of freedom, and the deviance. You can plot the model object using the plot function, which will show you how the coefficients change as a function of lambda.
To choose the optimal value of lambda, you need to use crossvalidation, a technique that splits the data into several subsets and uses some for training and some for testing. By repeating this process for different lambda values, you can compare the performance of the models on the test subsets and choose the value of lambda that gives the lowest test error. The glmnet package provides a function called cv.glmnet, which performs crossvalidation for ridge regression models. The function returns an object of class cv.glmnet, which contains information about the crossvalidation results, such as the lambda values, the crossvalidated errors, and the optimal lambda value. You can plot the cv_model object using the plot function, which will show you how the crossvalidated error changes as a function of lambda.
I hope you have enjoyed this article and learned something new and useful. If you have any questions or feedback, please contact me at info@rstudiodatalab.com or leave a comment below.
You can also hire me for your data analysis projects by visiting Get a Quote. If you want to learn more about data analysis and R, please subscribe to my YouTube channel, Data Analysis, and join my community groups.
Frequently Asked Questions (FAQs)
What is the difference between ridge regression and linear regression?
Linear regression is a method of fitting a linear model to the data, by minimizing the sum of squared residuals (SSR). Ridge regression is a method of fitting a linear model to the data, by minimizing the sum of squared residuals (SSR) and the sum of squared coefficients (SSC). Ridge regression adds a penalty term to the linear regression objective function, which shrinks the coefficients towards zero, and reduces the variance of the estimates.
What is the advantage of ridge regression over linear regression?
Ridge regression has an advantage over linear regression when the data has multicollinearity, a situation where some of the predictor variables are highly correlated. Multicollinearity can cause the linear regression estimates to be unstable, have high variance, and be sensitive to small changes in the data. By shrinking the coefficients, ridge regression reduces the estimates' variance and makes them more robust to multicollinearity.
What is the disadvantage of ridge regression over linear regression?
Ridge regression has a disadvantage over linear regression when the data is not multicollinear, and the linear regression model is already a good fit. Ridge regression introduces some bias in the estimates, meaning they can deviate from the true values. The bias increases as the lambda increases, and the model becomes less flexible and more prone to underfitting.
How to choose the value of lambda for ridge regression?
A common way to choose the value of lambda for ridge regression is to use crossvalidation, a technique that splits the data into several subsets and uses some for training and some for testing. By repeating this process for different lambda values, we can compare the performance of the models on the test subsets and choose the value of lambda that gives the lowest test error.
How to implement ridge regression in R?
To implement ridge regression in R, you need to use the glmnet package, which provides functions for fitting generalized linear models with various types of regularization. The main function for fitting ridge regression models is glmnet, which takes a matrix of predictor variables, a vector of response values, and an alpha parameter that controls the type of regularization. The function returns an object of class glmnet, which contains information about the fitted model, such as the coefficients, the lambda values, the degrees of freedom, and the deviance.
How to plot the ridge regression model in R?
To plot the ridge regression model in R, you can use the plot function, which takes an object of class glmnet as an argument. The function will show you how the coefficients change as a function of lambda. The xaxis is on a log scale, so the smaller lambda values are on the right, and the larger values are on the left. The yaxis shows the values of the coefficients, and each line corresponds to a different predictor variable. The vertical dotted lines indicate the values of lambda that give the minimum mean squared error (MSE) and the largest MSE within one standard error of the minimum.
How to compare ridge regression with other forms of regression?
To compare ridge regression with other forms of regression, you can use the same dataset and predictors and fit lasso and elastic net models using the glmnet function. You can use the alpha argument to specify the type of regularization and the lambda argument to specify the amount of regularization. You can also use the cv.glmnet function to perform crossvalidation and choose the optimal lambda values for each model. You can then use the predict function to generate the predicted values for the test set and calculate the MSE, the RMSE, and the Rsquared for each model. You can then compare the results and see which model performs best.
What is lasso regression?
Lasso regression is a form of regression that uses the sum of the absolute values of the coefficients as the penalty term, instead of the sum of the squared values. This means that lasso regression can perform variable selection by setting some of the coefficients to exactly zero and reducing the number of predictors in the model. Lasso regression is also known as L1 regularization because it is equivalent to the absolute values of the coefficients.
What is elastic net regression?
Elastic net regression is a combination of ridge and lasso regression, which uses a weighted sum of the squared and absolute values of the coefficients as the penalty term. This means that elastic net regression can balance the advantages and disadvantages of both ridge and lasso regression by shrinking some of the coefficients towards zero and setting some to zero. Elastic net regression has an additional parameter called alpha, which controls the relative weight of the two penalty terms. When alpha is zero, elastic net regression is equivalent to ridge regression. When alpha is one, elastic net regression is equivalent to lasso regression. When alpha is between zero and one, elastic net regression is a ridge and lasso regression mixture.
What are some applications of ridge regression?
Ridge regression can be applied to various fields and problems that involve data analysis and machine learning, such as:
 Predicting house prices based on features such as size, location, and amenities
 Classifying spam emails based on words and phrases in the text
 Recommending products to customers based on their preferences and ratings
 Analyzing gene expression data to identify biomarkers and pathways
 Estimating the effects of marketing campaigns on sales and revenue
Thank you for reading.