Statistics How To

Tikhonov Regularization: Simple Definition

Statistics Definitions >

What is Tikhonov Regularization?

Tikhonov Regularization (sometimes called Tikhonov-Phillips regularization) is a popular way to deal with linear discrete ill-posed problems. Ill-posed problems violate one of the terms of a well-posed problem. Well posed problems:

  • Have a solution.
  • Have a solution that is unique.
  • Have a solution that changes as the initial condition changes.

Regularization stabilizes ill­-posed problems, giving accurate approximate solutions — often by including prior information (Vogel).

For example, let’s say you wanted to find a vector x so that Ax = b. You could use ordinary least squares to find a solution, but you may find that no solutions exist, or multiple solutions exist. Tikhonov Regularization can give you a meaningful, approximate solution to this ill-posed problem.

Tikhivov’s method can produce solutions even when the data set contains a lot of statistical noise. It is essentially the same technique as ridge regression; The main difference is that Tikhonov’s has a larger set.

The Method

Tikhobv’s method uses the following problem in place of the problem that calculates the minimum-norm least squares:
tikhonov regularization

Where λ is the regularization parameter, which specifies the amount of regularization. λ essentially acts as a Lagrange multiplier, in that you are solving a minimization problem with ‖x‖ = R acting as a constraint for some R.
The basic idea is to make:

as small as possible (without minimizing it), without:

becoming too big.

More formally, the definition (from Kaipio) is:

Let λ > 0 be a given constant. The Tikhonov regularized
solution xλ ∈ H1 is

(the minimizer of the given function) provided that a minimizer exists.

The proof is beyond the scope of this site, but you can find an excellent outline in JP Kaipo’s Classical Regularization Methods, available as a pdf here.

Choosing λ

Choosing a value for λ can be challenging. Several methods exist for calculating a suitable values including L-curve methods and the Morozov discrepancy principle (discussed in Kaipo).

Alessandro Buccini. Regularizing preconditioners by non-stationary iterated Tikhonov with general penalty term.
Mark Gockenbach. Linear Inverse Problems and Regularization.
Curt Vogel. Regularization Methods.
JP Kaipio. “Classical Regularization Methods.” Statistical and Computational Inverse Problems. Volume 160 of the series Applied Mathematical Sciences pp 7-48.


Need help with a homework or test question? With Chegg Study, you can get step-by-step solutions to your questions from an expert in the field. If you rather get 1:1 study help, Chegg Tutors offers 30 minutes of free tutoring to new users, so you can try them out before committing to a subscription.

If you prefer an online interactive environment to learn R and statistics, this free R Tutorial by Datacamp is a great way to get started. If you're are somewhat comfortable with R and are interested in going deeper into Statistics, try this Statistics with R track.

Comments? Need to post a correction? Please post a comment on our Facebook page.

Check out our updated Privacy policy and Cookie Policy

Tikhonov Regularization: Simple Definition was last modified: October 12th, 2017 by Stephanie