Loss Functionals¶

Module: pycsou.func.loss

Repository of common loss functionals.

Distances

`L2Loss`(dim, data)	\(\ell_2\) loss functional, \(F(\mathbf{y},\mathbf{x}):=\\|\mathbf{y}-\mathbf{x}\\|_2\).
`SquaredL2Loss`(dim, data)	\(\ell^2_2\) loss functional, \(F(\mathbf{y},\mathbf{x}):=\\|\mathbf{y}-\mathbf{x}\\|^2_2\).
`L1Loss`(dim, data)	\(\ell_1\) loss functional, \(F(\mathbf{y},\mathbf{x}):=\\|\mathbf{y}-\mathbf{x}\\|_1\).
`SquaredL1Loss`(dim, data[, prox_computation])	\(\ell^2_1\) loss functional, \(F(\mathbf{y},\mathbf{x}):=\\|\mathbf{y}-\mathbf{x}\\|^2_1\).
`LInftyLoss`(dim, data)	\(\ell_\infty\) loss functional, \(F(\mathbf{y},\mathbf{x}):=\\|\mathbf{y}-\mathbf{x}\\|_\infty\).

Indicators

`L2BallLoss`(dim, data[, radius])	\(\ell_2\)-ball loss functional, \(\{\mathbf{x}\in\mathbb{R}^N: \\|\mathbf{y}-\mathbf{x}\\|_2\leq \text{radius}\}\).
`L1BallLoss`(dim, data[, radius])	\(\ell_1\)-ball loss functional, \(\{\mathbf{x}\in\mathbb{R}^N: \\|\mathbf{y}-\mathbf{x}\\|_1\leq \text{radius}\}\).
`LInftyBallLoss`(dim, data[, radius])	\(\ell_\infty\)-ball loss functional, \(\{\mathbf{x}\in\mathbb{R}^N: \\|\mathbf{y}-\mathbf{x}\\|_\infty\leq \text{radius}\}\).
`ConsistencyLoss`(dim, data)	Consistency loss functional \(\mathbf{y}=\mathbf{x}\).

Divergences

KLDivergence(dim, data)

Generalised Kullback-Leibler divergence \(D_{KL}(\mathbf{y}||\mathbf{x}):=\sum_{i=1}^N y_i\log(y_i/x_i) -y_i +z_i\).

Classes

`ProximableLoss`(func, data)	Constructor of proximable loss functions.
`DifferentiableLoss`(func, data)	Constructor of proximable loss functions.

ProximableLoss(func: pycsou.core.functional.ProximableFunctional, data: Union[numbers.Number, numpy.ndarray]) → pycsou.core.functional.ProximableFunctional[source]¶

Constructor of proximable loss functions.

Constructs a proximable loss from a proximable functional and a data vector. Let \(\varphi:\mathbb{R}^N\rightarrow \mathbb{R}\) be some proximable functional and \(\mathbf{y}\in\mathbb{R}^N\). This routine defines the loss functional \(F(\mathbf{x}; \mathbf{y}):= \varphi(\mathbf{x}-\mathbf{y}), \,\forall \mathbf{x}\in\mathbb{R}^N.\)

Parameters

func (ProximableFunctional) – Some proximable functional \(\varphi:\mathbb{R}^N\rightarrow \mathbb{R}\).
data (Union[Number, np.ndarray]) – Data vector \(\mathbf{y}\in\mathbb{R}^N\).

Returns

Proximable loss functional constructed as \(F(\mathbf{x}; \mathbf{y}):= \varphi(\mathbf{x}-\mathbf{y}), \,\forall \mathbf{x}\in\mathbb{R}^N.\)

Return type

ProximableFunctional

Examples

>>> y = np.arange(10)
>>> func = L1Norm(dim=y.size)
>>> loss = ProximableLoss(func=func, data=y)
>>> x = 2 * np.arange(10)
>>> np.allclose(loss(x), func(x-y))
True
>>> np.allclose(loss.prox(x, tau=1), func.prox(x-y, tau=1) + y)
True

Notes

The proximity operator of the loss functional is automatically computed from the one of the input functional \(\varphi\) using properties described in [ProxAlg] Section 2.1.

See also

DifferentiableLoss().

DifferentiableLoss(func: pycsou.core.functional.DifferentiableFunctional, data: Union[numbers.Number, numpy.ndarray]) → pycsou.core.functional.DifferentiableFunctional[source]¶

Constructor of proximable loss functions.

Constructs a differentiable loss from a differentiable functional and a data vector. Let \(\varphi:\mathbb{R}^N\rightarrow \mathbb{R}\) be some differentiable functional and \(\mathbf{y}\in\mathbb{R}^N\). This routine defines the loss functional \(F(\mathbf{x}; \mathbf{y}):= \varphi(\mathbf{x}-\mathbf{y}), \,\forall \mathbf{x}\in\mathbb{R}^N.\)

Parameters

func (DifferentiableFunctional) – Some differentiable functional \(\varphi:\mathbb{R}^N\rightarrow \mathbb{R}\).
data (Union[Number, np.ndarray]) – Data vector \(\mathbf{y}\in\mathbb{R}^N\).

Returns

Differentiable loss functional constructed as \(F(\mathbf{x}; \mathbf{y}):= \varphi(\mathbf{x}-\mathbf{y}), \,\forall \mathbf{x}\in\mathbb{R}^N.\)

Return type

DifferentiableFunctional

Examples

>>> y = np.arange(10)
>>> func = SquaredL2Norm(dim=y.size)
>>> loss = DifferentiableLoss(func=func, data=y)
>>> x = 2 * np.arange(10)
>>> np.allclose(loss(x), func(x-y))
True
>>> np.allclose(loss.gradient(x), 2*(x-y))
True

Notes

The derivative and Lipschitz constant of the loss functional are automatically computed from those of the input functional \(\varphi\).

See also

ProximableLoss().

L2Loss(dim: int, data: Union[numbers.Number, numpy.ndarray]) → pycsou.core.functional.ProximableFunctional[source]¶

\(\ell_2\) loss functional, \(F(\mathbf{y},\mathbf{x}):=\|\mathbf{y}-\mathbf{x}\|_2\).

Parameters

dim (int) – Dimension of the domain.
data (Union[Number, np.ndarray]) – Data vector \(\mathbf{y}\).

Returns

The \(\ell_2\) loss functional.

Return type

ProximableFunctional

Examples

>>> y = np.arange(10)
>>> loss = L2Loss(dim=y.size, data=y)
>>> func = L2Norm(dim=y.size)
>>> x = 2 * np.arange(10)
>>> np.allclose(loss.prox(x, tau=1), func.prox(x-y, tau=1) + y)
True

See also

L2Norm(), L1Loss(), LInftyLoss()

SquaredL2Loss(dim: int, data: Union[numbers.Number, numpy.ndarray]) → pycsou.core.functional.DifferentiableFunctional[source]¶

\(\ell^2_2\) loss functional, \(F(\mathbf{y},\mathbf{x}):=\|\mathbf{y}-\mathbf{x}\|^2_2\).

Parameters

dim (int) – Dimension of the domain.
data (Union[Number, np.ndarray]) – Data vector \(\mathbf{y}\).

Returns

The \(\ell^2_2\) loss functional.

Return type

DifferentiableFunctional

Examples

>>> y = np.arange(10)
>>> loss = SquaredL2Loss(dim=y.size, data=y)
>>> Gmat = np.arange(100).reshape(10, 10).astype(float)
>>> G = DenseLinearOperator(Gmat, is_symmetric=False)
>>> G.compute_lipschitz_cst()
>>> fwd_loss = loss * G
>>> x = 2 * np.arange(10)
>>> np.allclose(loss(x), np.linalg.norm(y - x) ** 2)
True
>>> np.allclose(fwd_loss(x), loss(G(x)))
True
>>> np.allclose(fwd_loss.diff_lipschitz_cst, 2 * (G.lipschitz_cst ** 2))
True
>>> np.allclose(fwd_loss.gradient(x), 2 * G.adjoint(G(x) - y))
True

Notes

The \(\ell_2^2\) functional is the likelihood of the data \(\mathbf{y}\) under the assumtpion of Gaussian white noise.

See also

SquaredL2Norm(), L2Loss().

L2BallLoss(dim: int, data: Union[numbers.Number, numpy.ndarray], radius: numbers.Number = 1) → pycsou.core.functional.ProximableFunctional[source]¶

\(\ell_2\)-ball loss functional, \(\{\mathbf{x}\in\mathbb{R}^N: \|\mathbf{y}-\mathbf{x}\|_2\leq \text{radius}\}\).

The \(\ell_2\)-ball loss functional is defined as:

\[\begin{split}\iota(\mathbf{x}):=\begin{cases} 0 \,\text{if} \,\|\mathbf{x}-\mathbf{y}\|_2\leq \text{radius},\\ \, 0\,\text{ortherwise}. \end{cases}\end{split}\]

Parameters

dim (int) – Dimension of the domain.
data (Union[Number, np.ndarray]) – Data vector \(\mathbf{y}\).
radius (Number) – Radius of the ball.

Returns

The \(\ell_2\)-ball loss functional.

Return type

ProximableFunctional

Examples

>>> y = np.arange(10)
>>> loss = L2BallLoss(dim=y.size, data=y, radius=2)
>>> func = L2Ball(dim=y.size, radius=2)
>>> x = 2 * np.arange(10)
>>> np.allclose(loss.prox(x, tau=1), func.prox(x-y, tau=1) + y)
True

Notes

The \(\ell_2\)-ball loss functional is particularly useful in the context of Gaussian white noise with known standard deviation. In which case, the \(\ell_2\)-ball defines a confidence region for the data \(\mathbf{y}\) ([FuncSphere] Section 5 of Chapter 7).

See also

L2Ball(), L2Loss().

L1Loss(dim: int, data: Union[numbers.Number, numpy.ndarray]) → pycsou.core.functional.ProximableFunctional[source]¶

\(\ell_1\) loss functional, \(F(\mathbf{y},\mathbf{x}):=\|\mathbf{y}-\mathbf{x}\|_1\).

Parameters

dim (int) – Dimension of the domain.
data (Union[Number, np.ndarray]) – Data vector \(\mathbf{y}\).

Returns

The \(\ell_1\) loss functional.

Return type

ProximableFunctional

Examples

>>> y = np.arange(10)
>>> loss = L1Loss(dim=y.size, data=y)
>>> func = L1Norm(dim=y.size)
>>> x = 2 * np.arange(10)
>>> np.allclose(loss.prox(x, tau=1), func.prox(x-y, tau=1) + y)
True

Notes

The \(\ell_1\) loss functional leads to sparse residuals, with most of the predicted samples matching exactly the observed samples, and a few –potentially large– misfits ([FuncSphere] Section 5 of Chapter 7). Such a functional is particularly useful in the context of salt-and-pepper noise with strong outliers, or more generally for noise distributions with heavy tails, templated by the Laplace distribution.

See also

L1Norm(), SquaredL1Loss().

SquaredL1Loss(dim: int, data: Union[numbers.Number, numpy.ndarray], prox_computation='sort') → pycsou.core.functional.ProximableFunctional[source]¶

\(\ell^2_1\) loss functional, \(F(\mathbf{y},\mathbf{x}):=\|\mathbf{y}-\mathbf{x}\|^2_1\).

Parameters

dim (int) – Dimension of the domain.
data (Union[Number, np.ndarray]) – Data vector \(\mathbf{y}\).

Returns

The \(\ell^2_1\) loss functional.

Return type

ProximableFunctional

Examples

>>> y = np.arange(10)
>>> loss = SquaredL1Loss(dim=y.size, data=y)
>>> func = SquaredL1Norm(dim=y.size)
>>> x = 2 * np.arange(10)
>>> np.allclose(loss.prox(x, tau=1), func.prox(x-y, tau=1) + y)
True

See also

SquaredL1Norm(), L1Loss().

L1BallLoss(dim: int, data: Union[numbers.Number, numpy.ndarray], radius: numbers.Number = 1) → pycsou.core.functional.ProximableFunctional[source]¶

\(\ell_1\)-ball loss functional, \(\{\mathbf{x}\in\mathbb{R}^N: \|\mathbf{y}-\mathbf{x}\|_1\leq \text{radius}\}\).

The \(\ell_1\)-ball loss functional is defined as:

\[\begin{split}\iota(\mathbf{x}):=\begin{cases} 0 \,\text{if} \,\|\mathbf{x}-\mathbf{y}\|_1\leq \text{radius},\\ \, 0\,\text{ortherwise}. \end{cases}\end{split}\]

Parameters

dim (int) – Dimension of the domain.
data (Union[Number, np.ndarray]) – Data vector \(\mathbf{y}\).
radius (Number) – Radius of the ball.

Returns

The \(\ell_1\)-ball loss functional.

Return type

ProximableFunctional

Examples

>>> y = np.arange(10)
>>> loss = L1BallLoss(dim=y.size, data=y, radius=2)
>>> func = L1Ball(dim=y.size, radius=2)
>>> x = 2 * np.arange(10)
>>> np.allclose(loss.prox(x, tau=1), func.prox(x-y, tau=1) + y)
True

Notes

The \(\ell_1\)-ball loss functional is particularly useful in the context of salt-and-pepper noise with known standard deviation. In which case, the \(\ell_1\)-ball defines a confidence region for the data \(\mathbf{y}\).

LInftyLoss(dim: int, data: Union[numbers.Number, numpy.ndarray]) → pycsou.core.functional.ProximableFunctional[source]¶

\(\ell_\infty\) loss functional, \(F(\mathbf{y},\mathbf{x}):=\|\mathbf{y}-\mathbf{x}\|_\infty\).

Parameters

dim (int) – Dimension of the domain.
data (Union[Number, np.ndarray]) – Data vector \(\mathbf{y}\).

Returns

The \(\ell_\infty\) loss functional.

Return type

ProximableFunctional

Examples

>>> y = np.arange(10)
>>> loss = LInftyLoss(dim=y.size, data=y)
>>> func = LInftyNorm(dim=y.size)
>>> x = 2 * np.arange(10)
>>> loss(x)
9
>>> np.allclose(loss.prox(x, tau=1), func.prox(x-y, tau=1) + y)
True

Notes

The \(\ell_\infty\) loss functional is particularly useful in the context of quantisation noise, or more generally for noise distributions with compact support.

See also

LInftyNorm(), LInftyBallLoss().

LInftyBallLoss(dim: int, data: Union[numbers.Number, numpy.ndarray], radius: numbers.Number = 1) → pycsou.core.functional.ProximableFunctional[source]¶

\(\ell_\infty\)-ball loss functional, \(\{\mathbf{x}\in\mathbb{R}^N: \|\mathbf{y}-\mathbf{x}\|_\infty\leq \text{radius}\}\).

The \(\ell_1\)-ball loss functional is defined as:

\[\begin{split}\iota(\mathbf{x}):=\begin{cases} 0 \,\text{if} \,\|\mathbf{x}-\mathbf{y}\|_\infty\leq \text{radius},\\ \, 0\,\text{ortherwise}. \end{cases}\end{split}\]

Parameters

dim (int) – Dimension of the domain.
data (Union[Number, np.ndarray]) – Data vector \(\mathbf{y}\).
radius (Number) – Radius of the ball.

Returns

The \(\ell_\infty\)-ball loss functional.

Return type

ProximableFunctional

Examples

>>> y = np.arange(10)
>>> loss = LInftyBallLoss(dim=y.size, data=y, radius=2)
>>> func = LInftyBall(dim=y.size, radius=2)
>>> x = 2 * np.arange(10)
>>> np.allclose(loss.prox(x, tau=1), func.prox(x-y, tau=1) + y)
True

Notes

The \(\ell_\infty\)-ball loss functional is particularly useful in the context of quantisation noise with compact support. In which case, the \(\ell_\infty\)-ball defines a confidence region for the data \(\mathbf{y}\).

ConsistencyLoss(dim: int, data: Union[numbers.Number, numpy.ndarray])[source]¶

Consistency loss functional \(\mathbf{y}=\mathbf{x}\).

The consistency loss functional is defined as:

\[\begin{split}\iota(\mathbf{x}):=\begin{cases} 0 \,\text{if} \,\mathbf{x}=\mathbf{y},\\ \, 0\,\text{ortherwise}. \end{cases}\end{split}\]

Parameters

dim (int) – Dimension of the domain.
data (Union[Number, np.ndarray]) – Data vector \(\mathbf{y}\) to match.

Returns

The consistency loss functional.

Return type

ProximableFunctional

Examples

>>> y = np.arange(10)
>>> loss = ConsistencyLoss(dim=y.size, data=y)
>>> x = 2 * np.arange(10)
>>> loss(x), loss(y)
(inf, 0)
>>> np.allclose(loss.prox(x, tau=1), y)
True

Notes

This functional enforces an exact match between the predicted and observed samples, as required in interpolation problems. Such a functional is mainly useful in the context of noiseless data as it can lead to serious overfitting issues in the presence of noise.

class KLDivergence(dim: int, data: Union[numbers.Number, numpy.ndarray])[source]¶

Bases: pycsou.core.functional.ProximableFunctional

Generalised Kullback-Leibler divergence \(D_{KL}(\mathbf{y}||\mathbf{x}):=\sum_{i=1}^N y_i\log(y_i/x_i) -y_i +z_i\).

The generalised Kullback-Leibler divergence is defined as:

\[D_{KL}(\mathbf{y}||\mathbf{x}):=\sum_{i=1}^N H(y_i,x_i) -y_i +z_i, \quad \forall \mathbf{y}, \mathbf{x} \in \mathbb{R}^N,\]

where

\[\begin{split}H(y,x):=\begin{cases} y\log(y/x) &\, \text{if} \,x>0, y>0,\\ 0&\, \text{if} \,x=0, y\geq 0,\\ +\infty &\,\text{otherwise.} \end{cases}\end{split}\]

Parameters

dim (int) – Dimension of the domain.
data (Union[Number, np.ndarray]) – Data vector \(\mathbf{y}\) to match.

Returns

The KL-divergence.

Return type

ProximableFunctional

Examples

>>> y = np.arange(10)
>>> loss = KLDivergence(dim=y.size, data=y)
>>> x = 2 * np.arange(10)
>>> loss(x)
13.80837687480246
>>> np.round(loss.prox(x, tau=1))
array([ 0.,  2.,  4.,  6.,  8., 10., 12., 14., 16., 18.])

Notes

In information theory, and in the case where \(\mathbf{y}\) and \(\mathbf{x}\) sum to one –and hence can be interpreted as discrete probability distributions, the KL-divergence can be interpreted as the relative entropy of \(\mathbf{y}\) w.r.t. \(\mathbf{x}\), i.e. the amount of information lost when using \(\mathbf{x}\) to approximate \(\mathbf{y}\). It is particularly useful in the context of count data with Poisson distribution. Indeed, the KL-divergence corresponds –up to an additive constant– to the likelihood of the data \(\mathbf{y}\) where each component is independent with Poisson distribution and respective intensities given by the entries of \(\mathbf{x}\). See [FuncSphere] Section 5 of Chapter 7 for the computation of its proximal operator.

See also

ShannonEntropy, LogBarrier

__init__(dim: int, data: Union[numbers.Number, numpy.ndarray])[source]¶

Parameters

dim (int,) – Dimension of the functional’s domain.
data (Union[None, Number, np.ndarray]) – Optional data vector.
is_linear (bool) – Whether the functional is linear or not.
is_differentiable (bool) – Whether the functional is differentiable or not.

__call__(x: Union[numbers.Number, numpy.ndarray]) → numbers.Number[source]¶

Call self as a function.

Parameters: arg (Union[Number, np.ndarray]) – Argument of the map.
Returns: Value of arg through the map.
Return type: Union[Number, np.ndarray]

prox(x: Union[numbers.Number, numpy.ndarray], tau: numbers.Number) → Union[numbers.Number, numpy.ndarray][source]¶

Proximal operator of the KL-divergence functional (see [FuncSphere] Section 5 of Chapter 7).

Parameters

x (Union[Number, np.ndarray]) – Input.
tau (Number) – Scaling constant.

Returns

Proximal point of x.

Return type

Union[Number, np.ndarray]