Consistency and Robustness Properties of SVMs for Heavy-Tailed Distributions

Arnout Van Messem

Research output: Chapter in Book/Report/Conference proceedingMeeting abstract (Book)


Recently some results on consistency and robustness of support
vector machines for nonparametric classification and nonparametric
regression were derived for non-negative convex loss functions $L$
of Nemitski type, see Christmann and Steinwart (2007) [3] and
Steinwart and Christmann (2008) [15], given that a weak moment
condition for the joint distribution $\P$ on $X\times Y$ is valid.
However, this condition excludes heavy-tailed distributions (e.g.,
Cauchy distribution) or several extreme value distributions which
can occur in financial or actuarial problems.

We will weaken the above condition on $\P$ to only a condition on
the marginal distribution $\P_X$, such that heavy-tailed conditional
distributions are also covered. As was already used by Huber (1967)
in a different setting, the idea is to shift the loss function $L:
X\times Y\times \R\to [0,\infty)$ by some function which is
independent of the estimator. More precisely, we define the shifted
loss function $\Ls(x,y,f(x)) := L(x,y,f(x)) - L(x,y,0)$. It is
obvious that the ``loss'' $\Ls(x,y,f(x))$ can be negative. Let $\fP$
be the decision function of the original support vector machine. We
\fPLs = \arg \inf_{f \in \mathcal{H}} \Ex_\P \Ls(X,Y,f(X)) + \lambda \| f \|_{\mathcal{H}}^2 \, ,
where $\mathcal{H}$ is the reproducing kernel Hilbert space of a
measurable kernel $k:X \times X \to \R$ and $\lambda>0$ is some
regularization parameter.

We will first discuss the used ``$\Ls$-trick'', cite some properties
of the new loss $\Ls$, and show that $\fPLs$ exists and is unique.
Furthermore, this function will coincide with $\fP$ if the latter
exists. We then give a representer theorem and results on both
risk-consistency as well as consistency of the solution. We will
also show that $\fPLs$ is robust in the sense of influence functions
if the kernel is bounded and if the loss function $L$ is Lipschitz
continuous. This result holds true for Hampel's influence function
and also for the Bouligand influence function proposed by Christmann
and Van Messem (2008) [5].
Original languageEnglish
Title of host publication17th Annual Meeting of the Belgian Statistical Society, Oct 14-16, 2009, Lommel (Belgium)
Publication statusPublished - 14 Oct 2009

Publication series

Name17th Annual Meeting of the Belgian Statistical Society, Oct 14-16, 2009, Lommel (Belgium)


  • SVM
  • heavy tails
  • robustness
  • consistency
  • Bouligand


Dive into the research topics of 'Consistency and Robustness Properties of SVMs for Heavy-Tailed Distributions'. Together they form a unique fingerprint.

Cite this