Natural Regularization from Generative Models

Generative models describe how data might have been produced. Support vector machines take a different route: they learn a boundary that separates examples. The two approaches were usually treated separately. This work asked whether the structure learned by one could guide the other.

The proposed natural kernels used an estimated probability distribution to define which changes in a classifier should count as small, instead of relying only on distances in the original feature space. The Fisher kernel was one example. Seen through regularization, these kernels favor functions that vary smoothly in ways suggested by the data-generating model.

The result was a family of kernels, not an end-user system. And it came with a plain limitation: everything depends on whether the generative model captures useful structure. A poor model can provide the wrong notion of similarity. The analysis also helped explain what these kernels regularize and how their spectral properties could inform model selection.

Publications

@incollection{oliver2000naturalsvms,
  author = {Nuria Oliver and Bernhard Sch{\"o}lkopf and Alexander J. Smola},
  title = {Natural Regularization in {SVMs}},
  booktitle = {Advances in Large Margin Classifiers},
  pages = {51--60},
  year = 2000,
  publisher = {MIT Press},
  cites = 26
}

@incollection{oliver2000naturalgen,
  author = {Nuria Oliver and Bernhard Sch{\"o}lkopf and Alexander J. Smola},
  title = {Natural Regularization from Generative Models},
  booktitle = {Advances in Large-Margin Classifiers},
  pages = {51--60},
  year = 2000,
  doi = {10.7551/mitpress/1113.003.0007},
  publisher = {MIT Press},
  cites = 9,
  citesdate = {2026-03-23}
}