Natural Regularization from Generative Models

Generative models describe how data might have been produced. Support vector machines take a different route: they learn a boundary that separates examples. The two approaches were usually treated separately. We asked whether the structure learned by one could guide the other.

Our natural kernels used an estimated probability distribution to define which changes in a classifier should count as small, instead of relying only on distances in the original feature space. The Fisher kernel was one example. Seen through regularization, these kernels favor functions that vary smoothly in ways suggested by the data-generating model.

We derived a family of kernels, not an end-user system. The approach came with a plain limitation: everything depends on whether the generative model captures useful structure. A poor model can provide the wrong notion of similarity. Our analysis also helped explain what these kernels regularize and how their spectral properties could inform model selection.

Publications

@incollection{oliver2000naturalgen,
  author = {Nuria Oliver and Bernhard Sch{\"o}lkopf and Alexander J. Smola},
  title = {Natural Regularization from Generative Models},
  booktitle = {Advances in Large-Margin Classifiers},
  pages = {51--60},
  year = 2000,
  doi = {10.7551/mitpress/1113.003.0007},
  publisher = {MIT Press},
  cites = 9,
  citesdate = {2026-03-23}
}