Intraclass clustering: an implicit learning ability that regularizes DNNs

Carbonnelle, Simon;De Vleeschouwer, Christophe
(2021) International Conference on learning representations — Location: Virtual (4.May.2021)

Files

intraclass_clustering_an_implicit_learning_ability_that_regularizes_dnns.pdf
  • Open Access
  • Adobe PDF
  • 2.02 MB
Download

Details

Authors
Abstract
Several works have shown that the regularization mechanisms underlying deep neural networks' generalization performances are still poorly understood. In this paper, we hypothesize that deep neural networks are regularized through their ability to extract meaningful clusters among the samples of a class. Since no explicit training mechanisms or supervision target such behaviour, this learning ability constitutes an implicit form of regularization. To support our hypothesis, we design four different measures of intraclass clustering, based on the neuron- and layer-level representations of the training data. We then show that these measures constitute accurate predictors of generalization performance across variations of a large set of hyperparameters (learning rate, batch size, optimizer, weight decay, dropout rate, data augmentation, network depth and width).
Affiliations

Citations

  • APA
  • Chicago
  • FWB

Carbonnelle, S., & De Vleeschouwer, C. (2021). Intraclass clustering: an implicit learning ability that regularizes DNNs. Proceedings of ICLR. Published. International Conference on learning representations, Virtual. https://hdl.handle.net/2078.5/224760