Are Straight-Through gradients and Soft-Thresholding all you need for Sparse Training?

Vanderschueren, Antoine;De Vleeschouwer, Christophe
(2023) IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) — Location: Waikoloa, HI, USA (4.January.2023)

Files

1614-cameraready.pdf
  • Open Access
  • Adobe PDF
  • 1.18 MB
Download

Details

Authors
Abstract
Turning the weights to zero when training a neural network helps in reducing the computational complexity at inference. To progressively increase the sparsity ratio in the network without causing sharp weight discontinuities during training, our work combines soft-thresholding and straight-through gradient estimation to update the raw, i.e. non-thresholded, version of zeroed weights. Our method, named ST-3 for straight-through/soft-thresholding/sparsetraining2, obtains SoA results, both in terms of accuracy/ sparsity and accuracy/FLOPS trade-offs, when progressively increasing the sparsity ratio in a single training cycle. In particular, despite its simplicity, ST-3 favorably compares to the most recent methods, adopting differentiable formulations [42] or bio-inspired neuroregeneration principles [25]. This suggests that the key ingredients for effective sparsification primarily lie in the ability to give the weights the freedom to evolve smoothly across the zero state while progressively increasing the sparsity ratio.
Affiliations

Citations

  • APA
  • Chicago
  • FWB

Vanderschueren, A., & De Vleeschouwer, C. (2023). Are Straight-Through gradients and Soft-Thresholding all you need for Sparse Training? Proceedings of IEEE/CVF Winter Conference on Applications of Computer Vision, 1(1), 1-10. https://hdl.handle.net/2078.5/105916 (Original work published 2022)