Spurious Valleys and Clustering Behavior of Neural Networks

Samuele Pollaci

Spurious Valleys and Clustering Behavior of Neural Networks

Research output: Chapter in Book/Report/Conference proceeding › Conference paper

Abstract

Neural networks constitute a class of functions that are typically non-surjective, with high-dimensional fibers and complicated image. We prove two main results concerning the geometry of the loss landscape of a neural network. First, we provide an explicit effective bound on the sizes of the hidden layers so that the loss landscape has no spurious valleys, which guarantees the success of gradient descent methods. Second, we present a novel method for analyzing whether a given neural network architecture with monomial activation function can represent a target function of interest. The core of our analysis method is the study of a specific set of error values, and its behavior depending on different training datasets.

Original language	English
Title of host publication	Proceedings of the 40th International Conference on Machine Learning
Editors	Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, Jonathan Scarlett
Publisher	Proceedings of Machine Learning Research (PMLR)
Number of pages	21
Volume	202
Publication status	Published - Jul 2023
Event	Fortieth International Conference on Machine Learning - Hawaiʻi Convention Center, 1801 Kalākaua Avenue , Honolulu, United States Duration: 23 Jul 2023 → 29 Jul 2023 Conference number: 40 https://icml.cc/Conferences/2023

Publication series

Name	Proceedings of Machine Learning Research
Volume	202
ISSN (Print)	2640-3498

Conference

Conference	Fortieth International Conference on Machine Learning
Abbreviated title	ICML 2023
Country/Territory	United States
City	Honolulu
Period	23/07/23 → 29/07/23
Internet address	https://icml.cc/Conferences/2023

Access to Document

Fortieth International Conference on Machine Learning
Samuele Pollaci (Participant)
23 Jul 2023 → 29 Jul 2023
Activity: Participating in or organising an event › Participation in conference

Cite this

Pollaci, S. (2023). Spurious Valleys and Clustering Behavior of Neural Networks. In A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, & J. Scarlett (Eds.), Proceedings of the 40th International Conference on Machine Learning (Vol. 202). (Proceedings of Machine Learning Research; Vol. 202). Proceedings of Machine Learning Research (PMLR). https://openreview.net/forum?id=TuHgrnPHZq

Pollaci, Samuele. / Spurious Valleys and Clustering Behavior of Neural Networks. Proceedings of the 40th International Conference on Machine Learning. editor / Andreas Krause ; Emma Brunskill ; Kyunghyun Cho ; Barbara Engelhardt ; Sivan Sabato ; Jonathan Scarlett. Vol. 202 Proceedings of Machine Learning Research (PMLR), 2023. (Proceedings of Machine Learning Research).

@inproceedings{2085f69e234a4731b04ea3219a3f48d5,

title = "Spurious Valleys and Clustering Behavior of Neural Networks",

abstract = "Neural networks constitute a class of functions that are typically non-surjective, with high-dimensional fibers and complicated image. We prove two main results concerning the geometry of the loss landscape of a neural network. First, we provide an explicit effective bound on the sizes of the hidden layers so that the loss landscape has no spurious valleys, which guarantees the success of gradient descent methods. Second, we present a novel method for analyzing whether a given neural network architecture with monomial activation function can represent a target function of interest. The core of our analysis method is the study of a specific set of error values, and its behavior depending on different training datasets.",

author = "Samuele Pollaci",

year = "2023",

month = jul,

language = "English",

volume = "202",

series = "Proceedings of Machine Learning Research",

publisher = "Proceedings of Machine Learning Research (PMLR)",

editor = "{ Krause}, Andreas and Brunskill, {Emma } and Cho, {Kyunghyun } and Engelhardt, {Barbara } and Sivan Sabato and { Scarlett}, Jonathan",

booktitle = "Proceedings of the 40th International Conference on Machine Learning",

note = "Fortieth International Conference on Machine Learning, ICML 2023 ; Conference date: 23-07-2023 Through 29-07-2023",

url = "https://icml.cc/Conferences/2023",

}

Pollaci, S 2023, Spurious Valleys and Clustering Behavior of Neural Networks. in A Krause, E Brunskill, K Cho, B Engelhardt, S Sabato & J Scarlett (eds), Proceedings of the 40th International Conference on Machine Learning. vol. 202, Proceedings of Machine Learning Research, vol. 202, Proceedings of Machine Learning Research (PMLR), Fortieth International Conference on Machine Learning, Honolulu, Hawaii, United States, 23/07/23. <https://openreview.net/forum?id=TuHgrnPHZq>

Spurious Valleys and Clustering Behavior of Neural Networks. / Pollaci, Samuele.

Proceedings of the 40th International Conference on Machine Learning. ed. / Andreas Krause; Emma Brunskill; Kyunghyun Cho; Barbara Engelhardt; Sivan Sabato; Jonathan Scarlett. Vol. 202 Proceedings of Machine Learning Research (PMLR), 2023. (Proceedings of Machine Learning Research; Vol. 202).

Research output: Chapter in Book/Report/Conference proceeding › Conference paper

TY - GEN

T1 - Spurious Valleys and Clustering Behavior of Neural Networks

AU - Pollaci, Samuele

N1 - Conference code: 40

PY - 2023/7

Y1 - 2023/7

N2 - Neural networks constitute a class of functions that are typically non-surjective, with high-dimensional fibers and complicated image. We prove two main results concerning the geometry of the loss landscape of a neural network. First, we provide an explicit effective bound on the sizes of the hidden layers so that the loss landscape has no spurious valleys, which guarantees the success of gradient descent methods. Second, we present a novel method for analyzing whether a given neural network architecture with monomial activation function can represent a target function of interest. The core of our analysis method is the study of a specific set of error values, and its behavior depending on different training datasets.

AB - Neural networks constitute a class of functions that are typically non-surjective, with high-dimensional fibers and complicated image. We prove two main results concerning the geometry of the loss landscape of a neural network. First, we provide an explicit effective bound on the sizes of the hidden layers so that the loss landscape has no spurious valleys, which guarantees the success of gradient descent methods. Second, we present a novel method for analyzing whether a given neural network architecture with monomial activation function can represent a target function of interest. The core of our analysis method is the study of a specific set of error values, and its behavior depending on different training datasets.

M3 - Conference paper

VL - 202

T3 - Proceedings of Machine Learning Research

BT - Proceedings of the 40th International Conference on Machine Learning

A2 - Krause, Andreas

A2 - Brunskill, Emma

A2 - Cho, Kyunghyun

A2 - Engelhardt, Barbara

A2 - Sabato, Sivan

A2 - Scarlett, Jonathan

PB - Proceedings of Machine Learning Research (PMLR)

T2 - Fortieth International Conference on Machine Learning

Y2 - 23 July 2023 through 29 July 2023

ER -

Spurious Valleys and Clustering Behavior of Neural Networks

Abstract

Publication series

Conference

Access to Document

Fingerprint

Activities

Fortieth International Conference on Machine Learning

Cite this