Spurious Valleys and Clustering Behavior of Neural Networks

Samuele Pollaci

Spurious Valleys and Clustering Behavior of Neural Networks

Onderzoeksoutput: Conference paper

Samenvatting

Neural networks constitute a class of functions that are typically non-surjective, with high-dimensional fibers and complicated image. We prove two main results concerning the geometry of the loss landscape of a neural network. First, we provide an explicit effective bound on the sizes of the hidden layers so that the loss landscape has no spurious valleys, which guarantees the success of gradient descent methods. Second, we present a novel method for analyzing whether a given neural network architecture with monomial activation function can represent a target function of interest. The core of our analysis method is the study of a specific set of error values, and its behavior depending on different training datasets.

Originele taal-2	English
Titel	Proceedings of the 40th International Conference on Machine Learning
Redacteuren	Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, Jonathan Scarlett
Uitgeverij	Proceedings of Machine Learning Research (PMLR)
Pagina's	28079-28099
Aantal pagina's	21
Volume	202
Status	Published - jul. 2023
Evenement	Fortieth International Conference on Machine Learning - Hawaiʻi Convention Center, 1801 Kalākaua Avenue , Honolulu, United States Duur: 23 jul. 2023 → 29 jul. 2023 Congresnummer: 40 https://icml.cc/Conferences/2023

Publicatie series

Naam	Proceedings of Machine Learning Research
ISSN van elektronische versie	2640-3498

Conference

Conference	Fortieth International Conference on Machine Learning
Verkorte titel	ICML 2023
Land/Regio	United States
Stad	Honolulu
Periode	23/07/23 → 29/07/23
Internet adres	https://icml.cc/Conferences/2023

Bibliografische nota

Publisher Copyright:
© 2023 Proceedings of Machine Learning Research. All rights reserved.

Toegang tot document

Handle.net

Permanente koppeling

Andere bestanden en links

Link naar publicatie in Scopus

Spurious Valleys and Clustering Behavior of Neural Networks
Pollaci, S., 21 sep. 2023, (Unpublished).
Onderzoeksoutput: Poster

Spurious Valleys and Clustering Behavior of Neural Networks
Pollaci, S. (Speaker)
26 jul. 2023
Activiteit: Talk or presentation at a conference
Fortieth International Conference on Machine Learning
Pollaci, S. (Participant)
23 jul. 2023 → 29 jul. 2023
Activiteit: Participation in conference

Citeer dit

Pollaci, S. (2023). Spurious Valleys and Clustering Behavior of Neural Networks. In A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, & J. Scarlett (editors), Proceedings of the 40th International Conference on Machine Learning (Vol. 202, blz. 28079-28099). (Proceedings of Machine Learning Research). Proceedings of Machine Learning Research (PMLR). https://openreview.net/forum?id=TuHgrnPHZq

Pollaci, Samuele. / Spurious Valleys and Clustering Behavior of Neural Networks. Proceedings of the 40th International Conference on Machine Learning. redacteur / Andreas Krause ; Emma Brunskill ; Kyunghyun Cho ; Barbara Engelhardt ; Sivan Sabato ; Jonathan Scarlett. Vol. 202 Proceedings of Machine Learning Research (PMLR), 2023. blz. 28079-28099 (Proceedings of Machine Learning Research).

@inproceedings{2085f69e234a4731b04ea3219a3f48d5,

title = "Spurious Valleys and Clustering Behavior of Neural Networks",

abstract = "Neural networks constitute a class of functions that are typically non-surjective, with high-dimensional fibers and complicated image. We prove two main results concerning the geometry of the loss landscape of a neural network. First, we provide an explicit effective bound on the sizes of the hidden layers so that the loss landscape has no spurious valleys, which guarantees the success of gradient descent methods. Second, we present a novel method for analyzing whether a given neural network architecture with monomial activation function can represent a target function of interest. The core of our analysis method is the study of a specific set of error values, and its behavior depending on different training datasets.",

author = "Samuele Pollaci",

note = "Funding Information: The results contained in this paper are part of the author's Master's thesis project, carried out at Bonn University with advisors Daniel Huybrechts and Emre Set{\"o}z. The paper was completed during the first year of the author's doctoral studies at Vrije Universiteit Brussel with supervisor Bart Bogaerts. This work was partially supported by Fonds Wetenschappelijk Onderzoek - Vlaanderen (project G0B2221N) and the Flemish Government (Onderzoeksprogramma Artifici{\"e}le Intelligentie (AI) Vlaanderen). Funding Information: The results contained in this paper are part of the author{\textquoteright}s Master{\textquoteright}s thesis project, carried out at Bonn University with advisors Daniel Huybrechts and Emre Set{\"o}z. The paper was completed during the first year of the author{\textquoteright}s doctoral studies at Vrije Universiteit Brussel with supervisor Bart Bo-gaerts. This work was partially supported by Fonds Weten-schappelijk Onderzoek – Vlaanderen (project G0B2221N) and the Flemish Government (Onderzoeksprogramma Arti-fici{\"e}le Intelligentie (AI) Vlaanderen). Publisher Copyright: {\textcopyright} 2023 Proceedings of Machine Learning Research. All rights reserved.; Fortieth International Conference on Machine Learning, ICML 2023 ; Conference date: 23-07-2023 Through 29-07-2023",

year = "2023",

month = jul,

language = "English",

volume = "202",

series = "Proceedings of Machine Learning Research",

publisher = "Proceedings of Machine Learning Research (PMLR)",

pages = "28079--28099",

editor = "{ Krause}, Andreas and Brunskill, {Emma } and Cho, {Kyunghyun } and Engelhardt, {Barbara } and Sivan Sabato and { Scarlett}, Jonathan",

booktitle = "Proceedings of the 40th International Conference on Machine Learning",

url = "https://icml.cc/Conferences/2023",

}

Pollaci, S 2023, Spurious Valleys and Clustering Behavior of Neural Networks. in A Krause, E Brunskill, K Cho, B Engelhardt, S Sabato & J Scarlett (uitgave), Proceedings of the 40th International Conference on Machine Learning. vol. 202, Proceedings of Machine Learning Research, Proceedings of Machine Learning Research (PMLR), blz. 28079-28099, Fortieth International Conference on Machine Learning, Honolulu, Hawaii, United States, 23/07/23. <https://openreview.net/forum?id=TuHgrnPHZq>

Spurious Valleys and Clustering Behavior of Neural Networks. / Pollaci, Samuele.
Proceedings of the 40th International Conference on Machine Learning. uitgave / Andreas Krause; Emma Brunskill; Kyunghyun Cho; Barbara Engelhardt; Sivan Sabato; Jonathan Scarlett. Vol. 202 Proceedings of Machine Learning Research (PMLR), 2023. blz. 28079-28099 (Proceedings of Machine Learning Research).

Onderzoeksoutput: Conference paper

TY - GEN

T1 - Spurious Valleys and Clustering Behavior of Neural Networks

AU - Pollaci, Samuele

N1 - Conference code: 40

PY - 2023/7

Y1 - 2023/7

N2 - Neural networks constitute a class of functions that are typically non-surjective, with high-dimensional fibers and complicated image. We prove two main results concerning the geometry of the loss landscape of a neural network. First, we provide an explicit effective bound on the sizes of the hidden layers so that the loss landscape has no spurious valleys, which guarantees the success of gradient descent methods. Second, we present a novel method for analyzing whether a given neural network architecture with monomial activation function can represent a target function of interest. The core of our analysis method is the study of a specific set of error values, and its behavior depending on different training datasets.

AB - Neural networks constitute a class of functions that are typically non-surjective, with high-dimensional fibers and complicated image. We prove two main results concerning the geometry of the loss landscape of a neural network. First, we provide an explicit effective bound on the sizes of the hidden layers so that the loss landscape has no spurious valleys, which guarantees the success of gradient descent methods. Second, we present a novel method for analyzing whether a given neural network architecture with monomial activation function can represent a target function of interest. The core of our analysis method is the study of a specific set of error values, and its behavior depending on different training datasets.

UR - http://www.scopus.com/inward/record.url?scp=85174396610&partnerID=8YFLogxK

M3 - Conference paper

VL - 202

T3 - Proceedings of Machine Learning Research

SP - 28079

EP - 28099

BT - Proceedings of the 40th International Conference on Machine Learning

A2 - Krause, Andreas

A2 - Brunskill, Emma

A2 - Cho, Kyunghyun

A2 - Engelhardt, Barbara

A2 - Sabato, Sivan

A2 - Scarlett, Jonathan

PB - Proceedings of Machine Learning Research (PMLR)

T2 - Fortieth International Conference on Machine Learning

Y2 - 23 July 2023 through 29 July 2023

ER -

Spurious Valleys and Clustering Behavior of Neural Networks

Samenvatting

Publicatie series

Conference

Bibliografische nota

Toegang tot document

Handle.net

Andere bestanden en links

Vingerafdruk

Onderzoekersoutput