LOCALLY ACCUMULATED ADAM FOR DISTRIBUTED TRAINING WITH SPARSE UPDATES

Yiming Chen; Nikos Deligiannis

doi:10.1109/ICIP49359.2023.10222032

LOCALLY ACCUMULATED ADAM FOR DISTRIBUTED TRAINING WITH SPARSE UPDATES

Research output: Chapter in Book/Report/Conference proceeding › Conference paper › Research

Abstract

The high bandwidth required for gradient exchange is a bottleneck for the distributed training of large transformer models. Most sparsification approaches focus on gradient compression for convolutional neural networks (CNNs) optimized by SGD. In this work, we show that performing local gradient accumulation when using Adam to optimize transformers in distributed fashion leads to a misled optimization direction and we address this problem by accumulating the optimization direction locally. We also empirically demonstrate most sparse gradients do not overlap and thus show that sparsification is comparable to an asynchronous update. Our experiments with classification and segmentation tasks show that our method can still maintain the correct optimization direction in distributed training event under highly sparse updates

Original language	English
Title of host publication	2023 IEEE International Conference on Image Processing
Place of Publication	IEEE
Publisher	IEEE
Pages	2395-2399
Number of pages	5
ISBN (Electronic)	978-1-7281-9835-4
ISBN (Print)	978-1-7281-9836-1
DOIs	https://doi.org/10.1109/ICIP49359.2023.10222032
Publication status	Published - 2023
Event	2023 IEEE International Conference on Image Processing - Kuala Lumpur Convention Center (KLCC), Kuala Lumpur, Malaysia Duration: 8 Oct 2023 → 11 Oct 2023 https://2023.ieeeicip.org/

Publication series

Name	IEEE International Conference on Image Processing
Publisher	IEEE
ISSN (Print)	1053-5888
ISSN (Electronic)	1558-0792

Conference

Conference	2023 IEEE International Conference on Image Processing
Abbreviated title	ICIP2023
Country/Territory	Malaysia
City	Kuala Lumpur
Period	8/10/23 → 11/10/23
Internet address	https://2023.ieeeicip.org/

Keywords

Distributed Learning
Vision Transformer
Gradient Compression
Optimization

Access to Document

10.1109/ICIP49359.2023.10222032

Cite this

@inproceedings{577d8ad16db946f88c6c9a8389f049f6,

title = "LOCALLY ACCUMULATED ADAM FOR DISTRIBUTED TRAINING WITH SPARSE UPDATES",

abstract = "The high bandwidth required for gradient exchange is a bottleneck for the distributed training of large transformer models. Most sparsification approaches focus on gradient compression for convolutional neural networks (CNNs) optimized by SGD. In this work, we show that performing local gradient accumulation when using Adam to optimize transformers in distributed fashion leads to a misled optimization direction and we address this problem by accumulating the optimization direction locally. We also empirically demonstrate most sparse gradients do not overlap and thus show that sparsification is comparable to an asynchronous update. Our experiments with classification and segmentation tasks show that our method can still maintain the correct optimization direction in distributed training event under highly sparse updates",

keywords = "Distributed Learning, Vision Transformer, Gradient Compression, Optimization",

author = "Yiming Chen and Nikos Deligiannis",

year = "2023",

doi = "10.1109/ICIP49359.2023.10222032",

language = "English",

isbn = "978-1-7281-9836-1",

series = "IEEE International Conference on Image Processing",

publisher = "IEEE",

pages = "2395--2399",

booktitle = "2023 IEEE International Conference on Image Processing",

note = "2023 IEEE International Conference on Image Processing, ICIP2023 ; Conference date: 08-10-2023 Through 11-10-2023",

url = "https://2023.ieeeicip.org/",

}

Chen, Y & Deligiannis, N 2023, LOCALLY ACCUMULATED ADAM FOR DISTRIBUTED TRAINING WITH SPARSE UPDATES. in 2023 IEEE International Conference on Image Processing. IEEE International Conference on Image Processing, IEEE, IEEE, pp. 2395-2399, 2023 IEEE International Conference on Image Processing, Kuala Lumpur, Malaysia, 8/10/23. https://doi.org/10.1109/ICIP49359.2023.10222032

TY - GEN

T1 - LOCALLY ACCUMULATED ADAM FOR DISTRIBUTED TRAINING WITH SPARSE UPDATES

AU - Chen, Yiming

AU - Deligiannis, Nikos

PY - 2023

Y1 - 2023

N2 - The high bandwidth required for gradient exchange is a bottleneck for the distributed training of large transformer models. Most sparsification approaches focus on gradient compression for convolutional neural networks (CNNs) optimized by SGD. In this work, we show that performing local gradient accumulation when using Adam to optimize transformers in distributed fashion leads to a misled optimization direction and we address this problem by accumulating the optimization direction locally. We also empirically demonstrate most sparse gradients do not overlap and thus show that sparsification is comparable to an asynchronous update. Our experiments with classification and segmentation tasks show that our method can still maintain the correct optimization direction in distributed training event under highly sparse updates

AB - The high bandwidth required for gradient exchange is a bottleneck for the distributed training of large transformer models. Most sparsification approaches focus on gradient compression for convolutional neural networks (CNNs) optimized by SGD. In this work, we show that performing local gradient accumulation when using Adam to optimize transformers in distributed fashion leads to a misled optimization direction and we address this problem by accumulating the optimization direction locally. We also empirically demonstrate most sparse gradients do not overlap and thus show that sparsification is comparable to an asynchronous update. Our experiments with classification and segmentation tasks show that our method can still maintain the correct optimization direction in distributed training event under highly sparse updates

KW - Distributed Learning

KW - Vision Transformer

KW - Gradient Compression

KW - Optimization

UR - http://www.scopus.com/inward/record.url?scp=85180763286&partnerID=8YFLogxK

U2 - 10.1109/ICIP49359.2023.10222032

DO - 10.1109/ICIP49359.2023.10222032

M3 - Conference paper

SN - 978-1-7281-9836-1

T3 - IEEE International Conference on Image Processing

SP - 2395

EP - 2399

BT - 2023 IEEE International Conference on Image Processing

PB - IEEE

CY - IEEE

T2 - 2023 IEEE International Conference on Image Processing

Y2 - 8 October 2023 through 11 October 2023

ER -

LOCALLY ACCUMULATED ADAM FOR DISTRIBUTED TRAINING WITH SPARSE UPDATES

Abstract

Publication series

Conference

Keywords

Access to Document

Other files and links

Fingerprint

FWOAL883: Video Processing for Multiview Multimodal Camera Systems