Understanding and Minimising Outlier Features in Transformer Training

Record

Authors
He, B.; Noci, L.; Paliotta, D.; Schlag, I.; Hofmann, T.
Year
2024
Title
Understanding and Minimising Outlier Features in Transformer Training
Published in
Advances in Neural Information Processing Systems 37 (NeurIPS 2024)
Type
Conference paper
arXiv
arXiv:2405.19279
Our note
none yet
Verification
verified: checked by hand against the source named below (the open indexes miss or misdate it).
Notes: checked by hand against https://proceedings.neurips.cc/paper_files/paper/2024/hash/986292a930c3692168b177a770025ab3-Abstract-Conference.html (see the correction below); the open indexes gave needs-review (title similarity 0.894).
Id
he-2024-understanding-minimising-outlier-features-transformer-training

Corrections applied to this record: fields (cited as the NeurIPS 2024 paper (the parser read the arXiv id as a preprint); source: https://proceedings.neurips.cc/paper_files/paper/2024/hash/986292a930c3692168b177a770025ab3-Abstract-Conference.html); verified (the NeurIPS 2024 version of record is titled "...in Transformer Training", as cited; both arXiv versions (v1, May 2024; v2, the camera-ready, November 2024) are titled "...in Neural Network Training"; source: https://proceedings.neurips.cc/paper_files/paper/2024/hash/986292a930c3692168b177a770025ab3-Abstract-Conference.html).

Cited on

As cited

The pages cite this work in one form.

  • He, B., Noci, L., Paliotta, D., Schlag, I., and Hofmann, T. (2024). Understanding and Minimising Outlier Features in Transformer Training. Advances in Neural Information Processing Systems 37 (NeurIPS 2024). arXiv:2405.19279. (Later work on normalization layers, optimizers and outlier features; locations refer to the NeurIPS version.)

BibLaTeX

@inproceedings{he-2024-understanding-minimising-outlier-features-transformer-training,
  author = {He, B. and Noci, L. and Paliotta, D. and Schlag, I. and Hofmann, T.},
  title = {{Understanding and Minimising Outlier Features in Transformer Training}},
  booktitle = {Advances in Neural Information Processing Systems 37 (NeurIPS 2024)},
  date = {2024},
  eprint = {2405.19279},
  eprinttype = {arxiv}
}

Download · JSON with CSL-JSON · the whole bibliography