Softmax Linear Units

Record

Authors
Elhage, N.; et al.
Year
2022
Title
Softmax Linear Units
Published in
Transformer Circuits Thread
Type
Web page
Links
https://transformer-circuits.pub/2022/solu/index.html
Our note
Softmax Linear Units
Verification
url-ok: the web address responds (the page itself is not checked).
https://transformer-circuits.pub/2022/solu/index.html: HTTP 200, checked 2026-11-14
Id
elhage-2022-softmax-linear-units

Cited on

As cited

The pages cite this work in 3 forms.

  • Elhage, N., Hume, T., Olsson, C., Nanda, N., Henighan, T., Johnston, S., El Showk, S., Joseph, N., DasSarma, N., Mann, B., Hernandez, D., Askell, A., Ndousse, K., Jones, A., Drain, D., Chen, A., Bai, Y., Ganguli, D., Lovitt, L., Hatfield-Dodds, Z., Kernion, J., Conerly, T., Kravec, S., Fort, S., Kadavath, S., Jacobson, J., Tran-Johnson, E., Kaplan, J., Clark, J., Brown, T., McCandlish, S., Amodei, D., and Olah, C. (2022). Softmax Linear Units. Transformer Circuits Thread. https://transformer-circuits.pub/2022/solu/index.html. The page gives its publication date as 27 June 2022; Anthropic's research page lists the item under 17 June 2022. Section and figure numbers in this note refer to the web version.
  • Elhage, N., Hume, T., Olsson, C., Nanda, N., et al. (2022). Softmax Linear Units. Transformer Circuits Thread. https://transformer-circuits.pub/2022/solu/index.html. Sections 6.3 ("Qualitative Exploration of SoLU Models") and 7.1 ("Understanding Transformer MLPs").
  • Elhage, N., et al. (2022). Softmax Linear Units. Transformer Circuits Thread. https://transformer-circuits.pub/2022/solu/index.html

BibLaTeX

@online{elhage-2022-softmax-linear-units,
  author = {Elhage, N. and others},
  title = {{Softmax Linear Units}},
  organization = {Transformer Circuits Thread},
  date = {2022},
  url = {https://transformer-circuits.pub/2022/solu/index.html}
}

Download · JSON with CSL-JSON · the whole bibliography