ChatGPT

8915 readers

1 users here now

Unofficial ChatGPT community to discuss anything ChatGPT

founded 1 year ago

MODERATORS

[email protected]

140

GPT-4's details are leaked. (threadreaderapp.com)

submitted 1 year ago by [email protected] to c/[email protected]

27 comments fedilink hide all child comments

cross-posted from: https://lemmy.intai.tech/post/72919

Parameters count:

GPT-4 is more than 10x the size of GPT-3. We believe it has a total of ~1.8 trillion parameters across 120 layers. Mixture Of Experts - Confirmed.

OpenAI was able to keep costs reasonable by utilizing a mixture of experts (MoE) model. They utilizes 16 experts within their model, each is about ~111B parameters for MLP. 2 of these experts are routed to per forward pass.

Related Article: https://lemmy.intai.tech/post/72922

no comments (yet)

sorted by: hot top controversial new old

there doesn't seem to be anything here