OpenAI's GPT-3 is a large language model with a parameter count of up to 175B. Despite the amazing results that GPT-3 shows, it is not open source, so if you want to try it, AI Dungeon (https://play.aidungeon.io/main/landing) or Philosopher AI(https://philosopherai.com/) should be used. In addition, it is highly likely that it will be paid for in the future due to an exclusive licensing agreement with Microsoft.
GPT-Neo, published by Eleuther AI, a non-profit open source research organization, is a large language model learned using the structure of GPT-3. The dataset named Pile and pre-trained models were also released. Here are links to GPT-Neo and Pile's github repositories:
GPT-Neo was created based on mesh-tensorflow, a library for large-scale parallel learning, and a pre-trained model of a model with 1.3B parameters and a model with 2.7B parameters are disclosed. In addition, GPT-Neo has been added to HuggingFace, making it easy to use. The following is HuggingFace's GPT-Neo link, and additional models with 125M and 350M parameters are provided. You can try total four models of GPT-Neo 125M, GPT-Neo 350M, GPT-Neo 1.3B and GPT-Neo 2.7B.
Meanwhile, Eleuther AI is also working on GPT-NeoX, the successor project of GPT-Neo. Unlike GPT-Neo, which is based on mesh-tensorflow, GPT-NeoX uses NVidia Megatron and DeepSpeed (https://smilegate.ai/2021/01/27/deepspeed-fairscale/). And the codebase is changed to pytorch instead of tensorflow. According to Eleuther AI, we plan to finally train a model with a number of parameters similar to that of GPT-3 175B, so it is expected that various additional analyzes and applications will be possible. Here is a link to GPT-NeoX's github repository.
