GPT-3 configuration for a v3-32 TPU

[stefan-it]

Hi,

many thanks for releasing this GPT training code 👍

I just wanted to train a new model from scratch (with own vocab), so I was using the following configuration file

https://github.com/EleutherAI/gpt-neo/blob/master/configs/gpt3_small_256.json

However, I'm not 100% sure what to use for mesh_shape and layout, because I'm not using a 256 TPU pod, I'm using a v3-32 only.

Could you please provide some more information about how to use the correct values?

Many thanks in advance and best,

Stefan