Updating Alpha Zero

[ramizouari]

This is a feature proposal.

While basing my work on this version of Alpha Zero (Tensorflow, both in Python and C++), I have many points that I have addressed, including:

1. It is clear that the Alpha Zero implementation is depending on the compatibility layer of TensorFlow.
2. Some constants are hard coded. I have two examples of this, stage_count=7 and wait(0.01). While their meaning is clear from context and from the subsequent logic, it will be nice to parameterise them.
3. The replay buffer seems to be manually implemented. It will be also nicer to base it on the sibling project dm-reverb
4. The implementation seems to be tightly coupled. By service, I mean an actor or a learner or an evaluator.

My proposed implementation is based on TensorFlow 2, and is almost functionally equivalent to the old one, only the model definition is not the same (I am working on a specific game, but I can follow the "standard" architectures exactly if needed)
There are also later improvements concerning point 3 and 4, but I will be happy to propose the migration from TF1 logic to TF2 as a starter.

Before anything else, I want to know if you are open to such idea?

[lanctot]

Hi @ramizouari,

That's great! I think we will soon need to replace (or deprecate) all of our TF1 implementations soon / eventually, so I'd be happy to upgrade our TF AlphaZero to be base primarily on TF2. Luckily on the basic RL side, we have most of main algorithms already in JAX & PyTorch so this won't be too problematic, but definitely TF AlphaZero is not in that category.

I'm surprised you got the Tensorflow-based C++ AlphaZero version to work. Last I remember, we only had that working internally. Either way: it's not used very much because most people could not get it to compile externally. So we left it there mostly so people could see it. It'd be OK if the TF2 version of AlphaZero didn't support the C++ TF API because it's not nearly as user-friendly as the LibTorch, which most of the C++ users have used instead.

Would you be willing to submit a PR that upgrades it? I'd like some evidence that it works as well, on maybe say Tic-Tac-Toe and/or Connect Four. We should also loop in @tewalds who was the original author of that code to see if he has anything to add.

Let me know what you think!

[ramizouari]

Hi @lanctot,

First of all, thank you for your reply. I will be more than happy to submit a PR for this proposal.

To be honest, getting to C++ version to work was not an easy task. The main problem was getting TensorFlow to work. While I was not able to build TensorFlow from source, nor build it as mentioned by the documentation via TensorFlowCC. I was able to link the library against the pip installed version (It contains the *.so files, and fortunately, it does also contain the header files).

Also, to load the model, I had to use the SavedModelBundle interface. Still, I was not able to implement these scenarios yet:

• Loading particular checkpoints
• Calling the Fit function
  I think they are possible, but need further inspection of the protobuf file.

For the Python version, everything works with the new API.

For the PR, I will try to do it before the end of July.

For a quick inspection of the code before the PR, you can find it on my fork under the same name open_spiel. It should contain:

• For Python: The files of the new implementation on the same path with a suffix _v2.
• For C++: It is under open_spiel/algorithms/alpha_zero_mpg, but this is an implementation for a very specific game. I will rewrite a generalised version from it for the PR.

If you need further clarification, Let me know.

[lanctot]

Alright, cool! Take all the time you need. In the mean time, I will point @tewalds to the thread to see if he has any comments.

[tewalds]

It'd be great to have a C++ version of alphazero that uses TF2 and reverb. I didn't use reverb initially with the goal of maximizing what is doable on a single machine, ignoring the multi-machine case, but it'd be great to support multi-machine as well.

As long as you can get it general enough to learn any similar game, I'm all for inclusion.

[ramizouari]

Hello @tewalds.

Thank you for your reply.

In the game that I am working on (Mean Payoff Game), I had to deploy it on a HPC cluster for faster trajectory generation.
For my use case, trajectories were sent with reverb. Also, broadcasting the model and monitoring were done with a HTTP server on each service.

Also, the HTTP part is generic on the sense that we can switch it with another protocol (or default to the multiprocessing queues as is implemented by default), one only have to change:

1. Model broadcasting function
2. The ReplayBuffer implementation (Local via queues / Reverb via gRPC, or a custom one)
3. The model update part on the actors and evaluators.

I had to switch to Python on that part due to the lack of documentation of C++'s implementation of Reverb, but of course it is doable. I will need to contact the Reverb team for more intuition on their C++ code.

Also another limitation of the C++ part, I am still not find the correct format to call the fit function on C++. I only was able to do inference. Also I was not able to load individual checkpoints, but that can be mitigated by simply loading the whole SavedModel bundle on each update.

On the other hand, assuming the Reverb problem in C++ will be resolved, what I can do is implement the learner in Python, and the actors and evaluators in C++. And have them communicate using for example HTTP + Reverb.

Now, as that will constitute a big code addition, I think it will be best if we split them on PR at a time. And for that I will start with the TF2 update.

And as a performance measure, can you please tell me what games should the new implementation be able to learn?

[tewalds]

That all sounds great!

I was using tic-tac-toe as my basic test as it should learn that pretty quickly, then connect four as something a bit harder that it should get great at, and havannah as my real challenge since that's what I worked on for my masters so had a strong opponent I could play via the gtp connector.

[aadharna]

@ramizouari I was playing around with your python-based tf2.0 fork on connect_four and (maybe this is a wsl issue), but I'm getting cuda errors on the learner and evaluator workers when initializing the error because tensorflow doesn't like to share the GPU when using multi-processing. How did you get the code running quickly? When I was just using the CPU, the workers were really really slow to the point of being unusable. (I would have simply raised this as an issue on your fork, but didn't see that as an option)

[Nightbringers]

Hi @ramizouari,

That's great! I think we will soon need to replace (or deprecate) all of our TF1 implementations soon / eventually, so I'd be happy to upgrade our TF AlphaZero to be base primarily on TF2. Luckily on the basic RL side, we have most of main algorithms already in JAX & PyTorch so this won't be too problematic, but definitely TF AlphaZero is not in that category.

I'm surprised you got the Tensorflow-based C++ AlphaZero version to work. Last I remember, we only had that working internally. Either way: it's not used very much because most people could not get it to compile externally. So we left it there mostly so people could see it. It'd be OK if the TF2 version of AlphaZero didn't support the C++ TF API because it's not nearly as user-friendly as the LibTorch, which most of the C++ users have used instead.

Would you be willing to submit a PR that upgrades it? I'd like some evidence that it works as well, on maybe say Tic-Tac-Toe and/or Connect Four. We should also loop in @tewalds who was the original author of that code to see if he has anything to add.

Let me know what you think!

where is AlphaZero algorithms in JAX? what about the speed of AlphaZero in jax? Is there a big difference in speed between AlphaZero JAX and AlphaZero C++?

[lanctot]

where is AlphaZero algorithms in JAX? what about the speed of AlphaZero in jax? Is there a big difference in speed between AlphaZero JAX and AlphaZero C++?

There is no JAX AlphaZero in OpenSpiel. It would make a welcome contribution, though!

[lanctot]

Hi @ramizouari ,

I'm doing a bit of spring cleaning and looking into removing the TF-based C++ AlphaZero, here: #1201. @tewalds, we never managed to get this to work externally and I'm not sure it's worth maintaining long-term if it's not being used. wdyt?

If you have worked on a JAX or TF2-based implementation, we'd love to have it in the repos (if it's well-tested on a few small games), so I'd encourage you to submit it as a PR.

Otherwise users interested in C++-based AlphaZero should use the one based LibTorch. There's also the Python TF implementation.