Saving Trained QLearner

[blurblur12]

I trained a QLearner, and I want to save it to be used later. One way I can think of is export the _q_values attribute to a json file, then load the json file later and assign it to the new QLearner. However, one problem I came accross is loading json file takes quite a bit of time, and I need to reformat the loaded json file to collections.defaultdict, which became a bit confusing. So I am wondering is there any better ways to save my trained agent?

Thanks

[lanctot]

Hmmm.... do you mean the tabular Q-learner? I assume you mean the Python Tabular QLearner. AFAIK that's not currently built-in, but you could just serialize the entire QLearning object every X iterations using pickle and reload that from disk, right? Does that work?

If you mean the C++ tabular q-learning, you'd have to do it manually.. but that functionality would make a nice contribution :)

If you mean the Q-learning with function approximation, it'd depend on which ML framework; most of them already support it but in different ways.

Probably easiest to first point me out to the specific implementation/file you're talking about and I'll follow-up if necessary.

[blurblur12]

Yes I am talking about the tabular Q-learner class from Open Spiel and it's for python. It is located in open_spiel/python/algorithms/tabular_qlearner.py
I trained an instance of the QLearner class and I want to some how save it so that I could use it later in other scripts without having to train it again. I am running my script in Google Colab, so I don't want to save the model to Colab's disk, ideally it would be nice if I could save it into a file and download it to my local machine. Then I could always just upload the saved model in Colab whenever I needed to.

[lanctot]

Ah yes, I'm not a colab expert but I just did a bit of searching. Turns out that you can mount your google drive and copy the files there (and load the files from the drive).

I just tested it from the simple example linked on the main page, and it worked!

Here's the code I added to the end of the example colab:

import pickle
import tempfile
from google.colab import drive
drive.mount('/content/drive')

with open("state.pkl", 'wb') as f:
  pickle.dump(state, f)

!cp "state.pkl" "/content/drive/MyDrive"

In your case you'd save/load the whole QLearner object or a custom checkpoint object that contains the Q-learner plus any other information you want to save (total number of iterations, etc.)

Here are some helpful links to learn more:

• Save Any File From Colab to Your Google Drive
• Pickling in Python - The Very Basics

Hope this helps!