Ability to serialize/deserialize Schema on the server side

[jrabek]

Thanks again @endel for the great multiplayer game framework! If you would provide some basic guidance on this enhancement I would be happy to implement it in a PR.

Summary

This is a request for a Schema method for completely serializing/deserializing the server Schema (i.e. game state) so that it can be persisted in a database.

Motivation

In the current colyseus server architecture, the game state (i.e. some Schema derived class) persists in memory in one specific server.

One of the shortcomings is that it prevents seamless deploys and crash recovery since the game state is in memory and therefore lost when the server process is killed. For certain game types (e.g. long running games, games involving money) it is critical that game state is not lost.

To support a more robust game state, the ability to fully serialize the server Schema in a way that it could be stored in a database is needed.

Currently there are two ways to serialize the server Schema neither of which are meant for deserializing it later.

1. toJSON: This method is meant for logging / debugging
2. encodeAll / decode: These are meant to send state updates to the clients and not for recreating the entire server Schema.

For (2) this is what happens when the server state is serialized/deserialized using encodeAll / decode: https://gist.github.com/jrabek/6303d3f94b82c692de7721f0fc331f8c.

[jrabek]

Some related discord discussions:

Original conversation: https://discord.com/channels/525739117951320081/526083188108296202/880867521824170005

Follow up when encodeAll / decode wasn't working: https://discord.com/channels/525739117951320081/526083188108296202/889878308983046215.

[Wenish]

as a workaround:

in the onCreate function of the room call a loadState function which gets the room state from a database and maps it to the state (mapping has to be manual)

and in the onDispose function of the room call a saveState function which save the room in a database (mapping has to be manual)

at least thats how i do it :)

and maybe its worth for your game to auto save the state every 5 minutes in case server crashes. so only 5 minutes of game gets lost.

[jrabek]

@Wenish so I am not using the rest of colyseus, just the schema. Also I don't think what you are proposing will work because not all the internal values are saved (most importantly the ids of the fields meaning the diff that gets sent after reloading the data breaks). Check out the gist I had sent before: https://gist.github.com/jrabek/6303d3f94b82c692de7721f0fc331f8c

[Wenish]

@jrabek im talking about a workaround where you write your own mapper from the schema to your database model and visa versa. not using any built in function you try to use.
and it does work. i load hole maps from a json file

[lpsandaruwan]

Hi @jrabek

Still investigating this issue. Hopefully plan to fix this in future. You will be able to use this workaround for now.

 // Simulate a serialization to a database
  if (doDecode) {
    state = new State();
    state.decode(encodedState);
    state = state.clone();
  }

cc @endel

[jrabek]

Thanks for the update @lpsandaruwan! Definitely would love to try out colyseus again!

I must have missed clone before. Was that added more recently?

With this approach will the cloned state work with per client filters?

[lpsandaruwan]

Hi @jrabek
Cloning the state and reassigning, was a shot in the dark :-D. It resolved the undefined property exception when decoding state back.
It should work without any interruption. I'm still testing client's behavior against this. But if you have any inputs/help to offer on this please feel free to mention.

cc @endel

[jrabek]

@lpsandaruwan I had a chance to play with your suggestion and while I don't see the exception I also don't see the correct behavior when decoding from an encoded state and then cloning.

You can check out my experiment here https://github.com/jrabek/colyseus-schema-experiments/tree/main. I put a few notes in the README.md.

[lpsandaruwan]

Thank you @jrabek. Will look into this.

[lpsandaruwan]

Hi @jrabek
From your provided sample, I think after cloning the schema you might have to set the schema inside the serializer object again since the serializer object is referenced to the old schema object, not the cloned one. Please let us know if that worked for you. In, https://github.com/jrabek/colyseus-schema-experiments/blob/def4ed67724ce2da52e5bd6491fb4a7fc41136fd/server.ts#L23

    if (this.doDecode) {
      this.state = new GameState();
      this.state.decode(encodedState);
      this.state = this.state.clone();
      this.serializer = new Serializer(this.state);
    }

[jrabek]

Thanks for the follow-up! I'll check when I have a second this week.

[jrabek]

@lpsandaruwan sorry for the delay and thanks again for the follow up!

I tried this out locally and the initial results look really promising! I'll need to spend some more time with it but this seems like an acceptable approach to the original problem.

[jrabek]

Hmmm, I may have spoken too soon. It seems that any modification to the state followed by a save/load cycle result in the full state getting sent to all clients. Per client filtering is working as expected but the full state is always sent.

So it seems like the data required to know what the last state update was is lost when doing a serialization/deserialization.

To be extra sure I added a small test that simple sets a string in the state to "a". The resulting patch sent to each client is the entire state.

console.log("--- minor update ---");
server.smallUpdateTest();
updateClientsStates();

If doDecode is false then the update sent to each client is of length 5.

============= smallUpdateTest =============
==== saveLoadState ==== (length:179)
Server state {
  currentPlayerUserId: 'player2',
  players: {
    player1: { userId: 'player1', isVip: false },
    player2: { userId: 'player2', isVip: false },
    player3: { userId: 'player3', isVip: false },
    player4: { userId: 'player4', isVip: false },
    player5: { userId: 'player5', isVip: false }
  },
  words: [ 'hey', 'there' ],
  secret: 'secretWord for player 2',
  title: 'a'
}
Getting updates for  [ 'player1', 'player2', 'player3', 'player4', 'player5' ]
--- Updating client state ---
Client updateState (length:5) before (player2) {
  currentPlayerUserId: 'player2',
  players: {
    player1: { userId: 'player1', isVip: false },
    player2: { userId: 'player2', isVip: false },
    player3: { userId: 'player3', isVip: false },
    player4: { userId: 'player4', isVip: false },
    player5: { userId: 'player5', isVip: false }
  },
  words: [ 'hey', 'there' ],
  secret: 'secretWord for player 2'
}
Client updateState (length:5) after (player2) {
  currentPlayerUserId: 'player2',
  players: {
    player1: { userId: 'player1', isVip: false },
    player2: { userId: 'player2', isVip: false },
    player3: { userId: 'player3', isVip: false },
    player4: { userId: 'player4', isVip: false },
    player5: { userId: 'player5', isVip: false }
  },
  words: [ 'hey', 'there' ],
  secret: 'secretWord for player 2',
  title: 'a'
}
Client updateState (length:5) before (player1) {
  currentPlayerUserId: 'player2',
  players: {
    player1: { userId: 'player1', isVip: false },
    player2: { userId: 'player2', isVip: false },
    player3: { userId: 'player3', isVip: false },
    player4: { userId: 'player4', isVip: false },
    player5: { userId: 'player5', isVip: false }
  },
  words: [ 'hey', 'there' ]
}
Client updateState (length:5) after (player1) {
  currentPlayerUserId: 'player2',
  players: {
    player1: { userId: 'player1', isVip: false },
    player2: { userId: 'player2', isVip: false },
    player3: { userId: 'player3', isVip: false },
    player4: { userId: 'player4', isVip: false },
    player5: { userId: 'player5', isVip: false }
  },
  words: [ 'hey', 'there' ],
  title: 'a'
}

If doDecode is true then the length of the client update is 206 or 156 (depending on filtering).

[jrabek]

So I guess the option at this point would be to serialize the state to the database continually, but only as a backup. If the server crashes it can reread the state and would send the full update to the clients to ensure they are back in sync. Which is probably what you want anyway.