NotInMaps false positive

[ivan-coder]

We started using the plugin a few days ago, but shortly after implementation, we encountered erratic failures in our unit test suite. One test, responsible for validating scene references across our prefabs, began exhibiting inconsistent behavior.

Our test looks like this:

            if (scene.State == SceneReferenceState.Unsafe)
            {
                Debug.LogError($"There's a problem with the scene settings. Reason: '{scene.UnsafeReason}'");
            }

So while all works well in editor, turns out that the unit tests running on a remote self-hosted linux machine fail sometimes.
Investigating a bit, it seems that the SceneDataMap was sometimes not generated by the time the test was trying to run, thus failing with "NotInMaps" message.

We added this to make sure the maps were generated before running any tests, and it seems to be working fine now.

        [SetUp]
        public void SetUpTests()
        {
            SceneDataMapsGenerator.Run(false);
        }

However, we're left wondering if this behavior is expected or if others have encountered similar challenges when running tests through GitHub Actions.

[Ale1]

It could be caused by github actions running unity with -batchmode flag.
(side note: Its infuriating that unity's batchmode initializes things in different order to editor enviroment... )

@ivan-coder . You could test locally on your computer running the unit tests in batchmode to check if same error pops up.
Btw, have you tried [OneTimeSetup] instead for your workaround? It seems like regenerating the maps on [SetUp] would be overkill.

[starikcetin]

I think @Ale1 is right. In my previous job we have encountered situations where the build hooks were not executed when the editor is run in batchmode. We instead ran our builds without batchmode and manually called editor quit in the code after we are done.

In this case, since the map generator runs in a pre-build hook, it might not be running in batchmode.

[starikcetin]

That being said, what you did in the setup method is correct. So you can continue using it also. Just be careful about testing in players, then you would want to call the generator with true instead, since you would want file outputs in that case.