Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Flaky tests #564

Open
BMurri opened this issue Jan 9, 2024 · 0 comments
Open

Flaky tests #564

BMurri opened this issue Jan 9, 2024 · 0 comments
Labels
bug Something isn't working

Comments

@BMurri
Copy link
Collaborator

BMurri commented Jan 9, 2024

Describe the bug
Several tests will fail from time to time due to the forward passage of time (e.g. ArmUrlTransformationStrategyTests.TransformUrlWithStrategyAsync_CallTwiceForSameStorageAccount_CachesKey)

Steps to Reproduce
Steps to reproduce the behavior:
Run tests. Sooner or later, a test will incorrectly fail because some component of time included in the comparison will have changed enough to generate an unexpected result (e.g. creating a sas with a cached user key with an expiration datetime that is one second later than the reference sas).

Expected behavior
A clear and concise description of what you expected to happen.
Tests that don't depend on a portion of the test completing within some arbitrary time interval (e.g. within the same second of wall-clock time).

Additional context
Add any other context about the problem here.

  Failed TransformUrlWithStrategyAsync_CallTwiceForSameStorageAccount_CachesKey [2 ms]
  Error Message:
   Assert.AreEqual failed. Expected:<https://foo.blob.core.windows.net/?skoid=8d61b50e-e71e-4ea7-a53d-0a443ff67bb3&sktid=e6d03617-7c83-4d25-95c9-aa47ac129de1&skt=2024-01-09T05%3A16%3A39Z&ske=2024-01-09T06%3A16%3A39Z&sks=SIGNED_SERVICE&skv=V1_0&sv=2022-11-02&se=2024-01-16T05%3A16%3A39Z&sr=c&sp=r&sig=fXBaEzjzTwJsoPIjT1XJl%2Fn8EjlxDqT%2FdgDwTCY4MV4%3D>. Actual:<https://foo.blob.core.windows.net/?skoid=8d61b50e-e71e-4ea7-a53d-0a443ff67bb3&sktid=e6d03617-7c83-4d25-95c9-aa47ac129de1&skt=2024-01-09T05%3A16%3A39Z&ske=2024-01-09T06%3A16%3A39Z&sks=SIGNED_SERVICE&skv=V1_0&sv=2022-11-02&se=2024-01-16T05%3A16%3A40Z&sr=c&sp=r&sig=1pdOXMwx4BGnE2XHWymVcKjP5kgCffuQTRG1gOP1NVY%3D>. 

Note that the user delegation key is identical in both tokens, but the expiration date/time is one second later. The two tokens were obtained one right after the other.

Fixing this will require mocking the "system clock" or repeating tests/portions of tests as appropriate to report correct test results without clock skews producing false test failures.

@BMurri BMurri added the bug Something isn't working label Jan 9, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant