-
Notifications
You must be signed in to change notification settings - Fork 357
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fixing three flaky tests in AuthorisationsTest and ViewTest #3327
Open
Sitara-a
wants to merge
5
commits into
gchq:develop
Choose a base branch
from
Sitara-a:flaky-test-fix
base: develop
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## develop #3327 +/- ##
==========================================
Coverage 67.94% 67.94%
Complexity 2596 2596
==========================================
Files 955 955
Lines 30530 30530
Branches 3369 3369
==========================================
Hits 20744 20744
Misses 8306 8306
Partials 1480 1480 ☔ View full report in Codecov by Sentry. |
Looks good to me. |
Sitara-a
changed the title
Fixing two flaky tests in AuthorisationsTest and ViewTest
Fixing three flaky tests in AuthorisationsTest and ViewTest
Nov 18, 2024
Quality Gate passedIssues Measures |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Two flaky tests were fixed in this pull request-
Flakiness in the tests -
To check for flakiness in the tests, a tool called nondex was used.
Steps to reproduce flakiness using nondex -
Test 1:
The third command shows us the flakiness in the test. There are test failures due to differences in the order of elements
Test 2:
The third command shows us the flakiness in the test. There are test failures due to differences in order of elements
Test 3:
Source of Flakiness in tests and fixes:
The Authorisation object created in the test case appends three strings to the auths member of the class which is a HashSet. The toString() method of the class then iterates over this HashSet and builds a String after concatenating the members of the set. Since a Java HashSet does not store order, the test case fails if the elements of the HashSet are iterated over in any order other than the order in which they were appended during object instantiation. To fix this, the test was updated to check if all strings in the auths member of the object are a part of the String returned by toString() instead of focusing on the order.
In this test case, the source of flakiness is the following lines -
final byte[] viewJson = view.toCompactJson();
final byte[] cloneJson = clone.toCompactJson();
assertThat(cloneJson).containsExactly(viewJson);
The method toCompactJson() internally uses JSONSerialiser.serialise() to convert the object into a byte stream. This does not preserve the order of attributes. Therefore, the containsExactly() method fails when the order of attributes differs in the two objects. To fix this, an ObjectMapper object was created and two JsonNode objects were created after calling the readTree() method on the two byte arrays. The equality check was then applied on these two JsonNode objects.
Similar to test 2, the source of flakiness is from the toCompactJson() method which does not preserve the order of attributes.
Please let me know if you have any questions or need any additional justification/changes from my side.