Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Visual Studio Test Explorer does not show executing tests #263

Open
jchesshirAspire opened this issue Dec 17, 2024 · 3 comments
Open

Visual Studio Test Explorer does not show executing tests #263

jchesshirAspire opened this issue Dec 17, 2024 · 3 comments

Comments

@jchesshirAspire
Copy link

Before we installed xRetry, when we looked at the currently running tests in Visual Studio's Test Explorer, xUnit would almost always display a spinner beside every test that was currently running.

The only exception to this was when a test was data driven AND it's data objects were too complex for Test Explorer to display in its tree view. In this case, it would display the spinner while the first test case was running, and then it would display the results of the test cases in that test that had been run. If those already run had all succeeded, it would display the success icon. If one or more had failed, it would display the failure icon. But until all of the test cases had been run, you could not tell just by eyeing the test tree which test was actually running at the time. This was an XUnit bug that was quite annoying.

Since we have installed xRetry and outfitted the needed tests with RetryFact or RetryTheory, this exception has become more like the norm for the tests with the xRetry attributes, albeit with slightly different symptoms. Rarely does the spinner ever appear on running tests, and when it does, it is only a second or two before the test changes back to the icon indicating the test has not yet been run. Unitl all retries have completed, the only clue that the test has failed is the message bar at the top, which indicates the latest test that has failed and which retry out of how many is currently being run.

It would be nice if xRetry could interact with xUnit in such a way that it is clear which test is being run or rerun.

Currently, Visual Studio is on version 17.12.3. We are running XUnit 2.9.0, and xRetry 1.9.0.

@JoshKeegan
Copy link
Owner

Hi, this doesn't sound like something that xRetry could be causing. I'd try switching the RetryFact back to Fact and RetryTheory back to Theory and see if it changes anything.

The way xRetry works under the hood means that once a test is running, no updates get sent to xUnit until the test passes, is skipped, or we run out of retries. That means the Visual Studio test runner wouldn't know that a test is being retried to treat it any differently to a normal test.

@jchesshirAspire
Copy link
Author

jchesshirAspire commented Dec 20, 2024

I think your description of how xRetry works under the hood fits what I'm seeing exactly. If xRetry isn't reporting to the XUnit architecture that the test is running before it starts running the test, then XUnit has nothing triggering it to relay that report on to Visual Studio.

When it waits until the test has it's final result to report that it's running and then that it is finished, the result in Visual Studio is that the running spinner for the test appears for around 1 second, and then it goes straight to the result. Meanwhile, while the test is running for seconds or (in the case of GUI Tests) minutes on end, it is VERY hard to be able to tell exactly what is running at the time.

It seems that the BlockingMessageBus in the code you linked could be given some smarts to determine whether each message it gets is a message to start the test. If it is, it could go ahead and pass it on to XUnit, while continuing to queue up all of the messages indicating that the test is finished until it's done retrying.

@JoshKeegan
Copy link
Owner

If xRetry isn't reporting to the XUnit architecture that the test is running before it starts running the test

I should have provided more context in my previous message as I've caused some confusion:
There's a discovery phase, where xunit finds tests it can run, and an execution phase where it actually runs them. xRetry is called by xunit during each of these phases, first we can discover retryable tests and report back on them by creating a runner for each of them, then each test we discovered is called individually to be run. This allows individual tests to be run, mixing of xRetry and non-xRetry tests etc... The important thing though is that we don't tell xunit that a test is running, xunit runs the test, and the runner implementation for that is xRetry code since we discovered it and created the appropriate runner.

As an aside: additional tests can be discovered at runtime, e.g. theories with dynamic data; these are the tests that you previously mentioned do not show in the test tree until they start running. I don't think these are relevant to the problem, but it's an exception to the usual pattern.

Back to your problem: I don't know how the test explorer is monitoring which test is running, but since the spinner shows for you sometimes it sounds like it must know that a test has started. Since we're blocking all messages with the BlockingMessageBus, it can't be listening for one of those, which is why I said previously that it seems unlikely this is an xRetry issue.

The only way I can see this being xRetry related is if the test explorer expects to see some message within a certain amount of time of the test starting, and we are blocking that, but I don't know if this is the case, or which one(s) it expects. If you find that is the case, we can look at passing through some messages immediately. Depending on which messages are needed though, this may not be straight-forward, as we need to keep messages in the right order, and currently only the messages of the final run are reported back to xunit.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants