-
Notifications
You must be signed in to change notification settings - Fork 15
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Visual Studio Test Explorer does not show executing tests #263
Comments
Hi, this doesn't sound like something that xRetry could be causing. I'd try switching the The way xRetry works under the hood means that once a test is running, no updates get sent to xUnit until the test passes, is skipped, or we run out of retries. That means the Visual Studio test runner wouldn't know that a test is being retried to treat it any differently to a normal test. |
I think your description of how xRetry works under the hood fits what I'm seeing exactly. If xRetry isn't reporting to the XUnit architecture that the test is running before it starts running the test, then XUnit has nothing triggering it to relay that report on to Visual Studio. When it waits until the test has it's final result to report that it's running and then that it is finished, the result in Visual Studio is that the running spinner for the test appears for around 1 second, and then it goes straight to the result. Meanwhile, while the test is running for seconds or (in the case of GUI Tests) minutes on end, it is VERY hard to be able to tell exactly what is running at the time. It seems that the BlockingMessageBus in the code you linked could be given some smarts to determine whether each message it gets is a message to start the test. If it is, it could go ahead and pass it on to XUnit, while continuing to queue up all of the messages indicating that the test is finished until it's done retrying. |
I should have provided more context in my previous message as I've caused some confusion: As an aside: additional tests can be discovered at runtime, e.g. theories with dynamic data; these are the tests that you previously mentioned do not show in the test tree until they start running. I don't think these are relevant to the problem, but it's an exception to the usual pattern. Back to your problem: I don't know how the test explorer is monitoring which test is running, but since the spinner shows for you sometimes it sounds like it must know that a test has started. Since we're blocking all messages with the The only way I can see this being xRetry related is if the test explorer expects to see some message within a certain amount of time of the test starting, and we are blocking that, but I don't know if this is the case, or which one(s) it expects. If you find that is the case, we can look at passing through some messages immediately. Depending on which messages are needed though, this may not be straight-forward, as we need to keep messages in the right order, and currently only the messages of the final run are reported back to xunit. |
Before we installed xRetry, when we looked at the currently running tests in Visual Studio's Test Explorer, xUnit would almost always display a spinner beside every test that was currently running.
The only exception to this was when a test was data driven AND it's data objects were too complex for Test Explorer to display in its tree view. In this case, it would display the spinner while the first test case was running, and then it would display the results of the test cases in that test that had been run. If those already run had all succeeded, it would display the success icon. If one or more had failed, it would display the failure icon. But until all of the test cases had been run, you could not tell just by eyeing the test tree which test was actually running at the time. This was an XUnit bug that was quite annoying.
Since we have installed xRetry and outfitted the needed tests with RetryFact or RetryTheory, this exception has become more like the norm for the tests with the xRetry attributes, albeit with slightly different symptoms. Rarely does the spinner ever appear on running tests, and when it does, it is only a second or two before the test changes back to the icon indicating the test has not yet been run. Unitl all retries have completed, the only clue that the test has failed is the message bar at the top, which indicates the latest test that has failed and which retry out of how many is currently being run.
It would be nice if xRetry could interact with xUnit in such a way that it is clear which test is being run or rerun.
Currently, Visual Studio is on version 17.12.3. We are running XUnit 2.9.0, and xRetry 1.9.0.
The text was updated successfully, but these errors were encountered: