test_run: new tag "<-- BROKEN --> by {#bug_id of breaking bug]"

classic Classic list List threaded Threaded
2 messages Options
Samuel GOUGEON Samuel GOUGEON
Reply | Threaded
Open this post in threaded view
|

test_run: new tag "<-- BROKEN --> by {#bug_id of breaking bug]"

Hello,

I am wondering about the processing of any unitary or non-regression test that fails not because the bug that it tests is back, but because another external bug breaks the code of the test.
This is currently the case for instance for

test_run graphics datatipSetDisplay
test_run graphics bug_10298
test_run cacsd bug_13359
test_run cacsd bug_15827

that all are broken by the bug 15790.

Currently, we tend to reopen these tests with the "<-- NOT FIXED -->" tag.
But this processing is not really fair. It is rather puzzling.

As a general policy, i am not in favour of multiplying the number of available tags.
However, what about creating a new one to point this situation?
It could be
 // <-- BROKEN -->  by 15790
Such a test would be skipped, and in the test_run report, "skipped: broken by 15790" would be displayed.

What do you think about this proposal?

Regards
Samuel


_______________________________________________
dev mailing list
[hidden email]
http://lists.scilab.org/mailman/listinfo/dev
Samuel GOUGEON Samuel GOUGEON
Reply | Threaded
Open this post in threaded view
|

Re: test_run: new tag "<-- BROKEN --> by {#bug_id of breaking bug]"

Le 02/02/2019 à 19:32, Samuel Gougeon a écrit :

As a general policy, i am not in favour of multiplying the number of available tags.
However, what about creating a new one to point this situation?
It could be
 // <-- BROKEN -->  by 15790
Such a test would be skipped, and in the test_run report, "skipped: broken by 15790" would be displayed.


Another processing could be: the test is run anyway. Then,
  • if it passes: "passed. No longer broken by 15790" is displayed.
    This would focus on the fact that the test's header must be updated.

  • otherwise: "failed: broken by 15790" is displayed
    This would focus on fixing the bug 15790.


_______________________________________________
dev mailing list
[hidden email]
http://lists.scilab.org/mailman/listinfo/dev