Test Suites excluded from Test Discovery

[turbophil]

We noticed that our CI process was executing individual Caraya tests that are defined in a Test Suite, but not running the Test Suite itself. In our particular case, it was noticed because we have a Suite that runs a bunch of tests in a parallel-for loop (mostly to cut down on total execution time, but there is also a slight confidence boost in seeing that there is no negative interaction between the VIs under test running in parallel). Upon investigation, we found that the "Discover Tests.vi" is explicitly excluding Test Suites from the results. There is a comment in "Find Test and Suite Index.vi" that gives a little hint as to the rationale:

Quote

Define Test Suite; we DON'T want to include anything with this in it because nested test suites aren't supported.

However, I still don't quite understand why we would want to exclude Test Suites.

It's not necessarily a dealbreaker problem for the example of the call-tests-in-a-parallel-for-loop situation, since that's mostly just done to save time, but I could see a situation where someone might want to define tests to run in a particular order in a Test Suite, but then when automating it, they lose that explicit ordering.

Is there a way to get the test discovery to respect Test Suite grouping? I suppose we could roll our own child class of TestRunner to implement this behavior, but before we go too far down that road, I'd like to make sure we're not reinventing a wheel (or risking reintroducing a problem that this exclusion is specifically avoiding).

In my mind, I would imagine we could:

• Find all defined tests
• Find all define test suites
• For each suite found, check its linker info to find the tests it contains
  • Remove those discrete tests from the list, as they are redundant with the test suite
  • Similarly, if a given test suite contains other suites as dependencies, remove those, as well (also redundant)

 

But, again, before I go too far down that path, is there already some other workaround for this problem? Or if not, does this sound like a viable solution? 

Thanks!

[Jim Kring]

Hi @turbophil.

Which version of Caraya are you using?

@Francois Normandin and team have been working hard and released a new version (1.1.0.119) just this week with at least one improvement to Test Suites. I don't know if what you're reporting was fixed or not.

Thanks for reporting this issue and I'm sure we'll get you an answer or a fix soon.

-Jim

 

[turbophil]

This is with the latest v1.1.0.119

Thanks for looking into it!

[Jim Kring]

Hey @turbophil. Just a thought. Would using nested tests (instead of test suites) meet your requirements?  You can call tests as subVIs inside of a VI that calls it's own "Define Test.vi" -- this would allow you to control the order of execution of those test subVIs.

[Francois Normandin]

@turbophil

The reason we exclude the Test Suites from the list of scanned VIs is that we need to create a Test Suite when finding tests programmatically (from the CLI or through VI Server calls). Since the Caraya 1.x architecture does not support nested Test Suites, we have to exclude them from the list, otherwise we'd throw errors.

It is not a fundamental choice, but rather a legacy decision to preserve backward compatibility when we upgraded from 0.6 to 1.x. The feature, I think, requires a modification of the Test Suite architecture, one that has a potentially rather large impact on the amount of testing we will need to do to ensure we don't break existing tests and workflows, so it was put aside for the moment. (This is a personal opinion, but as the main developer for this project, I'd rather first decouple the Test Manager (test engine) from the Caraya UI before adding support for nested Test Suites. That would diminish the risk of breaking existing code and be much less of a worry.)

I think Jim's suggestion is currently the best workaround : having a top-level Test that ensures your tests will run in parallel as you intend it to run.

It could perhaps be an interesting feature request to support a node that would find a Test Suite and run it programmatically, without wrapping it in another Test Suite. If you're interested in making this suggestion on Github, I think it could gain traction quickly: https://github.com/JKISoftware/Caraya