Organizing Tests With xUnit Traits

Organizing Minion

Once you start to have a larger number of tests it can be important to be able to break them down into different categories or groupings. From a functionality perspective this allows you to only run a subset of tests. They can also help to provide clarity or insight to your test code, by replacing comments or bloated test names with code based cues to preconditions or domain of the tests.

Both NUnit and MSTest make use of the Category verbage. NUnit using the Category Attribute and MSTest using the TestCategory Attribute. You might expect xUnit to also have something named similarly, but instead they have chosen the Trait attribute.

Traits

If you atre used to using categories from other frameworks, the Trait attribute is slightly confusing when you first look at it.
Instead of:

[Category("MyCategoryName")]
public void ...

The trait attribute uses a name and value pair

[Trait("Category","MyCategoryName")]
public void ...

When I first saw this I wasn’t sure if the name property value had any significance, i.e. is it a set of magic strings I ended up peeking through the framework code on GitHub to confirm that the name parameter is up to user preference. If you are familiar with NUnit then it’s like a hybrid of the category and property attributes.

Test Runner View

Another benefit to organizing tests using traits is that when using a GUI based test runner, like the test runner in Visual Studio the tests can be sorted accordingly.
For Example:

[Fact]
[Trait("Category","Unit")]
public void Test1(){
    ...}

[Fact]
[Trait("Category","Integration")]
public void Test2(){
    ...}

[Fact]
[Trait("Category","UI")]
public void Test3(){
    ...}

Results in the following display where you can run tests individually or by category.
test view by categories

On The Commandline

When executing tests from the commandline you can run only a specific category or cateogries.

xunit.console.exe ... -trait "Category=Unit"
-or-
xunit.console.exe ... -trait "Category=Integration" -trait "Category=UI"

Exclude a category or categories

xunit.console.exe ... -notrait "Category=Integration"
-or-
xunit.console.exe ... -notrait "Category=UI" -notrait "Category="UI"

Or you can mix and match to your hearts content.

Custom Traits

I found traits to feel a little messy, it felt very flexible but at the price of being less inutitive to people that are not already familiar with xUnit. It may not be a huge issue but there seems to be a higher liklihood of misspellings or typo’s causing tests to not be run. Luckily xUnit supports creating custom traits.

In ealrier versions it was as simple as subclassing the trait attribute but in later versions that class has been sealed. The current process involves implenting the ITraitAttribute ITraitDiscoverer interfaces for your custom trait. The xUnit Samples repo on GitHub provides sample code for Category.

The only issue is the Visual Studio and Resharper test runners do not use the newer process to discover traits. This means that you cannot currently visually group test by custom traits until they update their test runners.

Friendly xUnit Categories

If you want to run tests by trait, both key and value of a trait must be specified on the commandline. This makes sense when the trait name is generic like category. What if you want something more specific like marking tests related to bugs like [Trait(“Category”,”Bug”)]. It looks fine, but the problem is you can’t run only tests for a specific bug unless you add another attribute like [Trait(“Bug”,”8675309″)].

To make the process more friendly and to add a little flexibilty/functionality check out my xUnit.Categories project on GitHub.

Or use the nuget package.

Example Bug trait

        [Fact]
        [Bug]
        public void TestBug()
        {
            throw new NotImplementedException("I'm a bug");
        }

        [Fact]
        [Bug("777")]
        public void TestBugWithId()
        {
            throw new NotImplementedException("I've got your number");
}

Using this single attribute you get descriptive information and flexibility when running tests.
You can run all tests marked as Bugs

xunit.console.exe ... -trait "Category=Bug"

or get more granular

xunit.console.exe ... -trait "Bug=777"
  • Robert Werner

    Brendan, I’ve just implemented 2 custom traits: Url & UseCase. Together they refer a person to a specific JIRA page and section within. Having this in the test code is valuable on its own. However, I’m wondering if programmatically one can retrieve the values associated with these custom traits?

    • Hi Robert,
      You should be able to retrieve the values of your traits programmatically, the xUnit test runners have to do this so you should be able to leverage some of the existing code to extract them. Are you looking to do this inside your test code or in an app that would extra data from test assemblies? I’ll try to get some example code together.

      Thanks for reading!
      B.

      • Robert Werner

        Thanks for responding, Brendan. There are two places where I’d like to be able to retrieve the custom attributes I’m decorating each test with:

        1. Within my test code. Right now in my test code’s base class I’m simply outputting “Starting ABC test” and “Ending ABC test”. It would be more useful if I could also output the Url & UseCase attribute values. To that end, I have posted this: https://forums.asp.net/p/2113047/6107319.aspx

        2. Within Resharper’s Test Explorer it would be nice to be able to see these custom attributes and/or group by then. This is not as high a priority as #1.

        https://uploads.disquscdn.com/images/0b96cdd2dd0504ecb6395a49b05398dd61e7aaa93b29547b510ebfc34b65d950.jpg

        • In your base class try using the Traits dictionary on the TestCase property. The code below works with my custom traits. If it doesnt for you maybe post a gist of sample code and I can take a look.
          var theUrl= test.TestCase.Traits[“Url”].First();

          • Robert Werner

            It says there are 0 Traits.

            I’ve got pretty close with this code: test.TestCase.TestMethod.Method.GetCustomAttributes(typeof(Xunit.TraitExtensibility.UrlAttribute))

            That displays these contents:

            Attribute: {Xunit.TraitExtensibility.UrlAttribute}
            AttributeData: {[Xunit.TraitExtensibility.UrlAttribute(“https://jira.rakuten-it.com/jira/browse/REM-18497”)]}

            But then I get stuck. My lack of experience in LINQ is showing, I fear. 🙁

          • If there are 0 traits I think the issue may be with your custom trait. Looking at the code on the link you sent try having the TraitDiscoverer attribute on the UrlAttribute class use fully qualified names. Assuming XunitUrl is the assembly name:
            [TraitDiscoverer(“XunitUrl.UrlDiscoverer”, “XunitUrl”)]

            Also once resharper updates its test runner to use the trait discoverers instead looking for the Trait attribute then you would be able to visually group them like you wanted. Not sure if or when that will happen.

          • Robert Werner

            Brendan, THANK YOU for your patience with me!!! I do remember seeing the last part of this posting: https://dennymichael.net/2015/01/23/how-to-extendmigrate-traitattribute-to-xunit-v2-and-why-has-become-sealed/ but I didn’t heed too much attention to it.

            My original TraitDiscoverer attribute was this: [TraitDiscoverer(“UrlDiscoverer”, “XunitUrl”)]

            Now it looks like this: [TraitDiscoverer(“Xunit.TraitExtensibility.UrlDiscoverer”, “Xunit.TraitExtensibility”)]

            and it works just like you said!!! The simple/magic code for me is this: test.TestCase.Traits[“Url”][0]

            Once again, thank you!!!

            P.S. Nothing appears to have changed in the Resharper Test Explorer but that’s okay as 90% of what I wanted now I have access to.