Scott Hanselman

Automatic Unit Testing in .NET Core plus Code Coverage in Visual Studio Code

March 22, '18 Comments [13] Posted in DotNetCore | Open Source
Sponsored By

I was talking to Toni Edward Solarin on Skype yesterday about his open source spike (early days) of Code Coverage for .NET Core called "coverlet." There's a few options out there for cobbling together .NET Core Code Coverage but I wanted to see if I could use the lightest tools I could find and make a "complete" solution for Visual Studio Code that would work for .NET Core cross platform. I put my own living spike of a project up on GitHub.

Now, keeping in mind that Toni's project is just getting started and (as of the time of this writing) currently supports line and method coverage, and branch coverage is in progress, this is still a VERY compelling developer experience.

Using VS Code, Coverlet, xUnit, plus these Visual Studio Code extensions

Here's what we came up with.

Auto testing, code coverage, line coloring, test explorers, all in VS Code

There's a lot going on here but take a moment and absorb the screenshot of VS Code above.

  • Our test project is using xunit and the xunit runner that integrates with .NET Core as expected.
    • That means we can just "dotnet test" and it'll build and run tests.
  • Added coverlet, which integrates with MSBuild and automatically runs when you "dotnet test" if you "dotnet test /p:CollectCoverage=true"
    • (I think this should command line switch should be more like --coverage" but there may be an MSBuild limitation here.)

I'm interested in "The Developer's Inner Loop." . That means I want to have my tests open, my code open, and as I'm typing I want the solution to build, run tests, and update code coverage automatically the way Visual Studio proper does auto-testing, but in a more Rube Goldbergian way. We're close with this setup, although it's a little slow.

Coverlet can produce opencover, lcov, or json files as a resulting output file. You can then generate detailed reports from this. There is a language agnostic VS Code Extension called Coverage Gutters that can read in lcov files and others and highlight line gutters with red, yellow, green to show test coverage. Those lcov files look like this, showing file names, file numbers, coverage, and number of exceptions.

SF:C:\github\hanselminutes-core\hanselminutes.core\Constants.cs
DA:3,0
end_of_record
SF:C:\github\hanselminutes-core\hanselminutes.core\MarkdownTagHelper.cs
DA:21,5
DA:23,5
DA:49,5

I should be able to pick the coverage file manually with the extension, but due to a small bug, it's easier to just tell Coverlet to generate a specific file name in a specific format.

dotnet test /p:CollectCoverage=true /p:CoverletOutputFormat=lcov /p:CoverletOutput=./lcov.info .\my.tests

The lcov.info files then watched by the VSCode Coverage Gutters extension and updates as the file changes if you click watch in the VS Code Status Bar.

You can take it even further if you add "dotnet watch test" which will compile and re-run tests if code changes:

dotnet watch --project .\my.tests test /p:CollectCoverage=true /p:CoverletOutputFormat=lcov /p:CoverletOutput=./lcov.info 

I can run "WatchTests.cmd" in another terminal, or within the VS Code integrated terminal.

tests automatically running as code changes

NOTE: If you're doing code coverage you'll want to ensure your tests and tested assembly are NOT the same file. You might be able to get it to work but it's easier to keep things separate.

Next, add in the totally under appreciated .NET Core Test Explorer extension (this should have hundreds of thousands of downloads - it's criminal) to get this nice Test Explorer pane:

A Test Explorer tree view in VS Code for NET Core projects

Even better, .NET Test Explorer lights up some "code lens" style interfaces over each test as well as a green checkmark for passing tests. Having "debug test" available for .NET Core is an absolute joy.

Check out "run test" and "debug test"

Finally we make some specific improvements to the .vscode/tasks.json file that drives much of VS Code's experience with our app. The "BUILD" label is standard but note both the custom "test" and "testwithcoverage" labels, as well as the added group with kind: "test."

{
    "version": "2.0.0",
    "tasks": [
        {
            "label": "build",
            "command": "dotnet",
            "type": "process",
            "args": [
                "build",
                "${workspaceFolder}/hanselminutes.core.tests/hanselminutes.core.tests.csproj"
            ],
            "problemMatcher": "$msCompile",
            "group": {
                "kind": "build",
                "isDefault": true
            }
        },
        {
            "label": "test",
            "command": "dotnet",
            "type": "process",
            "args": [
                "test",
                "${workspaceFolder}/hanselminutes.core.tests/hanselminutes.core.tests.csproj"
            ],
            "problemMatcher": "$msCompile",
            "group": {
                "kind": "test",
                "isDefault": true
            }
        },
        {
            "label": "test with coverage",
            "command": "dotnet",
            "type": "process",
            "args": [
                "test",
                "/p:CollectCoverage=true",
                "/p:CoverletOutputFormat=lcov",
                "/p:CoverletOutput=./lcov.info",
                "${workspaceFolder}/hanselminutes.core.tests/hanselminutes.core.tests.csproj"
            ],
            "problemMatcher": "$msCompile",
            "group": {
                "kind": "test",
                "isDefault": true
            }
        },
    ]
}

This lets VS Code know what's for building and what's for testing, so if I use the Command Palette to "Run Test" then I'll get this dropdown that lets me run tests and/or update coverage manually if I don't want the autowatch stuff going.

Test or Test with Coverage

Again, all this is just getting started but I've applied it to my Podcast Site that I'm currently rewriting and the experience is very smooth!

Here's a call to action for you! Toni is just getting started on Coverlet and I'm sure he'd love some help. Head over to the Coverlet github and don't just file issues and complain! This is an opportunity for you to get to know the deep internals of .NET and create something cool for the larger community.

What are your thoughts?


Sponsor: Get the latest JetBrains Rider for debugging third-party .NET code, Smart Step Into, more debugger improvements, C# Interactive, new project wizard, and formatting code in columns.

About Scott

Scott Hanselman is a former professor, former Chief Architect in finance, now speaker, consultant, father, diabetic, and Microsoft employee. He is a failed stand-up comic, a cornrower, and a book author.

facebook twitter subscribe
About   Newsletter
Sponsored By
Hosting By
Dedicated Windows Server Hosting by SherWeb
Thursday, 22 March 2018 19:57:08 UTC
Just goes to show it's who you know, not what you know. I've been working on something similar (only it includes classic .net and Mono as well) for the last few months, and I haven't gone viral.
Thursday, 22 March 2018 20:05:14 UTC
My question is a philosophical one, Scott, and it's not sarcastic: being mindful of its "lightweight cousin" role, are there really any of the richer and bulkier features of Visual Studio that would be off-limits in your eyes for bolting onto VS Code?
Cory
Thursday, 22 March 2018 20:07:12 UTC
While this is interesting, I can't help but long for this same ease of use when you aren't running application on the same machine as the code and instead are building / running the code inside a Docker container.
wdspider
Thursday, 22 March 2018 20:52:50 UTC
Hmm, this seems pretty interesting. I'm going to give it a shot.

@Steve Gilham,

Have you tried posting a link to this on social media, etc with hashtags? I haven't found anything on Twitter at least that references your work, but I have seen quite a few things about coverlet. Try throwing something on Twitter and I can guarantee someone will at least pick up on it :).

At the same time, this probably isn't the right context to speak about virality 😎.
Thursday, 22 March 2018 21:53:51 UTC
Steve Gilham - I'm looking for you on Twitter and your blog and I don't know how I'd discover your awesome work! I'll take a look!

wdspider - I don't see why these tests couldn't run in a docker container as well!

Cory - I think VS "Full" shines with large UI things like Application Insights, and VS Code shines at the text/terminal level.
Friday, 23 March 2018 15:04:34 UTC
Wow, amazing work and thank you Toni, and thanks Scott for the recommendation. I was able to get code coverage reports integrated into Linux Docker builds within a CI/CD pipeline in just a few minutes this morning thanks to your work.
Matt Anderson
Sunday, 25 March 2018 16:40:39 UTC
I recently tried to build a lightweight VS2017 with VSCode (for macOS) and code coverage was one thing that I found myself missing (I'm used to dotCover). This looks like a nice option for something similar in vscode. I'll give it a try, thanks Scott.
Monday, 26 March 2018 14:17:07 UTC
Scott -

No doubt the tests are capable of running inside a Docker container. The question is more "How?". Where is the sample dockerfile that I can download and add to my project that will enable the project to run self-contained within Docker while still providing me with the ease of dotnet watch / debugger support / etc. as a local machine project?

The concept of developing / deploying / executing my app in Docker is awesome. The tooling to help me achieve this? Not so much.
wdspider
Tuesday, 27 March 2018 07:20:41 UTC
Thanks for the links to the various tools!
If you want to create a coverage report (e.g. on a build server) you can use ReportGenerator.
It can create a HTML report based on the OpenCover output which Coverlet is able to generate.
Thursday, 29 March 2018 06:45:45 UTC
19%?!
Better work on that code coverage man!
Serban
Friday, 30 March 2018 11:35:17 UTC
looking forward to try!
Tuesday, 03 April 2018 19:30:27 UTC
Very useful code, thanks a lot Scott, your tutorials are very useful and informative.
Friday, 06 April 2018 17:47:19 UTC
great setup , thanks. works well on my C# parser generator project https://github.com/b3b00/csly
Comments are closed.

Disclaimer: The opinions expressed herein are my own personal opinions and do not represent my employer's view in any way.