Scott Hanselman

Using dotnet watch test for continuous testing with .NET Core and XUnit.net

October 29, '16 Comments [21] Posted in Open Source
Sponsored By

When teaching .NET Core I do a lot of "dotnet new" Hello World demos to folks who've never seen it before. That has it's place, but I also wanted to show how easy it is to get setup with Unit Testing on .NET Core.

For this blog post I'm going to use the command line so you know there's nothing hidden, but you can also use Visual Studio or Visual Studio Code, of course. I'll start the command prompt then briefly move to Code.

Starting from an empty folder, I'll make a SomeApp folder and a SomeTests folder.

C:\example\someapp> dotnet new
C:\example\someapp> md ..\sometests && cd ..\sometests
C:\example\sometests> dotnet new -t xunittest

At this point I've got a HelloWorld app and a basic test but the two aren't related - They aren't attached and nothing real is being tested.

Tests are run with dotnet test, not dotnet run. Tests are libraries and don't have an entry point, so dotnet run isn't what you want.

c:\example>dotnet test SomeTests
Project SomeTests (.NETCoreApp,Version=v1.0) was previously compiled. Skipping compilation.
xUnit.net .NET CLI test runner (64-bit win10-x64)
Discovering: SomeTests
Discovered: SomeTests
Starting: SomeTests
Finished: SomeTests
=== TEST EXECUTION SUMMARY ===
SomeTests Total: 1, Errors: 0, Failed: 0, Skipped: 0, Time: 0.197s
SUMMARY: Total: 1 targets, Passed: 1, Failed: 0.

I'll open my test project's project.json and add a reference to my other project.

{
"version": "1.0.0-*",
"buildOptions": {
"debugType": "portable"
},
"dependencies": {
"System.Runtime.Serialization.Primitives": "4.1.1",
"xunit": "2.1.0",
"dotnet-test-xunit": "1.0.0-rc2-*"
},
"testRunner": "xunit",
"frameworks": {
"netcoreapp1.0": {
"dependencies": {
"Microsoft.NETCore.App": {
"type": "platform",
"version": "1.0.1"
},
"SomeApp": "1.0.0-*"
},
"imports": [
"dotnet5.4",
"portable-net451+win8"
]
}
}
}

I'll make a little thing to test in my App.

public class Calc {
public int Add(int x, int y) => x + y;
}

And add some tests.

public class Tests
{
[Fact]
public void TwoAndTwoIsFour()
{
var c = new Calc();
Assert.Equal(4, c.Add(2, 2));
}

[Fact]
public void TwoAndThreeIsFive()
{
var c = new Calc();
Assert.Equal(4, c.Add(2, 3));
}
}

Because the Test app references the other app/library, I can just make changes and run "dotnet test" from the command line. It will build both dependencies and run the tests all at once.

Here's the full output inclding both build and test.

c:\example> dotnet test SomeTests
Project SomeApp (.NETCoreApp,Version=v1.0) will be compiled because inputs were modified
Compiling SomeApp for .NETCoreApp,Version=v1.0

Compilation succeeded.
0 Warning(s)
0 Error(s)

Time elapsed 00:00:00.9814887
Project SomeTests (.NETCoreApp,Version=v1.0) will be compiled because dependencies changed
Compiling SomeTests for .NETCoreApp,Version=v1.0

Compilation succeeded.
0 Warning(s)
0 Error(s)

Time elapsed 00:00:01.0266293


xUnit.net .NET CLI test runner (64-bit win10-x64)
Discovering: SomeTests
Discovered: SomeTests
Starting: SomeTests
Tests.Tests.TwoAndThreeIsFive [FAIL]
Assert.Equal() Failure
Expected: 4
Actual: 5
Stack Trace:
c:\Users\scott\Desktop\testtest\SomeTests\Tests.cs(20,0): at Tests.Tests.TwoAndThreeIsFive()
Finished: SomeTests
=== TEST EXECUTION SUMMARY ===
SomeTests Total: 2, Errors: 0, Failed: 1, Skipped: 0, Time: 0.177s
SUMMARY: Total: 1 targets, Passed: 0, Failed: 1.

Oops, I made a mistake. I'll fix that test and run "dotnet test" again.

c:\example> dotnet test SomeTests
xUnit.net .NET CLI test runner (64-bit .NET Core win10-x64)
Discovering: SomeTests
Discovered: SomeTests
Starting: SomeTests
Finished: SomeTests
=== TEST EXECUTION SUMMARY ===
SomeTests Total: 2, Errors: 0, Failed: 0, Skipped: 0, Time: 0.145s
SUMMARY: Total: 1 targets, Passed: 1, Failed: 0.

I can keep changing code and running "dotnet test" but that's tedious. I'll add dotnet watch as a tool in my Test project's project.json.

{
"version": "1.0.0-*",
"buildOptions": {
"debugType": "portable"
},
"dependencies": {
"System.Runtime.Serialization.Primitives": "4.1.1",
"xunit": "2.1.0",
"dotnet-test-xunit": "1.0.0-rc2-*"
},
"tools": {
"Microsoft.DotNet.Watcher.Tools": "1.0.0-preview2-final"
},
"testRunner": "xunit",
"frameworks": {
"netcoreapp1.0": {
"dependencies": {
"Microsoft.NETCore.App": {
"type": "platform",
"version": "1.0.1"
},
"SomeApp": "1.0.0-*"
},

"imports": [
"dotnet5.4",
"portable-net451+win8"
]
}
}
}

Then I'll go back and rather than typing  "dotnet test" I'll type "dotnet watch test."

c:\example> dotnet watch test
[DotNetWatcher] info: Running dotnet with the following arguments: test
[DotNetWatcher] info: dotnet process id: 14064
Project SomeApp (.NETCoreApp,Version=v1.0) was previously compiled. Skipping compilation.
Project SomeTests (.NETCoreApp,Version=v1.0) will be compiled because inputs were modified
Compiling SomeTests for .NETCoreApp,Version=v1.0
Compilation succeeded.
0 Warning(s)
0 Error(s)
Time elapsed 00:00:01.1479348

xUnit.net .NET CLI test runner (64-bit .NET Core win10-x64)
Discovering: SomeTests
Discovered: SomeTests
Starting: SomeTests
Finished: SomeTests
=== TEST EXECUTION SUMMARY ===
SomeTests Total: 2, Errors: 0, Failed: 0, Skipped: 0, Time: 0.146s
SUMMARY: Total: 1 targets, Passed: 1, Failed: 0.
[DotNetWatcher] info: dotnet exit code: 0
[DotNetWatcher] info: Waiting for a file to change before restarting dotnet...

Now if I make a change to either the Tests or the projects under test it will automatically recompile and run the tests!

[DotNetWatcher] info: File changed: c:\example\SomeApp\Program.cs
[DotNetWatcher] info: Running dotnet with the following arguments: test
[DotNetWatcher] info: dotnet process id: 5492
Project SomeApp (.NETCoreApp,Version=v1.0) will be compiled because inputs were modified
Compiling SomeApp for .NETCoreApp,Version=v1.0

I'm able to do all of this with any text editor and a command prompt.

How do YOU test?


Sponsor: Do you deploy the same application multiple times for each of your end customers? The team at Octopus have taken the pain out of multi-tenant deployments. Check out their latest 3.4 release!

About Scott

Scott Hanselman is a former professor, former Chief Architect in finance, now speaker, consultant, father, diabetic, and Microsoft employee. He is a failed stand-up comic, a cornrower, and a book author.

facebook twitter subscribe
About   Newsletter
Sponsored By
Hosting By
Dedicated Windows Server Hosting by SherWeb
Saturday, 29 October 2016 07:45:45 UTC
It's a bit like the poor man's NCrunch, but I didn't know this existed. Thanks for that.

However, what strikes me the most is that you're not yet using one of the decent open-source assertion libraries like www.fluentassertions.com. Any reason for that?
Saturday, 29 October 2016 12:30:44 UTC
Thanks Scott! I've now given that a try and it seems to work great (someone had mentioned to me previously but I'd never given it a go).

In my setup, I used a slightly different way to reference the main project from the test project:


{
"version": "1.0.0-*",
"testRunner": "xunit",
"dependencies": {
"dotnet-test-xunit": "2.2.0-preview2-build1029",
"xunit": "2.2.0-beta2-build3300",
"Microsoft.AspNetCore.TestHost": "1.0.0",
"Newtonsoft.Json": "9.0.1",
"Shouldly": "2.8.2",
"AntMeehan.Budget.WebApi": {
"target": "project"
},

"Microsoft.Extensions.Configuration.UserSecrets": "1.0.0",
"NSubstitute": "2.0.0-rc"
},


In fact, I was trying to find a reference, and I found your blog entry from only a few weeks ago :)

Saturday, 29 October 2016 14:14:04 UTC
For those of you looking for additional options, I noticed there is a big difference between dotnet test -help and dotnet test --help. See github issue.
Saturday, 29 October 2016 15:00:42 UTC
Dennis - it's because I wanted to show the simplest possible thing in the fewest number of steps. One should not introduce too many concepts all at once when teaching, in my opinion.

Ant - thanks! I need to make sure I know the difference between those two ways.
Scott Hanselman
Saturday, 29 October 2016 18:27:01 UTC
trying to repeat, getting error: Unable to resolve 'SomeApp (>= 1.0.0)' for '.NETCoreApp,Version=v1.0'.

do someapp and sometests should be in different folders?
Alex
Saturday, 29 October 2016 23:21:22 UTC
Alex, yes, be sure to see the directory of the command prompt for the dotnet new commands
Peter Ritchie
Sunday, 30 October 2016 00:19:44 UTC
@Peter_Ritchie, what do you mean? In folder Projects, I created 2 folders: someapp and sometests, in project.json of sometests I linked someapp, but it doesn't see it.
Alex
Sunday, 30 October 2016 16:48:33 UTC
@Alex, I ran into the same problem as you. It worked for me by setting SomeApp to lower case (someapp). I also ran into a few other problems trying to follow things step by step as it seems there are a few details left out. I created a Gist demoing how I got everything working at https://gist.github.com/elijahgagne/7cd1a589bec2bca26253617b7fd5eb1b
Sunday, 30 October 2016 19:23:37 UTC
Thanks for the write up.

This is my first toe in the water with .NET core and I have a couple of basic questions:

1. It didn't look like you did a "dotnet restore" at any point in the process. I had to in order to get the examples to run. Is my environment setup differently?

2. Is there any way to get the output of dotnet test to shut up about the minutia? All I want to see is that everything passed or what tests failed. All the details of compilation are just clutter.
Andy Davis
Monday, 31 October 2016 01:41:41 UTC
@Elijah_W._Gagne, thank you. I could make it work using your Gist, probably Scott should make some corrections.
Monday, 31 October 2016 09:41:54 UTC
Thank you for this post!
Pavel
Tuesday, 01 November 2016 01:14:20 UTC
Thanks for the tutorial Scott. I've recently been working on an F# project and trying to set up all the infrastructure (VS Code, FAKE, Paket, .fsproj compilation order) has been pretty hard-core. And this is coming from a Scala/sbt guy, heh.

I have gotten NUnit working but I still haven't set up file watch with FAKE because they seem to have quite a few steps for macOS. Maybe I should have actually started with dotnet since I'm really doing experimental stuff anyway.
Yawar
Tuesday, 01 November 2016 16:59:10 UTC
I tried this and got an error message:
No executable found matching command "dotnet-watch"


Turns out to be a missing step. If you haven't previously installed the Watcher tool, you'll need to run
dotnet restore
in order to make the watch command.

A minor thing, but perhaps that will save someone else a few minutes of head-scratching. ;-)
Tuesday, 01 November 2016 17:08:49 UTC
From examples a while ago, I saw that one should have the global.json file in a parent folder so that the test project can reference the main project. I've been doing this on projects. Is this required or optional? What difference does it make?
Tuesday, 01 November 2016 18:39:36 UTC
Thanks Scott I have been looking for this type of test runner.
Works like a dream on my windows environment but am having issues on Mac.
Ran into a known issue when targeting net451 where dotnet test cannot find dotnet-test-xunit.exe
Solved that by having the test project target netcoreapp1.0

Now cannot get the watcher to work.
(dotnet restore gave me Microsoft.DotNet.Watcher.Tools 1.0.0-preview3-final)
The specified framework 'Microsoft.NETCore.App', version '1.1.0-preview1-001100-00' was not found.

I am considering giving .net core a break until .netstandard 2.0 is all ready and the tooling settles down.
Bruce
Tuesday, 01 November 2016 18:59:21 UTC
Got it working by installing .NET Core 1.1 Preview 1

Probably needs a mention that the watcher requires the latest and greatest version.
bruce
Wednesday, 02 November 2016 03:09:28 UTC
it's better to remove some spam messages here (the same in previous post).
Wednesday, 02 November 2016 09:50:38 UTC
@ThatBlairGuy I was getting
No executable found matching command "dotnet-watch"
too, and wasted some time following a trail of suggestions involving alternate ways to specify the tool | dependency | version and to check that the package had been correctly downloaded and installed and that the resulting executable was available.

Turns out my command shell was in the wrong folder!

\example> dotnet test sometests
runs the tests in the sometests project, but
\example> dotnet watch test sometests
fails with this misleading message.
\example> cd sometests
\example\sometests> dotnet watch test
works as described.
brian
Thursday, 03 November 2016 10:32:35 UTC
Nice write-up, Scott.
The other week I hooked up VSTS with unit tests for .NET Core (blog post), which now automatically handles all builds and tests for my CI scenarios. It's pretty sweet to get these results (trx) integrated with MSBuild to get the full dashboards in VSTS.

If using tests with the above approach you mention, do you know if the integration scenarios with MSBuild/VSTS/TFS et al works the same way (or is possible)?

Cheers,
Tobi.
Thursday, 10 November 2016 21:06:12 UTC
Being new to dotnet core, I struggled a little with getting the example to run. The problem is that dependencies are CaseSensitive, so when you run "C:\example\someapp> dotnet new", adding the dependency as "SomeApp": "1.0.0-*" gives "Unable to resolve 'SomeApp (>= 1.0.0)'". Changing the dependency name to "someapp" resolves the problem.

Perhaps it would be useful to update the article to reflect this?
Johannes Brodwall
Friday, 11 November 2016 07:26:35 UTC
.NET core is like going back in time
Paul
Comments are closed.

Disclaimer: The opinions expressed herein are my own personal opinions and do not represent my employer's view in any way.