Scott Hanselman

Proper benchmarking to diagnose and solve a .NET serialization bottleneck

February 20, '15 Comments [23] Posted in ASP.NET | Learning .NET | Open Source
Sponsored By

From http://adrianotto.com/2010/08/dev-null-unlimited-scale/Here's a few comments and disclaimers to start with. First, benchmarks are challenging. They are challenging to measure, but the real issue is that often we forget WHY we are benchmarking something. We'll take a complex multi-machine financial system and suddenly we're hyper-focused on a bunch of serialization code that we're convinced is THE problem. "If I can fix this serialization by writing a 10,000 iteration for loop and getting it down to x milliseconds, it'll be SMOOOOOOTH sailing."

Second, this isn't a benchmarking blog post. Don't point this blog post and say "see! Library X is better than library Y! And .NET is better than Java!" Instead, consider this a cautionary tale, and a series of general guidelines. I'm just using this anecdote to illustrate these points.

  • Are you 100% sure what you're measuring?
  • Have you run a profiler like the Visual Studio profiler or DotTrace?
  • Are you considering warm-up time? Throwing out outliers? Are your results statistically significant?
  • Are the libraries you're using optimized for your use case? Are you sure what your use case is?

A bad benchmark

A reader sent me a email recently with concern of serialization in .NET. They had read some very old blog posts from 2009 about perf that included charts and graphs and did some tests of their own. They were seeing serialization names (of tens of thousands of items) over 700ms and sizes nearly 2 megs. The tests included serialization of their typical data structures in both C# and Java across a number of different serialization libraries and techniques. Techniques included their company's custom serialization, .NET binary DataContract serialization, as well as JSON.NET. One serialization format was small (1.8Ms for a large structure) and one was fast (94ms) but there was no clear winner. This reader was at their wit's end and had decided, more or less, that .NET must not be up for the task.

To me, this benchmark didn't smell right. It wasn't clear what was being measured. It wasn't clear if it was being accurately measured, but more specifically, the overarching conclusion of ".NET is slow" wasn't reasonable given the data.

Hm. So .NET can't serialize a few tens of thousands of data items quickly? I know it can.

Related Links: Create benchmarks and results that have value and Responsible benchmarking by @Kellabyte

I am no expert, but I poked around at this code.

First: Are we measuring correctly?

The tests were using DateTime.UtcNow which isn't advisable.

startTime = DateTime.UtcNow;
resultData = TestSerialization(foo);
endTime = DateTime.UtcNow;

Do not use DateTime.Now or DateTime.Utc for measuring things where any kind of precision matters. DateTime doesn't have enough precision and is said to be accurate only to 30ms.

DateTime represents a date and a time. It's not a high-precision timer or Stopwatch.

As Eric Lippert says:

In short, "what time is it?" and "how long did that take?" are completely different questions; don't use a tool designed to answer one question to answer the other.

And as Raymond Chen says:

"Precision is not the same as accuracy. Accuracy is how close you are to the correct answer; precision is how much resolution you have for that answer."

So, we will use a Stopwatch when you need a stopwatch. In fact, before I switch the sample to Stopwatch I was getting numbers in milliseconds like 90,106,103,165,94, and after Stopwatch the results were 99,94,95,95,94. There's much less jitter.

Stopwatch sw = new Stopwatch();
sw.Start();

// stuff

sw.Stop();

You might also want to pin your process to a single CPU core if you're trying to get an accurate throughput measurement. While it shouldn't matter and Stopwatch is using the Win32 QueryPerformanceCounter internally (the source for the .NET Stopwatch Class is here) there were some issues on old systems when you'd start on one proc and stop on another.

// One Core
var p = Process.GetCurrentProcess();
p.ProcessorAffinity = (IntPtr)1;

If you don't use Stopwatch, look for a  simple and well-thought-of benchmarking library.

Second: Doing the math

In the code sample I was given, about 10 lines of code were the thing being measured, and 735 lines were the "harness" to collect and display the data from the benchmark. Perhaps you've seen things like this before? It's fair to say that the benchmark can get lost in the harness.

Have a listen to my recent podcast with Matt Warren on "Performance as a Feature" and consider Matt's performance blog and be sure to pick up Ben Watson's recent Book called "Writing High Performance .NET Code".

Also note that Matt is currently exploring creating a mini-benchmark harness on GitHub. Matt's system is rather promising and would have a [Benchmark] attribute within a test.

Considering using an existing harness for small benchmarks. One is SimpleSpeedTester from Yan Cui. It makes nice tables and does a lot of the tedious work for you. Here's a screenshot I stole borrowed from Yan's blog.

image11

Something a bit more advanced to explore is HdrHistogram, a library "designed for recoding histograms of value measurements in latency and performance sensitive applications." It's also on GitHub and includes Java, C, and C# implementations.

PercentileHistogramExample

And seriously. Use a profiler.

Third: Have you run a profiler?

Use the Visual Studio Profiler, or get a trial of the Redgate ANTS Performance Profiler or the JetBrains dotTrace profiler.

Where is our application spending its time? Surprise I think we've all seen people write complex benchmarks and poke at a black box rather than simply running a profiler.

Visual Studio Profiler

Aside: Are there newer/better/understood ways to solve this?

This is my opinion, but I think it's a decent one and there's numbers to back it up. Some of the .NET serialization code is pretty old, written in 2003 or 2005 and may not be taking advantage of new techniques or knowledge. Plus, it's rather flexible "make it work for everyone" code, as opposed to very narrowly purposed code.

People have different serialization needs. You can't serialize something as XML and expect it to be small and tight. You likely can't serialize a structure as JSON and expect it to be as fast as a packed binary serializer.

Measure your code, consider your requirements, and step back and consider all options.

Fourth: Newer .NET Serializers to Consider

Now that I have a sense of what's happening and how to measure the timing, it was clear these serializers didn't meet this reader's goals. Some of are old, as I mentioned, so what other newer more sophisticated options exist?

There's two really nice specialized serializers to watch. They are Jil from Kevin Montrose, and protobuf-net from Marc Gravell. Both are extraordinary libraries, and protobuf-net's breadth of target framework scope and build system are a joy to behold. There are also other impressive serializers in including support for not only JSON, but also JSV and CSV in ServiceStack.NET.

Protobuf-net - protocol buffers for .NET

Protocol buffers are a data structure format from Google, and protobuf-net is a high performance .NET implementation of protocol buffers. Think if it like XML but smaller and faster. It also can serialize cross language. From their site:

Protocol buffers have many advantages over XML for serializing structured data. Protocol buffers:

  • are simpler
  • are 3 to 10 times smaller
  • are 20 to 100 times faster
  • are less ambiguous
  • generate data access classes that are easier to use programmatically

It was easy to add. There's lots of options and ways to decorate your data structures but in essence:

var r = ProtoBuf.Serializer.Deserialize<List<DataItem>>(memInStream);

The numbers I got with protobuf-net were exceptional and in this case packed the data tightly and quickly, taking just 49ms.

JIL - Json Serializer for .NET using Sigil

Jil s a Json serializer that is less flexible than Json.net and makes those small sacrifices in the name of raw speed. From their site:

Flexibility and "nice to have" features are explicitly discounted in the pursuit of speed.

It's also worth pointing out that some serializers work over the whole string in memory, while others like Json.NET and DataContractSerializer work over a stream. That means you'll want to consider the size of what you're serializing when choosing a library.

Jil is impressive in a number of ways but particularly in that it dynamically emits a custom serializer (much like the XmlSerializers of old)

Jil is trivial to use. It just worked. I plugged it in to this sample and it took my basic serialization times to 84ms.

result = Jil.JSON.Deserialize<Foo>(jsonData);

Conclusion: There's the thing about benchmarks. It depends.

What are you measuring? Why are you measuring it? Does the technique you're suing handle your use case? Are you serializing one large object or thousands of small ones?

James Newton-King made this excellent point to me:

"[There's a] meta-problem around benchmarking. Micro-optimization and caring about performance when it doesn't matter is something devs are guilty of. Documentation, developer productivity, and flexibility are more important than a 100th of a millisecond."

In fact, James pointed out this old (but recently fixed) ASP.NET bug on Twitter. It's a performance bug that is significant, but was totally overshadowed by the time spent on the network.

Thanks to Marc Gravell and James Newton-King for their time helping with this post.

What are your benchmarking tips and tricks? Sound off in the comments!

About Scott

Scott Hanselman is a former professor, former Chief Architect in finance, now speaker, consultant, father, diabetic, and Microsoft employee. He is a failed stand-up comic, a cornrower, and a book author.

facebook twitter subscribe
About   Newsletter
Sponsored By
Hosting By
Dedicated Windows Server Hosting by ORCS Web

OmniSharp - Making cross-platform .NET a reality, and a pleasure

November 27, '14 Comments [60] Posted in ASP.NET | Open Source
Sponsored By

In case you missed it, make sure to read Announcing .NET 2015 - .NET as Open Source, .NET on Mac and Linux, and Visual Studio Community because there's been some big stuff going on.

Here's the summary of the .NET 2015 Wave of awesomeness.

The other thing I wanted to talk about is a newly organized group of technologies called OmniSharp. Just to be sure there's no confusion, OmniSharp isn't a Microsoft project. While there are two Microsoft folks on the team of 8 or so, we are working on it as community members, not in an official capacity.

I "launched" this project in my talk at the Connect() mini-conference in New York a few weeks back. You can watch that video here on Channel 9 now if you like. However, the technologies around and under OmniSharp have been around for years...like over a decade!

As a team and a community we pulled together a bunch of projects and plugins, got organized, and created https://github.com/omnisharp and http://www.omnisharp.net. Jonathan Channon has a great overview blog post you should check out that talks about how Jason Imison created OmniSharpServer which is an...

HTTP wrapper around NRefactory allowing C# editor plugins to be written in any language. NRefactory is the C# analysis library used in the SharpDevelop and MonoDevelop IDEs. It allows applications to easily analyze both syntax and semantics of C# programs. It is quite similar to Microsoft's Roslyn project; except that it is not a full compiler – NRefactory only analyzes C# code, it does not generate IL code.

OmniSharp runs as its own process and runs a local Nancy-based web api that your editor of choice talks to. If you have an editor that you like to use, why not get involved and make a plugin? Perhaps for Eclipse?

We now have plugins for these editors:

  • Sublime
  • Brackets from Adobe
  • Atom from GitHub
  • Emacs
  • Vim

And these work on (so far) all platforms! It's about choice. We wanted to bring more than autocomplete (which is basically "I think you typed that before") to your editor, instead we want actual type-smart intellisense, as well as more sophisticated features like refactoring, format document, and lots of other stuff you'd expect only to see in Visual Studio.

We also brought in the Sublime Kulture package which gives Sublime users support for ASP.NET 5 (formerly ASP.NET vNext), so they can launch Kestrel (our libuv based local webserver), run Entity Framework migrations, and other arbitrary commands from within Sublime.

.NET in Sublime, in Vim, in Brackets, in Atom, and everywhere else, cross-platform

Here's OmniSharp running in emacs on my Windows machine. The emacs setup (here is an example) is a little more complex than the others, but it also gives emacs folks an extreme level of control. Note that I had to launch the OmniSharp server manually for emacs, while it launches automatically for the other editors.

image

Here is an ASP.NET MVC app running in Sublime. The Sublime OmniSharp package output can be seen in the debug console there (Ctrl+~ to see it).

image

OmniSharp is in very active development. We are looking at bringing in Roslyn, using the new ASP.NET Design Time Host, and improving robustness. It's not perfect, but it's pretty darn cool. There's lots of details in Jonathan's writeup with great animated gifs showing features. Also note that we have a Yeoman generator for ASP.NET that can get you started when creating ASP.NET 5 apps on Mac or Linux. The yeoman generator can create Console apps, MVC apps, and NancyFx apps.

You can get started at http://omnisharp.net.  See you there!

About Scott

Scott Hanselman is a former professor, former Chief Architect in finance, now speaker, consultant, father, diabetic, and Microsoft employee. He is a failed stand-up comic, a cornrower, and a book author.

facebook twitter subscribe
About   Newsletter
Sponsored By
Hosting By
Dedicated Windows Server Hosting by ORCS Web

ASP.NET 5 (vNext) Work in Progress - Exploring TagHelpers

November 20, '14 Comments [143] Posted in ASP.NET | ASP.NET MVC
Sponsored By

TagHelpers are a new feature of ASP.NET 5 (formerly and colloquially ASP.NET vNext) but it's taken me (and others) some time to fully digest them and what they mean.

Note that this, and all of ASP.NET 5 is a work in progress. TagHelpers can and will change. There is NO tooling support in Visual Studio for them, as they are changing day to day, so just be aware. That's why this post (and series is called Work in Progress.)

Historically we've used HtmlHelpers within a Razor View, so when you wanted a Label or a TextBox you'd do this. This is from the ASP.NET 5 Starter Web example.

<li>@Html.ActionLink("Home", "Index", "Home")</li>

There you have some HTML, then we start some C# with @ and then switch out. It's inline C#, calling a function that will return HTML.

Here's the same thing, using a TagHelper.

<li><a controller="Home" action="Index">Home</a></li>

The source for TagHelpers is (as with all ASP.NET source) up on GitHub, here. This is an anchor, A, so it'll be in AnchorTagHelper. The code is very simple, in fact, gets a collection of attributes and decides which to act upon.

In this case, "controller" and "action" are not HTML5 attributes, but rather ones that ASP.NET is looking for.

Question for You - Would you rather have these attributes and ones like them (including your own) be prefixed? Perhaps asp:controller or asp-controller? That's an open issue you can comment on! You could do [HtmlAttributeName("asp:whatever")] on a property or [TagName("foo")] for a tag if you liked.

How do these attributes get mapped to a TagHelper? Well, an attribute name is mapped directly to a C# property and automatically injected. See how AnchorTagHelper has public properties Action and Controller?

It's important to note that this isn't the second coming of WebForms controls, while the possible asp:foo syntax may look familiar (even though a prefix is optional.) This is more like syntactic sugar that gives you a compact way to express your intent. It doesn't give you any "control lifecycle" or anything like that.

Personally, I'd love to see them look different in the editor. For example, rather than

Tag Helpers

I'd like to see italics, or maybe a desaturation to show what's server-side and what's not, which will be super important if I'm NOT using a prefix to distinguish my attributes.

Tag Helpers, desaturated

The code below in this Before and After results in the same HTML and the same behavior. A nice aspect of TagHelpers it that you avoid the context switch from markup to C#.

Here is another example, a login partial form, before...

@using System.Security.Principal

@if (User.Identity.IsAuthenticated)
{
using (Html.BeginForm("LogOff", "Account", FormMethod.Post, new { id = "logoutForm", @class = "navbar-right" }))
{
@Html.AntiForgeryToken()
<ul class="nav navbar-nav navbar-right">
<li>
@Html.ActionLink("Hello " + User.Identity.GetUserName() + "!", "Manage", "Account", routeValues: null, htmlAttributes: new { title = "Manage" })
</li>
<li><a href="javascript:document.getElementById('logoutForm').submit()">Log off</a></li>
</ul>
}
}
else
{
<ul class="nav navbar-nav navbar-right">
<li>@Html.ActionLink("Register", "Register", "Account", routeValues: null, htmlAttributes: new { id = "registerLink" })</li>
<li>@Html.ActionLink("Log in", "Login", "Account", routeValues: null, htmlAttributes: new { id = "loginLink" })</li>
</ul>
}

and after...with the Microsoft.AspNet.Mvc.TagHelpers package added in project.json and then @addtaghelper "MyAssemblyName" in either your ViewStart.cshtml to get this in all views, or separately within a single view page.

@using System.Security.Principal

@if (User.Identity.IsAuthenticated)
{
<form method="post" controller="Account" action="LogOff" id="logoutForm" class="navbar-right">
<ul class="nav navbar-nav navbar-right">
<li>
<a controller="Account" action="Manage" title="Manage">Hello @User.Identity.GetUserName()!</a>
</li>
<li><a href="javascript:document.getElementById('logoutForm').submit()">Log off</a></li>
</ul>
</form>
}
else
{
<ul class="nav navbar-nav navbar-right">
<li><a id="registerLink" controller="Account" action="Register">Register</a></li>
<li><a id="loginLink" controller="Account" action="Login">Log in</a></li>
</ul>
}

This makes for much cleaner markup-focused Views. Note that this Sample is a spike that Damian Edwards has on his GitHub, but you have TagHelpers in the Beta 1 build included with Visual Studio 2015 preview or OmniSharp. Get involved!

Remember also to check out http://www.asp.net/vnext and subscribe to my YouTube Channel and this playlist of the ASP.NET Weekly Community Standup. In this episode we talked about TagHelpers in depth!

Related Posts

About Scott

Scott Hanselman is a former professor, former Chief Architect in finance, now speaker, consultant, father, diabetic, and Microsoft employee. He is a failed stand-up comic, a cornrower, and a book author.

facebook twitter subscribe
About   Newsletter
Sponsored By
Hosting By
Dedicated Windows Server Hosting by ORCS Web

Announcing .NET 2015 - .NET as Open Source, .NET on Mac and Linux, and Visual Studio Community

November 12, '14 Comments [200] Posted in ASP.NET | ASP.NET MVC | Open Source | VS2015
Sponsored By

It's happening. It's the reason that a lot of us came to work for Microsoft, and I think it's both the end of an era but also the beginning of amazing things to come.

The .NET 2015 wave of releases is upon us. Here's what's happening and we announced it today in New York. There's a lot here, so drink it all in slowly.

Be sure to check out all the blog posts I'm linking to at the end, but here's my personal rollup and take on the situation.

  • We are serious about open source and cross platform.
    • .NET Core 5 is the modern, componentized framework that ships via NuGet. That means you can ship a private version of the .NET Core Framework with your app. Other apps' versions can't change your app's behavior.
    • We are building a .NET Core CLR for Windows, Mac and Linux and it will be both open source and it will be supported by Microsoft. It'll all happen at https://github.com/dotnet.
    • We are open sourcing the RyuJit and the .NET GC and making them both cross-platform.
  • ASP.NET 5 will work everywhere.
    • ASP.NET 5 will be available for Windows, Mac, and Linux. Mac and Linux support will come soon and it's all going to happen in the open on GitHub at https://github.com/aspnet.
    • ASP.NET 5 will include a web server for Mac and Linux called kestrel built on libuv. It's similar to the one that comes with node, and you could front it with Nginx for production, for example.
  • Developers should have a great experience.
    • There is a new FREE SKU for Visual Studio for open source developers and students called Visual Studio Community. It supports extensions and lots more all in one download. This is not Express. This is basically Pro.
    • Visual Studio 2015 and ASP.NET 5 will support gulp, grunt, bower and npm for front end developers.
    • A community team (including myself and Sayed from the ASP.NET and web tools team) have created the OmniSharp organization along with the Kulture build system as a way to bring real Intellisense to Sublime, Atom, Brackets, Vim, and Emacs on Windows, Linux, and Mac. Check out http://www.omnisharp.net as well as blog posts by team members Jonathan Channon
  • Even more open source.
    • Much of the .NET Core Framework 4.6 and its Reference Source source is going on GitHub. It's being relicensed under the MIT license, so Mono (and you!) can use that source code in their .NET implementations.
    • There's a new hub for Microsoft open source that is hosted GitHub at http://microsoft.github.io.

Open sourcing .NET makes good sense. It makes good business sense, good community sense, and today everyone at Microsoft see this like we do.

Open .NET

Related Links


Sponsor: Big thanks to Aspose for sponsoring the feed this week! Working with Files? Aspose.Total for .NET has all the APIs you need to create, manipulate and convert Microsoft Office documents and many other formats in your applications. Start a free trial today.

About Scott

Scott Hanselman is a former professor, former Chief Architect in finance, now speaker, consultant, father, diabetic, and Microsoft employee. He is a failed stand-up comic, a cornrower, and a book author.

facebook twitter subscribe
About   Newsletter
Sponsored By
Hosting By
Dedicated Windows Server Hosting by ORCS Web

NuGet Package of the Week - Courtesy Flush to flush buffers earlier and optimize time to first byte

November 6, '14 Comments [24] Posted in ASP.NET | ASP.NET MVC | NuGet | NuGetPOW
Sponsored By

Yes, really. It's got to be the best name for an open source library out there. It's a great double entendre and a great name for this useful little library. Perhaps English isn't your first language, so I'll just say that a courtesy flush gives the next person a fresh bowl. ;)

However, in the computer world "flushing a buffer" means forcing a buffer to be moved along, usually to a file or the network. Rather than holding data, you flush it, and move it along.

Nik from Glimpse has a small NuGet package called Courtesy Flush. He's got a good write-up on his blog.

image

It's a library to enable easier flushing of your buffer in ASP.NET MVC. From their site:

Why Flush Early?

Flushing early can provide performance improvements in web applications and has been a recomended best practice in the web performance community since 2007.

To find out more, check out Nik's blog where he covered the benefits of flushing early in two posts:

It builds on top of ASP.NET ActionFilters, which you can apply as attributes to your methods, or call within controllers.

Let's say that you have some server-side work that's very slow. That slow operation could hold up the rendering of your page until it completes. With a pre-flush like this you can get bytes onto the network and into the user's browser faster.

Here we render some information and get it out fast before we do something that's unavoidably slow.

public ActionResult About()
{
ViewBag.Title = DateTime.Now.Second;
this.FlushHead();

Thread.Sleep(2000);
ViewBag.Message = "Your application description page.";

return View();
}

Let's think about really specifically. It's important to know WHY you would want to do this and what exactly happens in the browser.

If you have a long running, but important process (we are pretending that Thread.Sleep(2000) is important) that takes 2 seconds, no HTML is sent to the browser. It's just waiting. The timeline looks like this:

Here we didn't flush

See that blue line? We waited almost 5 seconds for the /about page, and while we were waiting, no Javascript and no CSS were being loaded. Why not? How could the browser know if it isn't seen the <head> of your HTML?

For an optimization, we could FLUSH the buffers that we have up to this point, putting the HTML that we have so far onto the network.

The Layout.cshtml we have a call to @Html.FlushHead() to get the the _Head.cshtml out and into the hands of the browser. It might look like this:

<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0">
@if (ViewBag.Description != null)
{
<meta name="description" content="@ViewBag.Description">
}
<title>@ViewBag.Title - My ASP.NET Application</title>
@Styles.Render("~/Content/css")
@Scripts.Render("~/bundles/modernizr")
</head>

Here's what the SAME page looks like with the <head> flushed out first.

Here we flushed

Look closely at the timeline. Here, I'll do it for you...below shows when we flushed early versus just waiting.

See how when we flushed the head it gave the browser enough information to stat loading some css and javascript we had in the <head?> The whole page took 5 seconds, but when we flush at least we get things going early, while when we don't flush we can't do a thing until the long running task finishes.

See the difference?

See the difference? Now, to be clear, don't blindly go and add optimizations without reading the code and understanding what's going on, but this is a nice tool for our ASP.NET toolbox.

We may see a similar but even more powerful technique in ASP.NET vNext that includes async flushes at multiple points, while updating the model as data is available.


Sponsor: Big thanks to Aspose for sponsoring the feed this week! Working with Files? Aspose.Total for .NET has all the APIs you need to create, manipulate and convert Microsoft Office documents and many other formats in your applications. Start a free trial today.

About Scott

Scott Hanselman is a former professor, former Chief Architect in finance, now speaker, consultant, father, diabetic, and Microsoft employee. He is a failed stand-up comic, a cornrower, and a book author.

facebook twitter subscribe
About   Newsletter
Sponsored By
Hosting By
Dedicated Windows Server Hosting by ORCS Web
Page 1 of 164 in the ASP.NET category Next Page

Disclaimer: The opinions expressed herein are my own personal opinions and do not represent my employer's view in any way.