Back to Basics: Daylight Savings Time bugs strike again with SetLastModified

CC BY-NC 2.0 Creative Commons Clock Photo via Flickr ©Thomas Hawk No matter how well you know a topic, or a codebase, it's never to late (or early) to get nailed by a latest bug a over a half-decade old.

DasBlog, the ASP.NET 2 blog engine that powers this blog, is done. It's not dead, but it's done. It's very stable. We had some commits last year, and I committed a bug fix in February, but it's really well understood and very baked. My blog hasn't been down for traffic spike reasons in literally years as DasBlog scales nicely on a single machine.

It was 10:51pm PDT (that's Pacific Daylight Time) and I was writing a blog post about the clocks in my house, given that PST (that's Pacific Standard Time) was switching over soon. I wrote it up in Windows Live Writer, posted it to my blog, then hit Hanselman.com to check it out.

Bam. 404.

What? 404? Nonsense. Refresh.

404.

*heart in chest* Have I been hacked? What's going on? OK, to the logs!

l2    time    2011-11-06T05:36:31    code    1    message    Error:System.ArgumentOutOfRangeException: Specified argument was out of the range of valid values.
Parameter name: utcDate
at System.Web.HttpCacnhePolicy.UtcSetLastModified(DateTime utcDate)
at System.Web.HttpCachePolicy.SetLastModified(DateTime date)
at newtelligence.DasBlog.Web.Core.SiteUtilities.GetStatusNotModified(DateTime latest) in C:\dev\DasBlog\source\newtelligence.DasBlog.Web.Core\SiteUtilities.cs:line 1253
at newtelligence.DasBlog.Web.Core.SharedBasePage.NotModified(EntryCollection entryCollection) in C:\dev\DasBlog\source\newtelligence.DasBlog.Web.Core\SharedBasePage.cs:line 1182
at newtelligence.DasBlog.Web.Core.SharedBasePage.Page_Load(Object sender, EventArgs e) in C:\dev\DasBlog\source\newtelligence.DasBlog.Web.Core\SharedBasePage.cs:line 1213
at System.EventHandler.Invoke(Object sender, EventArgs e)
at System.Web.UI.Control.OnLoad(EventArgs e)
at System.Web.UI.Control.LoadRecursive()
at System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint) while processing http://www.hanselman.com/blog/default.aspx.

What's going on? Out of range? What's out of range. Ok, my site is down for the first time in years. I must have messed it up with my clock post. I'll delete that. OK, delete. Whew.

Refresh.

404.

WHAT?!?

Logs, same error, now the file is a meg and growing, as this messages is happening of hundreds of times a minute. OK, to the code!

UtcSetLastModified is used for setting cache-specific HTTP headers and for controlling the ASP.NET page output cache. It lets me tell HTTP that something hasn't been modified since a certain time. I've got a utility that figures out which post was the last modified or most recently had comments modified, then I tell the home page, then the browser, so everyone can decide if there is fresh content or not.

public DateTime GetLatestModifedEntryDateTime(IBlogDataService dataService, EntryCollection entries)
{
//figure out if send a 304 Not Modified or not...
return latest //the lastTime anything interesting happened.
}

In the BasePage we ask ourselves, can we avoid work and give a 304?

//Can we get away with an "if-not-modified" header?
if (SiteUtilities.GetStatusNotModified(SiteUtilities.GetLatestModifedEntryDateTime(dataService, entryCollection)))
{
//snip
}

However, note that I'm have to call SetLastModified though. Seems that UtcSetLastModified is private. (Why?) When I call SetLastModified it does this:

public void SetLastModified(DateTime date)
{
DateTime utcDate = DateTimeUtil.ConvertToUniversalTime(date);
this.UtcSetLastModified(utcDate);
}

Um, OK. Lame. So that means I have to work in local time. I retrieve dates and convert them ToLocalTime().

At this point, you might say, Oh, I get it, he's called ToLocalTime() too many times and double converted his times. That's what I thought. However, after .NET 2 that is possible.

The value returned by the conversion is a DateTime whose Kind property always returns Local. Consequently, a valid result is returned even if ToLocalTime is applied repeatedly to the same DateTime.

But. We originally wrote DasBlog in .NET 1.1 first and MOVED it to .NET 2 some years later. I suspect that I'm actually counting on some incorrect behavior deep in own our (Clemens Vasters and mine) TimeZone and Data Access code that worked with that latent incorrect behavior (overconverting DateTimes to local time) and now that's not happening. And hasn't been happening for four years.

Hopefully you can see where this is going.

It seems a comment came in around 5:36am GMT or 10:36pm PDT which is 1:36am EST. That become the new Last Modified Date. At some point we an hour was added in conversion as PDT wasn't PST yet but EDT was EST.

Your brain exploded yet? Hate Daylight Saving Time? Ya, me too.

Anyway, that DateTime became 2:36am EST rather than 1:36am. Problem is, 2:36am EST is/was the future as 6:46 GMT hadn't happened yet.

A sloppy 5 year old bug that has been happening for an hour each year that was likely always there but counted on 10 year old framework code that was fixed 7 years ago. Got Unit Tests for DST? I don't.

My server is in the future, but actually not as far in the future as it usually is. My server in on the East Coast and it was 1:51am. However, the reasons my posts sometimes look like they are from the future, is I store everything in the neutral UTC/GMT zone, so it was 5:51am the next day on my file system.

Moral of the story?

I need to confirm that my server is on GMT time and that none of my storage code is affected my Daylight Saving Time.

Phrased differently, don't use DateTime.Now for ANY date calculations or to store anything. Use DateTime.UTCNow and be aware that some methods will freak out if you send them future dates, as they should. Avoid doing ANYTHING in local time until that last second when you show the DateTime to the user.

In my case, in the nine minutes it took to debug this, it resolved itself. The future became the present and the future last modified DateTime became valid. Is there a bug? There sure it, at least, there is for an hour, once a year. Now the real question is, do I fix it and possibly break something that works the other 8759 hours in a year. Hm, that IS still four 9's of uptime. (Ya, I know I need to fix it.)

"My code has no bugs, it runs exactly as it was written." - Some famous programmer

Until next year. ;)

The Weekly Source Code 59 - An Open Source Treasure: Irony .NET Language Implementation Kit

Select Grammars Dialog in Irony filled with grammarsOne of the best, if not the best way to sharpen the saw and keep your software development skills up to date is by reading code. Sure, write lots of code, but don't forget to explore other people's brains code. There's always fifteen different ways to create a "textboxes over data" application, and while it's interesting to take a look at whatever the newest way to make business software, sometimes it's nice to relax by looking at some implementations of classic software issues like parsers, lexers, and abstract syntax trees. If you didn't go to school or failed to take a compilers class at least knowing that this area of software engineering exists and is accessible to you is very important.

It's so nice to discover open source projects that I didn't know existed. One such project I just stumbled upon while doing research for a customer is "Irony," a .NET language implementation kit. From the CodePlex site:

Irony is a development kit for implementing languages on .NET platform. Unlike most existing yacc/lex-style solutions Irony does not employ any scanner or parser code generation from grammar specifications written in a specialized meta-language. In Irony the target language grammar is coded directly in c# using operator overloading to express grammar constructs. Irony's scanner and parser modules use the grammar encoded as c# class to control the parsing process. See the expression grammar sample for an example of grammar definition in c# class, and using it in a working parser.

Irony includes "simplified grammars for C#, Scheme, SQL, GwBasic, JSON and others" to learn from. There are different kinds of parsers that are grammar generators you might be familiar with. For example, ANTLR is a what's called a LL(*) grammar generator, while Irony is a LALR (Look Ahead Left to Right) grammar generator.

Here's a very basic SQL statement for getting a show from my Podcast database:

SELECT ID, Title FROM Shows WHERE ID = 1

Here's the Irony Parse Tree as viewed in the irony Grammar Explorer:

A complete parse tree of the SQL statement with every node expanded

Typically, in my experience, when creating a parser one will use a DSL (Domain Specific Language) like the GOLD Meta Language building on the BNF (Backus-Naur Form) expression grammar. These domain specific languages are tightly optimized to express exactly how a language is structured and how it should be parsed. You learn a language to create languages.

Remember in the Irony intro text earlier? Let me repeat:

Unlike most existing yacc/lex-style solutions Irony does not employ any scanner or parser code generation from grammar specifications written in a specialized meta-language. In Irony the target language grammar is coded directly in c# using operator overloading to express grammar constructs.

What Roman from Irony has done here is use C# language constructs as if it's a DSL. A fluent parser, as it were. So he's using C# classes and methods to express the language grammar. It's a very interesting and powerful idea if you are interested in creating DSLs but not interested in learning other parsers like GOLD. Plus, it's just fun.

The Irony Grammar Explorer

He has a very rich bass class called Grammar that you derive from, like:

[Language("SQL", "89", "SQL 89 grammar")]
public class SqlGrammar : Grammar {
public SqlGrammar() : base(false) { //SQL is case insensitive
...

But instead of a grammar language like this (simplified by me) to express a SQL SELECT Statement:

! =============================================================================
! Select Statement
! =============================================================================

<Select Stm> ::= SELECT <Columns> <Into Clause> <From Clause> <Where Clause> <Group Clause> <Having Clause> <Order Clause>

<Columns> ::= <Restriction> '*'
| <Restriction> <Column List>

...snip for clarity...

<Restriction> ::= ALL
| DISTINCT
|

<Aggregate> ::= Count '(' '*' ')'
| Count '(' <Expression> ')'
| Avg '(' <Expression> ')'
| Min '(' <Expression> ')'
| Max '(' <Expression> ')'
| StDev '(' <Expression> ')'
| StDevP '(' <Expression> ')'
| Sum '(' <Expression> ')'
| Var '(' <Expression> ')'
| VarP '(' <Expression> ')'

<Into Clause> ::= INTO Id
|

<From Clause> ::= FROM <Id List> <Join Chain>

<Join Chain> ::= <Join> <Join Chain>
|

...snip for clarity...

You'd have something like this instead, again, simplified so this doesn't turn into a giant listing of code rather than a blog post.

//Select stmt
selectStmt.Rule = SELECT + selRestrOpt + selList + intoClauseOpt + fromClauseOpt + whereClauseOpt +
groupClauseOpt + havingClauseOpt + orderClauseOpt;
selRestrOpt.Rule = Empty | "ALL" | "DISTINCT";
selList.Rule = columnItemList | "*";
columnItemList.Rule = MakePlusRule(columnItemList, comma, columnItem);
columnItem.Rule = columnSource + aliasOpt;
aliasOpt.Rule = Empty | asOpt + Id;
asOpt.Rule = Empty | AS;
columnSource.Rule = aggregate | Id;
aggregate.Rule = aggregateName + "(" + aggregateArg + ")";
aggregateArg.Rule = expression | "*";
aggregateName.Rule = COUNT | "Avg" | "Min" | "Max" | "StDev" | "StDevP" | "Sum" | "Var" | "VarP";
intoClauseOpt.Rule = Empty | INTO + Id;
fromClauseOpt.Rule = Empty | FROM + idlist + joinChainOpt;
joinChainOpt.Rule = Empty | joinKindOpt + JOIN + idlist + ON + Id + "=" + Id;
joinKindOpt.Rule = Empty | "INNER" | "LEFT" | "RIGHT";
whereClauseOpt.Rule = Empty | "WHERE" + expression;
groupClauseOpt.Rule = Empty | "GROUP" + BY + idlist;
havingClauseOpt.Rule = Empty | "HAVING" + expression;
orderClauseOpt.Rule = Empty | "ORDER" + BY + orderList;

Here the variables and terms that are being use to build the grammar were defined earlier like this, as an example:

var SELECT = ToTerm("SELECT"); 
var FROM = ToTerm("FROM");
var AS = ToTerm("AS");

You might immediately declare, Dear Reader, that this is blasphemy!  How can C# compete with a specialized DSL like the BNF? This is a C# shaped peg being shoved into a round hold. Well, maybe, but it's interesting to point out that the SQL GOLD Grammar is 259 lines and the C# version of essentially the same thing is 247 lines. Now, I'm not pointing out line numbers to imply that this is a better way or that this is even a valid 1:1 comparison. But, it's interesting that the C# class is even close. You might have assumed it would be much much larger. I think it's close because Roman, the Irony developer, has a very well factored and specialized base class for the derived class to "lean on." Each of his sample grammars are surprisingly tight. For example:

  • "Mini" Python - ~140 lines
  • Java - ~130 lines
  • Scheme - ~200 lines
  • JSON - 39 lines

To conclude, here's the JSON grammar generator. 

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using Irony.Parsing;

namespace Irony.Samples.Json {
[Language("JSON", "1.0", "JSON data format")]
public class JsonGrammar : Grammar {
public JsonGrammar() {
//Terminals
var jstring = new StringLiteral("string", "\"");
var jnumber = new NumberLiteral("number");
var comma = ToTerm(",");

//Nonterminals
var jobject = new NonTerminal("Object");
var jobjectBr = new NonTerminal("ObjectBr");
var jarray = new NonTerminal("Array");
var jarrayBr = new NonTerminal("ArrayBr");
var jvalue = new NonTerminal("Value");
var jprop = new NonTerminal("Property");

//Rules
jvalue.Rule = jstring | jnumber | jobjectBr | jarrayBr | "true" | "false" | "null";
jobjectBr.Rule = "{" + jobject + "}";
jobject.Rule = MakeStarRule(jobject, comma, jprop);
jprop.Rule = jstring + ":" + jvalue;
jarrayBr.Rule = "[" + jarray + "]";
jarray.Rule = MakeStarRule(jarray, comma, jvalue);

//Set grammar root
this.Root = jvalue;
MarkPunctuation("{", "}", "[", "]", ":", ",");
this.MarkTransient(jvalue, jarrayBr, jobjectBr);

}//constructor
}//class
}//namespace

Pretty clever stuff, and a well put together project and solution that is well structured. I could myself using this in a C# or Compiler class to teach some of these concepts.

It's also a great little tool for creating small languages of your own. Perhaps you have a Wiki-dialect that's specific to your company and you want to get rid of all that nasty manual parsing? Or many you have an old custom workflow engine or custom expression system embedded in your application and never got around to changing all your parsing to a proper grammar? Maybe now is the time to get that little language you've been thinking about off the ground!

I encourage you, Dear Reader, to support open source projects like this. Why not go leave a comment today on your favorite open source project's site and just let them know you appreciate what they're doing?

Related Links

Back to Basics: Big O notation issues with older .NET code and improving for loops with LINQ deferred execution

Earlier today Brad Wilson and I were discussing a G+ post by Jostein Kjønigsen where he says, "see if you can spot the O(n^2) bug in this code."

public IEnumerable<Process> GetProcessesForSession(string processName, int sessionId)
{
var processes = Process.GetProcessByName(processName);
var filtered = from p in processes
where p.SessionId == sessionId
select p;
return filtered;
}

This is a pretty straightforward method that calls a .NET BCL (Base Class Library) method and filters the result with LINQ. Of course, when any function calls another one that you can't see inside (which is basically always) you've lost control. We have no idea what's going on in GetProcessesByName.

Let's look at the source to the .NET Framework method in Reflector. Our method calls Process.GetProcessesByName(string).

public static Process[] GetProcessesByName(string processName)
{
return GetProcessesByName(processName, ".");
}

Looks like this one is an overload that passes "." into the next method Process.GetProcessesByName(string, string) where the second parameter is the machineName.

This next one gets all the processes for a machine (in our case, the local machine) then spins through them doing a compare on each one in order to build a result array to return up the chain.

public static Process[] GetProcessesByName(string processName, string machineName)
{
if (processName == null)
{
processName = string.Empty;
}
Process[] processes = GetProcesses(machineName);
ArrayList list = new ArrayList();
for (int i = 0; i < processes.Length; i++)
{
if (string.Equals(processName, processes[i].ProcessName, StringComparison.OrdinalIgnoreCase))
{
list.Add(processes[i]);
}
}
Process[] array = new Process[list.Count];
list.CopyTo(array, 0);
return array;
}

if we look inside GetProcesses(string), it's another loop. This is getting close to where .NET calls Win32 and as these classes are internal there's not much I can do to fix this function other than totally rewrite the internal implementation. However, I think I've illustrated that we've got at least two loops here, and more likely three or four.

public static Process[] GetProcesses(string machineName)
{
bool isRemoteMachine = ProcessManager.IsRemoteMachine(machineName);
ProcessInfo[] processInfos = ProcessManager.GetProcessInfos(machineName);
Process[] processArray = new Process[processInfos.Length];
for (int i = 0; i < processInfos.Length; i++)
{
ProcessInfo processInfo = processInfos[i];
processArray[i] = new Process(machineName, isRemoteMachine, processInfo.processId, processInfo);
}
return processArray;
}

This code is really typical of .NET circa 2002-2003 (not to mention Java, C++ and Pascal). Functions return arrays of stuff and other functions higher up filter and sort.

When using this .NET API and for looping over the results several times, I'm going for(), for(), for() in a chain, like O(4n) here.

Note: To be clear, it can be argued that O(4n) is just O(n), cause it is. Adding a number like I am isn't part of the O notation. I'm just saying we want to avoid O(cn) situations where c is a large enough number to affect perf.

image

Sometimes you'll see nested for()s like this, so O(n^3) here where things get messy fast.

Squares inside squares inside squares representing nested fors

LINQ is more significant than people really realize, I think. When it first came out some folks said "is that all?" I think that's unfortunate. LINQ and the concept of "deferred execution" is just so powerful but I think a number of .NET programmers just haven't taken the time to get their heads around the concept.

Here's a simple example juxtaposing spinning through a list vs. using yield. The array version is doing all the work up front, while the yield version can calculate. Imagine a GetFibonacci() method. A yield version could calculate values "just in time" and yield them, while an array version would have to pre-calculate and pre-allocate.

public void Consumer()
{
foreach (int i in IntegersList()) {
Console.WriteLine(i.ToString());
}

foreach (int i in IntegersYield()) {
Console.WriteLine(i.ToString());
}
}

public IEnumerable<int> IntegersYield()
{
yield return 1;
yield return 2;
yield return 4;
yield return 8;
yield return 16;
yield return 16777216;
}

public IEnumerable<int> IntegersList()
{
return new int[] { 1, 2, 4, 8, 16, 16777216 };
}

Back to our GetProcess example. There's two issues at play here.

First, the underlying implementation where GetProcessesInfos eventually gets called is a bummer but it's that way because of how P/Invoke works and how the underlying Win32 API returns the data we need. It would certainly be nice if the underlying API was more granular. But that's less interesting to me than the larger meta-issue of a having (or in this case, not having) a LINQ-friendly API.

The second and more interesting issue (in my option) is the idea that the 2002-era .NET Base Class Library isn't really setup for LINQ-friendliness. None of the APIs return LINQ-friendly stuff or IEnumerable<anything> so that when you change together filters and filters of filters of arrays you end up with O(cn) issues as opposed to nice deferred LINQ chains.

When you find yourself returning arrays of arrays of arrays of other stuff while looping and filtering and sorting, you'll want to be aware of what's going on and consider that you might be looping inefficiently and it might be time for LINQ and deferred execution.

image

Here's a simple conversion attempt to change the first implementation from this classic "Array/List" style:

ArrayList list = new ArrayList();
for (int i = 0; i < processes.Length; i++)
{
if (string.Equals(processName, processes[i].ProcessName, StringComparison.OrdinalIgnoreCase))
{
list.Add(processes[i]);
}
}
Process[] array = new Process[list.Count];
list.CopyTo(array, 0);
return array;

To this more LINQy way. Note that returning from a LINQ query defers execution as LINQ is chainable. We want to assemble a chain of sorting and filtering operations and execute them ONCE rather than for()ing over many lists many times.

if (processName == null) { processName = string.Empty; }

Process[] processes = Process.GetProcesses(machineName); //stop here...can't go farther?

return from p in processes
where String.Equals(p.ProcessName, processName, StringComparison.OrdinalIgnoreCase)
select p; //the value of the LINQ expression being returned is an IEnumerable<Process> object that uses "yield return" under the hood
Here's the whole thing in a sample program.
static void Main(string[] args)
{
var myList = GetProcessesForSession("chrome.exe", 1);
}

public static IEnumerable<Process> GetProcessesForSession(string processName, int sessionID)
{
//var processes = Process.GetProcessesByName(processName);
var processes = HanselGetProcessesByName(processName); //my LINQy implementation
var filtered = from p in processes
where p.SessionId == sessionID
select p;
return filtered;
}

private static IEnumerable<Process> HanselGetProcessesByName(string processName)
{
return HanselGetProcessesByName(processName, ".");
}

private static IEnumerable<Process> HanselGetProcessesByName(string processName, string machineName)
{
if (processName == null)
{
processName = string.Empty;
}
Process[] processes = Process.GetProcesses(machineName); //can't refactor farther because of internals.

//"the value of the LINQ expression being returned is an IEnumerable<Process> object that uses "yield return" under the hood" (thanks Mehrdad!)

return from p in processes where String.Equals(p.ProcessName == processName, StringComparison.OrdinalIgnoreCase) select p;

/* the stuff above replaces the stuff below */
//ArrayList list = new ArrayList();
//for (int i = 0; i < processes.Length; i++)
//{
// if (string.Equals(processName, processes[i].ProcessName, StringComparison.OrdinalIgnoreCase))
// {
// list.Add(processes[i]);
// }
//}
//Process[] array = new Process[list.Count];
//list.CopyTo(array, 0);
//return array;
}

This is a really interesting topic to me and I'm interested in your opinion as well, Dear Reader. As parts of the .NET Framework are being extended to include support for asynchronous operations, I'm wondering if there are other places in the BCL that should be updated to be more LINQ friendly. Or, perhaps it's not an issue at all.

Your thoughts?

Using Code Signing Certificates to sign downloaded MSIs and build reputation with IE9 SmartScreen

First, let me start that if you want a lot of people to download something, make sure that the words "HTML5," "Support" and "Update" appear in the title. I'm sure if the folks that are making Diablo 3 called it "Diablo 3 HTML5 Support Update" that a metric buttload more people would download it.

That said, a bunch of folks in the Web Platform and Tools team created the Web Standards Update package with HTML5 Support for the Visual Studio 2010 Editor.

This Web Standards Update is something that anyone in the community could have released, just extending Visual Studio in a standard way. Like many other (most) extensions in Visual Studio Extension Gallery, it was not "signed." It was not a formal project done by Microsoft. Ratherthis was something that a bunch of us did for the community in our after work hours.  The only reason why this got in spotlight was because press caught the wind of it having HTML5 and CSS3 support. 

Certainly a lot of people wanted it because in 4 days it's now the #1 most popular thing in the Visual Studio Gallery. Take that NuGet! ;)

Here's where the trouble starts. Then, it was written about in the press as if it were a "gaffe." I admit that we (mostly I) did a lousy mediocre job of making it clear that this update was a "community update from the inside," as it were. It's not official, but we're hoping support like this will make its way into the next version of Visual Studio.

When you downloaded the MSI installer with IE9, as with all MSIs that aren't signed, you get a message like this:

Do you want to Run or Save this MSI?

And that's normal and quite lovely. Then we see this scary red bar (this is a shot from another gallery item):

SmartScreen Red (BAD) Bar

This is the IE9 SmartScreen system warning us, rightfully so, that this is not something downloaded all the time. In fact, this is a really useful feature of IE9 and is fairly unique amongst the browsers so far. It's using some special sauce (some hash, some math, some metrics) to make a non-biased judgment about this download. Even though it's coming from a Microsoft.com website it doesn't matter. SmartScreen is unbiased. It's never seen this before, and it's not trusted.

UPDATE: Looks like as of my test just now that SmartScreen now recognizes our download as safe!

At this point, if I click Actions, I see this. (Yes, I realize these screenshots aren't all up to snuff).

  (38)

In fact, for most people, they can't even click "Run Anyway" yet. They'll have to click More Options to see the Run Anyway button. (If I am a developer-type and click More Options all the time, presumably I either know what I'm doing, or I like to live dangerously and the More Options choice will stick open after several downloads. It'll save me a click, but all the other warnings remain.)

As the publisher, we have a few choices. We could sign the binary file (the MSI) with the Microsoft code certificate. However, that requires a big manager to sign off and says explicitly that Microsoft is releasing this code officially. It's a big deal. This wasn’t an official release and as such, we can't sign it as Microsoft. A code signing certificate guarantees that a file hasn't been tampered with and that a known and verified organization or individual stands behind it.

Eventually SmartScreen would figure out that our MSI was OK, but we have no way of telling how long that would take. Could be weeks, months, it all depends. Regardless, the right thing to do is to sign your code, even if you are an individual or small company. For example, if I download Eric Lawrence's Fidder or Rick Brewster's Paint.NET, they are both signed and I can see their names in the User Account Control (UAC) dialog. I can click and view their certificates and know I'm downloading a file that has someone vouching for it.

Be sure to check out Eric Lawrence's excellent post on Authenticode Code Signing. It's extremely detailed and worth your time.

Getting a Code Signing Certificate

I got a Code Signing Certificate from InstantSSL.com. There's many options, they are one. It's spendy, $180 a year, or $166 a year if you got for 3 years, but I can use it for other stuff.

There's a few gotchas in the process, no matter who you pick.

  • Use the same computer, same OS, and same browser (preferably IE, for this, no joke) when you sign up for the certificate. That's because half the certificate (a cert request cert) comes down when you request a certificate and they match them up when you actually get the certificate.
  • Have P.O. Box, corporate address or ask them via tech support to remove your address. Otherwise your full details may get embedded in the cert.
  • You'll need to prove who you are. More on that now.

You'll need to prove you are really you. I needed to give their verification people a copy of the first page of my passport, driver's license, two utility bills, including phone whose address matched my credit card's address, AND they called the phone number on my utility bill to confirm it was really me. It's non-trivial, it takes a while, and they aren't screwing around. Good for you, the consumer, hassle for me, the producer. Still, good stuff.

Certificate Manager with my new Cert

When my cert shows up, I need to Export it and save it in a safe place with all its details and a strong password. It's unique and should be protected.

Signing Code

The actual signing, once the cert shows up is not too hard. Here's a command line used with the signtool.exe that came with Visual Studio. You can also download it separately.

C:\DEV> signtool sign /t http://timestamp.comodoca.com/authenticode /f "C:\DEV\HanselmanCODESIGNINGCERT.pfx" /p SecretPassword '.\MySpecial.msi'
Done Adding Additional Store
Successfully signed and timestamped: .\MySpecial.msi

When someone tries to download the new signed MSI, they see this slightly less scary yellow bar. What? I don't get a free pass for signing my code?

SmartScreen Yellow Bar

Well, just like getting an SSL certificate doesn't make me a bank, getting a Code Signing Certificate doesn't make me more trustworthy.

  • SSL Certificates for HTTPS guarantee privacy, not trust.
  • Code Signing Certificates guarantee identity, not trust.
    • It guarantees it's me, but you have to decide if you trust me.

If you click Actions now, you'll see my name as the Publisher, and you can validate the certificate and decide if you trust me. But SmartScreen doesn't trust me yet. Why?

 My code signing certificate in the Run Dialog

That's because my Certificate, unlike the Microsoft one, hasn't built up a reputation*. The "Scott Hanselman" code signing cert will have to earn trust, just as Rick Brewster and Eric Lawrence and every other signed shareware or freeware author has built trust. But, having this MSI signed means you now that I (and Mads, and Vishal, and the folks working on this MSI) stand behind it. Hopefully soon (some # days or weeks vs. downloads?) SmartScreen will trust us also, and this will make future projects I sign be trusted faster. At that point, my signed code will be trusted and SmartScreen won't frighten you with this download.

Remember also that code signing certificates and the Windows experience and UI for running signed MSI and EXEs is a separate from SmartScreen. They work together and compliment each other though. Learn more about SmartScreen on their team blog or their FAQ.

Hope this helps! Surf smart, and think about what you download and who you trust.

* Now it appears that SmartScreen trusts me!

NuGet Package of the Week #6 - Dynamic, Malleable, Enjoyable Expando Objects with Clay

Hey, have you implemented the NuGet Action Plan? Get on it, it'll take only 5 minutes: NuGet Action Plan - Upgrade to 1.2, Setup Automatic Updates, Get NuGet Package Explorer. NuGet 1.3 is out, so make sure you're set to automatically update!

The Backstory: I was thinking since the NuGet .NET package management site is starting to fill up that I should start looking for gems (no pun intended) in there. You know, really useful stuff that folks might otherwise not find. I'll look for mostly open source projects, ones I think are really useful. I'll look at how they built their NuGet packages, if there's anything interesting about the way the designed the out of the box experience (and anything they could do to make it better) as well as what the package itself does.

This weeks Package of the Week is "Clay." It makes working with dynamic objects even more fun. It was written for the open source Orchard Project by Louis DeJardin with an assist from Bertrand LeRoy.

image

Enjoyable Dynamics in a Static Language

Here's a little copy/paste from a post two years ago I did on the dynamic keyword in C#. I thought it was good, so I'll include it again here.

So I asked this guy, what's up with the dynamic keyword, and what type was it exactly? I mean, C# isn't dynamic, right? He says:

"Oh, well it's statically-typed as a dynamic type."

Then my brain exploded and began to leak out my ears. Honestly, though, it took a second. Here's a good example from some of Anders' slides:

Calculator calc = GetCalculator();
int sum = calc.Add(10, 20);

That's the creation of an object, invokation of a method, and the collection of a return value. This is the exact same code, as the "var" type is figured out at compile time.

var calc = GetCalculator();
int sum = calc.Add(10, 20);

If you wanted to do the exact same thing, except with Reflection (like if it were some other class, maybe old-COM interop, or something where the compiler didn't know a priori that Add() was available, etc) you'd do this:

object calc = GetCalculator();
Type calcType = calc.GetType();
object res = calcType.InvokeMember("Add",
BindingFlags.InvokeMethod, null,
new object[] { 10, 20 });
int sum = Convert.ToInt32(res);

It's pretty horrible to look at, of course. If the object is some dynamic thing (from any number of sources), we can do this:

dynamic calc = GetCalculator();
int sum = calc.Add(10, 20);

And get the dynamic method invocation and conversion of the return type. Basically it looks just like we're calling any other object.

My buddy Rob Conery and I love dynamic languages, but we also love the .NET CLR. If we had our way, there'd be a lot more support for the Dynamic Language Runtime and the Iron.NET languages. We wrote the http://thisdeveloperslife.com website using ASP.NET Web Pages largely because it uses the Razor template engine - which feels very dynamic - and we used dynamics throughout the code.

Some folks think that static languages have no business dipping their toes into the dynamic pool, but I disagree. Successful compilation is just the first unit test, as they say, and I like the ability to pick and choose between static and dyanamic.

Expandos and Dynamic

In .NET, the Expando object is a dynamic type that lets you add and remove members to it, ahem, dynamically. It's great for dealing with dynamic data. You might do this:

dynamic myObject = new ExpandoObject();
myObject.WhateverMakesMeHappy = "Scott";

And boom, I've got a new property. You can even "cast" Expandos as other types and start using them like that type. It's crazy. Play with it.

Anonymous objects via object initalizers are nice, but once you've made one, it's stuck that way. For example, from Bertrand Le Roy's blog

Html.TextBoxFor(m => m.CurrentUser, new {
title = "User Name",
style = "float:left;"
})

See the object intializer? It makes an anonymous object, but it'll have that shape with title and style, forever.

Why is Clay needed?

In Bertrand's words:

In terms of API usability [ExpandoObject is] not very daring and in particular it does not do much to help you build deep dynamic object graphs. Its behavior is also fixed and can’t be extended.

Clay on the other hand is highly extensible and focuses on creation and consumption of deep graphs.

Clay has a clever naming convention (although you may hate it. Relax, it's a convention) where you name the ClayFactory instance "New." Yes, capital-N "New." *brain explodes again*

You can do the usual stuff with Clay that you can also do with Expando, of course. But, you can use several different techniques depending on the situation you're in, and that's where it gets interesting. Here's some examples from Bertrand and Lou, starting with the ClayFactory creation:

dynamic New = new ClayFactory();

Now this “New” object will help us create new Clay objects, as the name implies (although this name is just a convention). Then:

var person = New.Person();
person.FirstName = "Louis";
person.LastName = "Dejardin";

For instance in Clay, indexer syntax and property accessors are equivalent, just as they are in JavaScript. This is very useful when you are writing code that accesses a property by name without knowing that name at compile-time:

var person = New.Person();
person["FirstName"] = "Louis";
person["LastName"] = "Dejardin";

You can also use properties as chainable setters, jQuery-style:

var person = New.Person()
    .FirstName("Louis")
    .LastName("Dejardin");

Or you can pass an anonymous object and it will become a Clay object:

var person = New.Person(new {
    FirstName = "Louis",
    LastName = "Dejardin"
});

Even better, Clay also understands named arguments, which enables us to write this:

var person = New.Person(
    FirstName: "Louis",
    LastName: "Dejardin"
);

Or even this as an array:

var people = New.Array(
New.Person().FirstName("Louis").LastName("Dejardin"),
New.Person().FirstName("Bertrand").LastName("Le Roy")
);

All of this also means that these are all equivalent:

person.FirstName
person["FirstName"]
person.FirstName()

To get started, rather than using NuGet to "install-package Clay," I'd recommend you install Clay.Sample. This is a common convention for open source projects to include sample packages that have a dependency on the project itself. Install the sample and you'll get both packages.

Here's some other cool samples that really give you an idea of how you can move like clay between the dynamic and static worlds:

public interface IPerson {
string FirstName { get; set; }
string LastName { get; set; }
}

public static void CastToCLRInterface() {
dynamic New = new ClayFactory();

var person = New.Person();
person.FirstName = "Louis";
person.LastName = "Dejardin";

// Concrete interface implementation gets magically created!
IPerson lou = person;

// You get intellisense and compile time check here
Console.WriteLine("{0} {1}", lou.FirstName, lou.LastName);
}

I'd like the see folks in power *cough* Anders *cough* check out things like Clay and make them built in. Yum.

Related Links

Page 1 of 28 in the Learning .NET category Next Page
Disclaimer: The opinions expressed herein are my own personal opinions and do not represent my employer’s view in any way.