Scott Hanselman

Back to (Parallel) Basics: Don't Block Your Threads, Make Async I/O Work For You

November 15, '10 Comments [11] Posted in Back to Basics | Learning .NET | Programming | Source Code
Sponsored By

Stephen Toub is one of my favorite folks at Microsoft. I've asked him questions before, sometimes for myself, sometimes on your behalf, Dear Reader, and I've always received thoughtful and well reasoned answered. Because I believe strongly in Jon Udell's "Conserve Your Keystrokes" philosophy, I always try to get great information out to the community, especially when it's been emailed. Remember, when you slap the keyboard and write an epic email to just a few people, there's millions of people out there that miss out. Email less, blog more. More on this in a moment.

TIP: If you're interested in Patterns for Parallel Programming, run, don't walk, and download the FREE and extensive eBook called, yes, you guessed it, Patterns for Parallel Programming: Understanding and Applying Parallel Patterns with the .NET Framework 4. Yes, that title is long, but it feels shorter if you process it in parallel. Seriously, it's free and there's a C# and Visual Basic version. It's brilliant.
Now, if you're REALLY interested in the topic, go get the book Parallel Programming with Microsoft .NET by Stephen Toub, Ade Miller, Colin Campbell, and Ralph Johnson. The complete book as HTML is also hosted here.

I recently noticed a blog post from my friend Steve Smith where he shares some quick sample code to "Verify a List of URLs in C# Asynchronously." As I know Steve wouldn't mind me digging into this, I did. I started by asking Stephen Toub in the Parallel Computing group at Microsoft.

Steve Smith wanted to verify a list of URLs for existence. This is the basic synchronous code:

private static bool RemoteFileExists(string url)
{
try
{
var request = WebRequest.Create(url) as HttpWebRequest;
request.Method = "HEAD";
var response = request.GetResponse() as HttpWebResponse;
return (response.StatusCode == HttpStatusCode.OK);
}
catch
{
return false;
}
}

Then Steve changed the code to be Parallel using the new Parallel features of .NET 4 as Stephen Toub helped me explain in "Back to Parallel Basics" in April.

var linkList = GetLinks();

Action<int> updateLink = i =>
{
UpdateLinkStatus(linkList[i]); //fetch URL and update its status in a shared list
};
Parallel.For(0, linkList.Count, updateLink);

Using Parallel.For is a really great way to introduce some basic naive parallelism into your applications.

I'm no expert in parallelism (I've read a great whitepaper...) but I asked Stephen Toub if this was the best and recommended way to solve this problem. Stephen responded from a plane using (his words) "email compiled and tested" examples. With his permission, I've included a derivation of his response here in this blog post for my own, and possibly your, edification.

From Stephen:

First, it looked like the author was proposing using a parallel loop to handle this.  That's ok, and certainly easy, but that’s the kind of thing you’d only really want to do in a client application and not a server application.  The issue here is that, while easy, it blocks threads; for a client application, having a few more threads that are blocked typically isn’t a big deal; for a server app, though, if for example you were doing this in response to an incoming ASP.NET or WCF service request, you'd be blocking several threads per request, which will greatly hinder scalability.  Still, to get up and running quickly, and if the extra few threads isn’t problematic, this is a fine way to go. 

Assuming you want you "fan out" quickly and easily and it's OK to block a few threads, you can either use a parallel loop, tasks directly, or Stephen's personal favorite, a PLINQ query, e.g. if I have a function "bool ValidateUrl(string url);", I can use PLINQ to process up to N at a time:

bool [] results = (from url in urls.AsParallel() select ValidateUrl(url)).ToArray();

In this example, PLINQ will use up to N threads from the ThreadPool, where N defaults to Environment.ProcessorCount, but you can tack on .WithDegreeOfParallelism(N) after the AsParallel() and provide your own N value.

If Steve was doing this in a console app, which is likely, then as Stephen points out, that's no big deal. You've usually got threads to spare on the client. On the server side, however, you want to avoid blocking threads as much as you can.

A better solution from a scalability perspective, says Stephen, is to take advantage of asynchronous I/O.  When you're calling out across the network, there's no reason (other than convenience) to blocks threads while waiting for the response to come back. Unfortunately, in the past it's been difficult to do this kind of aggregation of async operations.  We' need to rewrite our ValidateUrl method to be something more like:

public void ValidateUrlAsync(string url, Action<string,bool> callback);

where the method returns immediately and later calls back through the provided callback to alert whether a given URL is valid or not.  We'd then change our usage of this to be more like this. Notice the use of using System.Collections.Concurrent.ConcurrentQueue representing a thread-safe first in-first out (FIFO) collection, and CountdownEvent, that represents a synchronization primitive that is signaled when its count reaches zero.

using(var ce = new CountdownEvent(urls.Length))

{
var results = new ConcurrentQueue<Tuple<string,bool>>();

Action callback = (url,valid) =>
{
results.Enqueue(Tuple.Create(url,valid));
ce.Signal();
};

foreach(var url in urls) ValidateUrlAsync(url, callback);

ce.Wait();
}

Assuming ValidateUrlAsync is written to use async, e.g. (you'd really want the following to do better error-handling, but again, this is email-compiled):

public void ValidateUrlAsync(string url, Action<string,bool> callback)
{
var request = (HttpWebRequest)WebRequest.Create(url);
try
{
request.BeginGetResponse(iar =>
{
HttpWebResponse response = null;
try
{
response = (HttpWebResponse)request.EndGetResponse(iar);
callback(url, response.StatusCode == HttpStatusCode.OK);
}
catch { callback(url, false); }
finally { if (response != null) response.Close(); }
}, null);
}
catch { callback(url, false); }
}

This example would then this would end up only blocking the main thread launching all of the requests and then blocking waiting for all of the responses, rather than blocking one thread per request.  With a slight change, we could also make the launcher async, for example:

public static void ValidateUrlsAsync(string [] urls, Action<IEnumerable<Tuple<string,bool>> callback)
{
var ce = new CountdownEvent(urls.Length);
var results = new ConcurrentQueue<Tuple<string,bool>>();
Action callback = (url,valid) =>
{
results.Enqueue(Tuple.Create(url,valid));
if (ce.Signal()) callback(results);
};
foreach(var url in urls) ValidateUrlAsync(url, callback);
}

Still, this is all really complicated, and much more difficult than the original one-liner using PLINQ. 

This is where Tasks and the new Async CTP come in really handy. Imagine that instead of

void ValidateUrlAsync(string url, Action<bool> callback);

we instead had

Task<bool> ValidateUrlAsync(string url);

The Task<bool> being returned is much more composable, and represents the result (both the successful completion case and the exceptional case) of the async operation. 

BETA NOTE: It's not possible to have both ASP.NET MVC 3 and the Async CTP installed at the same time. This is a beta conflict thing, it'll be fixed, I'm sure.

If we had such an operation, and if we had a Task.WhenAll method that took any number of tasks and returned a task to represent them all, then we can easily await all of the results, e.g.

bool [] results = await Task.WhenAll(from url in urls select ValidateUrlAsync(url));

Nice and simple, entirely asynchronous, no blocked threads, etc. 

(Note that in the Async CTP, Task.WhenAll is currently TaskEx.WhenAll, because since it was an out-of-band CTP we couldn't add the static WhenAll method onto Task like we wanted to.)

With the Async CTP and the await keyword, it's also much easier to implement the ValidateUrlAsync method, and to do so with complete support for exception handling (which I didn't do in my previous example, i.e. if something fails, it doesn't communicate why).

public async Task<bool> ValidateUrlAsync(string url)
{
using(var response = (HttpWebResponse)await WebRequest.Create(url).GetResponseAsync())
return response.StatusCode == HttpStatusCode.Ok;
}

Even without the Async CTP, though, it's still possible to implement ValidateUrlAsync with this signature.

Notice the use of System.Threading.Tasks.TaskCompletionSource. From MSDN:

In many scenarios, it is useful to enable a Task (Of(TResult)) to represent an external asynchronous operation. TaskCompletionSource (Of( TResult)) is provided for this purpose. It enables the creation of a task that can be handed out to consumers, and those consumers can use the members of the task as they would any other.

public Task<bool> ValidateUrlAsync(string url)
{
var tcs = new TaskCompletionSource<bool>();
var request = (HttpWebRequest)WebRequest.Create(url);
try
{
request.BeginGetResponse(iar =>
{
HttpWebResponse response = null;
try
{
response = (HttpWebResponse)request.EndGetResponse(iar);
tcs.SetResult(response.StatusCode == HttpStatusCode.OK);
}
catch(Exception exc) { tcs.SetException(exc); }
finally { if (response != null) response.Close(); }
}, null);
}
catch(Exception exc) { tcs.SetException(exc); }
return tsc.Task;

}

So, with this method, even without the Async CTP, we can use existing .NET 4 support to handle this relatively easily:

Task.Factory.ContinueWhenAll(
(from url in urls select ValidateUrlAsync(url)).ToArray(),
completedTasks => { /* do some end task */ });

Now, using just what comes with .NET 4 proper I get the best of all worlds.

Big thanks to Stephen Toub. There's lots of new high- and low-level constructs for Task creation, Threading, and Parallelism in .NET 4. While the naive solution is often the right one, the components we have to work with in .NET 4 (and the even newer ones in the Visual Studio 2010 Async CTP adding the 'await' and 'async' keywords) will give you surprisingly fine-grained control over your multi-threaded parallel systems without a whole lot of code.

Related Links:

About Scott

Scott Hanselman is a former professor, former Chief Architect in finance, now speaker, consultant, father, diabetic, and Microsoft employee. He is a failed stand-up comic, a cornrower, and a book author.

facebook twitter subscribe
About   Newsletter
Sponsored By
Hosting By
Dedicated Windows Server Hosting by SherWeb

Streaming Live or On-Demand Video from IIS7 to iOS Devices (iPhone/iPad) and Silverlight Clients

November 9, '10 Comments [20] Posted in IIS | Silverlight
Sponsored By

DSC_0001 Earlier this year I posted on Installing and Setting Up and Encoding for IIS 7 Smooth Streaming and Silverlight. The general idea was to show folks that it's actually pretty easy to setup Smooth Streaming for Silverlight. Of course, not everyone uses Silverlight, and lots of folks (myself included) have iPhones and iPads.

I've long been singing the praises of IIS7. If you're on IIS6, get off it. IIS7 is the #1 reason to get on Windows Server 2008R2, IMHO. Mostly because of the crapload of awesome (CoA) that keeps coming out of the IIS team. Many of these modules aren't on the radar of the average developer. I think that Web Platform Installer helps some, but still, I'd like to get the word about some of this free stuff like the Web Farm Framework, the all new FTP 7.5, Bit Rate Throttling, and my favorite, Application Request Routing (which I use in my robot).

Anyway, last week IIS released IIS Media Services 4.0 that you can download now. It's free if you have IIS on your machine. They also released a number of things this summer like the Transform Manager 1.0 Alpha.

Getting existing Video from IIS7 to iOS devices elegantly

Why do I care? Well, I'm sitting here looking at an IPhone 3G, an iPhone 4 and an iPad on my desk. Many of your customers have these devices. How do you get video to them, cleanly, quickly, easily and from a single source file?

Turns out there's features in IIS7 and Media Services that target iOS directly. Yes, it's true. Cats and dogs, living together, mass hysteria. Get your snowballs out, hell has frozen over and the slopes look sweet.

On-Demand Encoded Video for iOS and Silverlight

First, I've got this file on my hard drive called ScottGuHDTest1. It's a picture in picture (PIP) test that ScottGu and I did a while ago while looking into new ways to deliver educational video content. Consider this my stock video for this example.

Windows Media Player - ScottGuTest Video Frame

If I want to deliver this content to the web over Silverlight Adaptive Smooth Streaming PLUS I want it to look great on an iPhone or iPad regardless of bandwidth, and I want it all done automatically, I'll need these things.

imageFor existing videos or videos you're making on a regular basis, the idea is that you'll just drop them in a folder and a task will automatically kick off. You have total control over everything, to be clear, but for the out of the box example, I'm going to drop my ScottGu WMV into a folder and two things will pop out:

  • A folder with HTML page with a Silverlight media player that supports Smooth Streaming. The ScottGu video will be encoded in various bit rates and be all set for adaptive Smooth Streaming
  • A folder with MPEG-2 TS format files with .m3u8 manifests that conform to the Apple Live Streaming Protocol.

Here's what you do. Once you've installed the Transform Manager, it'll show up in your IIS Manager as a node. You'll setup account credentials (an account with some power, as it'll need to run stuff and copy files into IIS folders. I used my own. Naughty.)

Now, go to Job Templates and notice there are a bunch of built-in templates. You can create whatever you want to encode video in whatever way makes you happy.

I double-clicked on them and checked all the paths and stuff. You'll need to change the templatePath property to decide on your input and output directories.

 Edit Job Template

I have my input folder here, in c:\inetpub\media\Transform Manager, and there's sub-folders beneath that as seen in this screenshot. Notice that tasks can do multiple things, so mine takes an input file and "fans out" into multiple tasks.

 Transform Manager

If I drop my ScottGu file in, a task immediately starts up (it's watching the folder) and I can see the progress from the IIS manager. Here it is in the middle of encoding.

Internet Information Services (IIS) Manager

The Transforms Manager has lots of interesting parameters that you can set, basically using simple template expansion. It'll make a folder with the Date and Time and a Unique Number, but again, you control all this.

Now I have two folders, one for the Silverlight format...

20101108231223_22BCFBBAA9864545A8864DEDC41E6A87

...and one for the Apple format:

 TS

I'll add a HTML page for an iOS device to hit. NOTE: You'll be pointing the iPhone/iPad to this m3u8.ism file and it'll decide what streams to watch:

<html>
<head>
<title>iPhone page</title>
</head>
<body>
<h1>Encoded stream</h1>
<video width="640"
height="480"
src="ScottGu720pTest1-m3u8-aapl.ism/manifest(format=m3u8-aapl).m3u8"
poster="ScottGuSplashPage.png"
autoplay="true"
controls="true" >Live</video>
</body>
</html>

I've also added splashscreen "poster" that's just a PNG frame from the video. Here's a photograph of my iPhone 3G, iPad and iPhone 4 with the video...

DSC_0001

...and here it is in Silverlight. Note the chart showing the smooth streaming bandwidth adjusting.

ScottGu720pTest1 - Windows Internet Explorer

Cool, that was easy. How about LIVE video streaming to both iPhone/iPad clients and Web Browsers?

Live Streaming Video from IIS7 to iOS devices and Silverlight

Doing LIVE Streaming video is pretty easy also. You start with IIS Media Services 4 and Expression Encoder 4 Pro. The encoder will dynamically transcode the incoming video into the right format for outputting to Apple Devices as well as Silverlight clients. It'll also archive the video for later.

Start by selecting Live Smooth Streaming, and create a Publishing Point.

image

Make sure to select "Enable Output to Apple Mobile Digital Devices" and also Enable iOS 3.0 Compatibility for older phones.

 Add Publishing Point

In the Encoder I can also add multiple live sources (multiple cameras, USB, Firewire, whatever) as well as recorded sources (we'll be right back, advertisements, whatever) and then cue them up appropriately.

Fire up Encoder 4 and select Live Broadcasting Project.

Load a new project

Add a Live Source (I used my Webcam, although Firewire and other video sources work great), then select the H.264 Encoding you want. I chose "H.264 IIS Smooth Streaming iPad Wifi"

image

Under Output, enter in your Publishing Point. This is a file on your IIS server. Mine was http://hexpower7/livestreaming/magicalgoodness.isml because that's the folder I started the Publishing Point in.

Click Cue and cue your live source, then Start the broadcasting.

Create an HTML page like before, this time using the ISML file.

<html>
<head>
<title>iPhone Live Baby!</title>
</head>
<body>
<h1>Live Stream</h1>

<video width="640"
height="360"
src="magicalGoodness.isml/manifest(format=m3u8-aapl).m3u8"
autoplay="true"
controls="true" >Live</video>
</body>
</html>

That's it! Here's a shot of me Encoding while streaming to an iPhone 4.

Me streaming LIVE locally to myself on my iPhone

Plus, just for fun (and to make a point) I can take the Silverlight Player from the first step at the beginning of the blog post (or make my own with the Silverlight Media Framework 2.2) and just point its source to my live streaming endpoint: "<MediaSource>MagicalGoodness.isml%5CManifest</MediaSource>" and I'm using the same live video feed to serve video to my Silverlight client, my iPhone and iPad. Also, the new Silverlight Media Framework 2.2 supports Windows Phone 7 as well. ;)

 ScottGu720pTest1 - Windows Internet Explorer (2)

And, it's not just a LIVE stream, but it's a live stream with DVR support, so folks can pause, back up, etc and it's all being archived to my server.

image

All this about a half an hour of setup. I was pretty impressed at how easy it was and with a little planning, I think I could this up for a local User Group with multiple lives streams and broad client support with some hardware a few hours. You might think about trying it for your local event as a practice, Dear Reader.

Related Links

About Scott

Scott Hanselman is a former professor, former Chief Architect in finance, now speaker, consultant, father, diabetic, and Microsoft employee. He is a failed stand-up comic, a cornrower, and a book author.

facebook twitter subscribe
About   Newsletter
Sponsored By
Hosting By
Dedicated Windows Server Hosting by SherWeb

PDC10: Building a Blog with Microsoft "Unnamed Package of Web Love"

November 5, '10 Comments [56] Posted in ASP.NET | ASP.NET MVC | NuGet | PDC | Speaking | VS2010 | WebMatrix
Sponsored By
imageI had a lovely time in Redmond this last week attending Microsoft's PDC (Professional Developer's Conference) 2010.

I did lots of stuff, met lots of people, did hours of "Live TV" on Channel 9 Live and presented a talk on behalf of the Web Platform and Tools team (WPT is IIS, Media Streaming, ASP.NET, everything web, etc) showcasing all the cool stuff we've been doing lately.

You can stream the talk with Silverlight Smooth Streaming as well as PIP (Picture in Picture) or you can download the talk in one of four formats. Note that for some weird reason there's 10 minutes of "we haven't started yet" dead video that you'll have to seek past. You should Right-Click Save As the video you want.

UPDATE: All these videos have been re-squished with better in-sync audio and a shorter lead-in.

If you noticed in my recent post introducing NuGet (was NuPack) I mentioned at the bottom that I was changing roles at Microsoft. I feel strongly that the Lego pieces we're building are the right size and that the Web Platform is moving in a direction that I want to influence more than I could while working in MSDN. I'll be working with the Razor, WebMatrix, IIS Express, SQL 4 CE+EF and ASP.NET WebForms and MVC teams building community on what I lovingly call "Unnamed Package of Web Thingies." Oh, and I'll also help name this stuff. Trust me, I'm voting against the usual names you've come to love us for. Anyway.

I gave this talk. A pile of folks were involved even though it's one random dude (me) on stage to make demos like this happen. PDC talks are also kind of "here's what we're thinking and what you could do in the future" type talks, so there was a LOT of daily builds and a lot of hacks/rough code that was put together for that talk. I wanted to share it with you so you can play with it and be involved in the conversation. The scaffolding stuff I showed was a proof of concept built on a prototype resting on a sample. That said, it's certainly a direction we want to head in.

In this post I'll show you how you can do the same demo yourself (mostly) with today's bits. I'll show you what'll work today with huge hacks and bug fixes, and how it'll be better tomorrow. I'm interested in your thoughts on how you'd like things to work tomorrow. I'm mostly interested in how you/us (the community) will build better stuff with these Lego, rather than just taking this stuff and callign it done. Cause it ain't done.

Building a Blog with Microsoft "Unnamed Package of Web Love"

Of course, watch the video to see how I do it and follow along, but here's what you'll need first.'

Remember, you asked for it, so this is the crazy version. The some alternate future version would more likely be "Get VS2010 and one other thing" and it'll just work™.

  • Visual Studio 2010 and ASP.NET MVC 3.
    • Get whatever version is current. As of today, that's ASP.NET MVC 3 Beta. An RC will be out soon. Doesn't matter, most of this will work. I'll point out weird stuff as we go.
  • NuGet Alpha Build from 10-26
    • Get this alpha and install the VSIX by double clicking it. It's alpha, so expect you WILL have to uninstall it at some point. No it won't mess up your system.
    • In Tools | Options | Package Manager, make sure you're pointing to http://go.microsoft.com/fwlink/?LinkId=204820. That's currently a redirect to http://feed.nuget.org/ctp2/odata/v1/. At some point in the future it'll be automatic and easy.
    • Worst case scenario and you want to uninstall it, Run VS2010 as Admin, to go Tools | Extension Manager and Uninstall it.

Again, disclaimer, this isn't a tutorial for new folks to learn ASP.NET MVC. This is a behind the curtains transparent "here's what I did for this proof of concept demo and I'm interested in your feedback as fellow geeks." Read carefully.

Here's the demo.

New Project

Go File | Note the new View Engine drop down. Perhaps Spark, Nhaml, etc will get in this dropdown at some point. We'll choose Razor. Run the default template. Note that it's using Visual Web Development Server since that's what's built in.

Alpha/Beta Software Hack Warning: Now, let's pretend this is a "Choose Your Own Adventure" book. You can skip this part, or you can do this hack.

I was using daily build stuff that enabled IIS Express to be installed on its own outside of WebMatrix (our small, free Web Development IDE). You'll get this ability in the future and it'll be built into Visual Studio in some way similar to my demo. For now, you can either.

1. Don't sweat IIS Express and move on. Continue to use Visual Web Developer. Things will work fine.

or

2. Try to hack IIS Express in. Here's how, assuming you have WebMatrix installed.

Open Notepad, paste in:

SET ExecPath=%ProgramFiles(x86)%
IF "%ExecPath%" == "" SET ExecPath = "%ProgramFiles%

"%ExecPath%\Microsoft WebMatrix\iisexpress.exe" /path:"C:\PATHTOMY\MvcApplication3" /port:12345

Save this file as "thisisahack.bat" somewhere and run it. You'll start and stop this on your own, remembering that it'll be built in someday. Go into your ASP.NET MVC applications Properties, click Web, the under "Use Custom Web Server" click that radio button and paste in http://localhost:12345 as the URL.

Get a package with NuGet...I need a DB

Now, I need a database, so I'll use the new SQLCE with the Entity Framework. From inside Visual Studio, show the Package Manager Console. Two ways, either Ctrl-W then Ctrl-Z or View | Other Windows | Package Manager Console.

Alpha/Beta Software Warning: From inside the NuGet console, you'll install a bunch of beta libraries with this command:

Install-Package SQLCE.EntityFramework

After this you'll get:

Successfully installed 'SQLCE.EntityFramework 4.0.8435.1'
Successfully added 'SQLCE 4.0.8435.1' to MvcApplication3
Successfully added 'EFCTP4 1.0' to MvcApplication3
Successfully added 'WebActivator 1.0.0.0' to MvcApplication3
Successfully added 'SQLCE.EntityFramework 4.0.8435.1' to MvcApplication3

Note the stuff that got added to your references and the new AppStart_SQLCEEntityFramework.cs that showed up. This is using the EFCTP4 (Entity Framework Magic Unicorn) I've blogged about before as well as SQL Compact Edition 4, a tiny xcopyable file based SQL Server. BOTH of these are not released products. All these came down off the live NuGet feed and will most likely be updated as they can be.

Create Models

Make a new file in the Models folder called something like BlogContext.cs. Put this in it and confirm you have the right namespaces. Something like this:

using System;
using System.Collections.Generic;
using System.Data.Entity;

namespace MvcApplication3.Models //yours will be different
{
public class Post
{
public int ID { get; set; }
public string Title { get; set; }
public string Text { get; set; }
public DateTime PublishDate { get; set; }
public virtual ICollection<Comment> Comments { get; set; }
}

public class Comment
{
public int ID { get; set; }
public int PostID { get; set; }
public string Text { get; set; }
public string Author { get; set; }

}

public class BlogContext : DbContext
{
public DbSet<Post> Posts { get; set; }
public DbSet<Comment> Comments { get; set; }
}
}

This should compile at this point. Now I need a controller for this blog of ours. Instead of making in with the Add Controller dialog, I'll go back to the Package Manager Console.

Create Controller for Blog

Install a new package...this time some totally horrible experimental scaffolding that started with David Ebbo, squeezed by David Fowler, then ruined by myself.

Alpha/Beta Software Hack Warning: There's blood on all our hands for this one. Feel free to dig into this package, if you can handle the evil, and look at some of the ideas being explored.

Install-Package MvcScaffold

They made me rename it. You'll be getting version 0.2. In this case, we're still using NuGet, except in an unusual way. We're not adding libraries to the project, rather we're extending the PowerShell environment with new functions.

Make sure you compile at least once before you run this! The PowerShell scripts are going to wander around in the project looking for types and they won't exist as CLR types if they haven't been compiled. One of the added method was Scaffold-Controller:

Scaffold-Controller -m Post -c BlogContext

This will creates the controller against the context and also create CRUD views.

image Note that you have a new folder created in your project called CodeTemplates. I've talked about T4 (Text Template Transformation Toolkit) before as well as in a recorded talk in the Netherlands called "ASP.NET MVC 2: Ninja Black Belt Tips" while wearing a horrible sweater.

This CodeTemplates folder is a folder that you'll actually find  deep inside the beast around: C:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE\VWDExpress\ItemTemplates\CSharp\Web\MVC 3 or something similar. As I show in my Ninja video, you can copy those templates out and override them simply by putting them local to your project. Once they are local, they can be edited to do whatever we like.

Hack Warning: I'm adding these to the project and we're using them in a slightly unconventional way. T4 templates - as the tooling is today - assume they will run before your build or when a file or your project is dirty/changed. Adding all these T4 templates at once causes Visual Studio to freak out and pop up "run this template" dialogs for every template. Also, once they are added they have a Custom Tool associated with them. I have done one thing and I propose another. First, I changed the extension of these files from .tt to .t4 to prevent that dialog from coming up. Changing the extension makes Visual Studio think nothing of them. Now, I propose that 3rd party T4 tools should know about this extension and enable syntax highlighting for these files. An extension of .TT runs as usual and the .T4 extension means no tooling runs. This way .T4 files can be includes, or in my example/demo here, just files that are "along for the ride and used occasionally."

Back to the Scaffolding. The Scaffold-Controller method used the ControllerWithContext.t4 template to generate the Controller for me directly against the Entity Framework CTP4 DbContext. The Views were generated from the Razor views you see in the CSHTML folder.

Compile and it should run.

Run application and visit /Post

At this point, I actually goofed up but I didn't want to change the scaffolding as it was 3am when I realized it. I should have pluralized things and make a Posts (plural) Controller rather than a PostController. I'd like to have the Entity Framework team give me a public API for "PluralizeThisThing()" or something. There's no reason for me to know about Octopus:Octopi.

Anyway, hit http://localhost:12345/Post and you'll see:

A super basic MVC page with Create New on it.

Hit Create New, add a Post and it'll show up in the list. Yay.

Index - Windows Internet Explorer (8)

Now, if you get this YSOD (Yellow Screen of Death) with SQL Compact Edition 4 CTP that declares "File already exists. Try using a different database name" here's the deal. Remember what I said about Hacks and Bugs?

File already exists. Try using a different database name. [ File name = CUsersScottDesktop - Windows Internet Explorer

Hack Bug Work Around Warning: There's a bug in the SQL Compact Edition 4 CTP (CTP==Beta) that gets confused about if the database exists or not. If you hit this, for now, hit F5.

Choose Your Own Adventure Time. You can:

1. Hit F5 and it'll just work©

2. Change the Database Initializer (you can do that, since things are more pluggable in Microsoft land now) and introduce this hack. Yes, it'll be fixed before it comes out.

Go into your AppStart_SQLCEEntityFramework.cs file and the DefaultConnectionFactory line to this:

Database.DefaultConnectionFactory = new SqlCeConnectionFactory(
    "System.Data.SqlServerCe.4.0",
    HostingEnvironment.MapPath("~/App_Data/"),"");

At this point, you should be able to:

  • See listing of posts (although none there to begin with)
  • Create New Post, now it shows up in list.
  • Click details to see the post.

At this point I have CRUD, I have a "just in time created SQL Compact Edition DB" in my App_Data folder. The code in my controller is going against the Entity Framework DbContext directly.

Trick: In my talk I showed Tooling that will let you open the SDF (SQL Database File) and make tables, edit things, etc. You don't have that because it's coming. There ARE tools out there though, if you really really want to see inside the SQL Compact Edition 4 database, like http://sqlcetoolbox.codeplex.com. But you didn't hear it from me.

Now, do note that the CRUD (Create, Read, Update, Delete) views were generated off the Model. Our blog posts have Title and Text and both are a single line. A blog's contents needs multiline text boxes (textareas) so there's two things we could do. If the template generated TextBoxFor() or TextAreaFor() based on some metadata, we could add [DataType(DataType.MultilineText)] to the Text field and regenerate the views passing in -f for "Force." Try typing a dash, the hitting tab. You get Intellisense in this PowerShell, also.

Screenshot of NuGet PowerShell Console with an Intellisense Combo popped up

But, since I've changed my templates to use EditorFor, I just need to add [DataType(DataType.MultilineText)] to the Text property of my Database and run my app again. The right Editor (TextBox or TextArea) is chosen at runtime, not compile- or scaffold-time.

Make Views pretty

The main list of posts is a table, and is not really nice for a blog. Let's look at the Razor syntax and make it the site nicer. Also, I'll change the Site.css to use the one from our designer. You can get the Site.css from here.

With Razor, if I want to spin through some posts, I could delete the generated table and add this to my Post\Index.cshtml file:

<div class="blog-posts">
@foreach (var item in Model) {
@item.Text
}
</div>

Or even just this. Notice how it knows that the ! isn't part of the expression?
<div>@item.Text!</div>

I'll just spin through the posts, set some CSS and layout my page.

<div class="blog-posts">
@foreach (var item in Model) {
<ul class="blog-post">
<li class="blog-title">@Html.ActionLink(item.Title, "Details", new { id=item.ID })</li>
<li class="blog-content">@MvcHtmlString.Create(item.Text)</li>
<li class="meta">Publish Date</li>
<li class="meta">@String.Format("{0:g}", item.PublishDate)</li>
<li class="meta">
@Html.ActionLink("Edit", "Edit", new { id=item.ID }) |
@Html.ActionLink("Details", "Details", new { id=item.ID }) |
@Html.ActionLink("Delete", "Delete", new { id=item.ID })
</li>
</ul>
<hr />
}
</div>

And here's better code for Post\Details.cshtml:
<h2 class="blog-title">@Model.Title</h2> 
<ul class="blog-post">
<li class="blog-content">@MvcHtmlString.Create(Model.Text)</li>
<li class="meta">Publish Date</li>
<li class="meta">@String.Format("{0:g}", Model.PublishDate)</li>
</ul>

Beta Note: Those places where we have MvcHtmlString.Create will probably turn into something easier like Html.Raw(). All Html strings are encoded unless you explicitly say otherwise.

You can hit refresh in browser after each change in Razor to incrementally see your page changing.

Better models with validation attributes

We don't have any data validation so I'll add add [Required] for everything, and StringLength[50] for Title, and whatever else you like. If I run my app now, I've got data validation.

 Create - Windows Internet Explorer (14)

However, this validation doesn't work until I click Create. It's server-side. Useful, but not taking advantage of JavaScript.

Beta Note: This is all stuff that could be set in the default template, meaning we wouldn't have to do any of this by default. Maybe just comment out some JS.

Let's turn on client-side validation in web.config. Note that it's using unobtrusive JavaScript, which is a fancy way of saying "no generated javascritp arrays." It's a good thing and it makes for tidier HTML:

<add key="ClientValidationEnabled" value="true" />
<add key="UnobtrusiveJavaScriptEnabled" value="true" />

Add these scripts to Create.cshtml and Edit.cshtml:

<script src="/Scripts/jquery-1.4.1.js" type="text/javascript"></script>
<script src="/Scripts/jquery.validate.js" type="text/javascript"></script>
<script src="/Scripts/jquery.validate.unobtrusive.js" type="text/javascript"></script>

Now, if we re-run the application and try to create a post, validation works great on the client side. This is all new in ASP.NET MVC 3.

Add Comments

Our blog needs comments. Let's not scaffold comments, just add a controller without CRUD using Right Click | Add Controller.

public class CommentController : Controller
{
public ActionResult Create(Comment c)
{
BlogContext db = new BlogContext();
Post post = db.Posts.Single(p => p.ID == c.PostID);
post.Comments.Add(c);
db.SaveChanges();
return RedirectToAction("Details", "Post", new { ID = c.PostID });
}
}

Then we'll modify the Details.cshtml view and have it include a Form to POST to the Comments controller.

Beta Note: You won't have syntax highlighting or intellisense for Razor yet. Soon.

@using(Html.BeginForm("Create", "Comment")) {
<div id="comments">
<input type="hidden" name="PostID" value="@Model.ID" />
<label for="Author">Email:</label> <input type="text" name="Author"/>
<label for="Text">Comment:</label> <textarea name="Text" rows="5" cols="40"></textarea>
<input type="submit" value="Add Comment"/>
</div>
}

And then add a foreach within Details to list the comments, if there are any.

<h2 class="comments-title">Comments</h2>
<ul class="comments">
@foreach(var comment in Model.Comments) {
<li class="group">
@comment.Author says: @comment.Text
</li>
}
</ul>

This is lovely, and will work if you run it, but I didn't use the Repository Pattern before when accessing the database. Let's rescaffold the PostController, except this time with a basic Repository.

Change to a Repository Pattern

It sure would be nicer if I could do something like this:

IPostRepository repo;

public CommentController(IPostRepository r)
{
repo = r;
}

public ActionResult Create(Comment c)
{
Post p = repo.GetById(c.PostID);
p.Comments.Add(c);
repo.Save();
return RedirectToAction("Details", "Post", new { ID = c.PostID });
}

I can make a Repository class in \Models with this:

Scaffold-Repository -m Post -c BlogContext

Let's change Post to use a Repository...make sure to include Force and NoViews. Force forces overwriting and -NoViews doesn't do Views.

Scaffold-Controller -m Post -c Blog -r Post -f -NoViews

Then I'll check out the code for the PostController and add a little sort since this is a blog.

return View(this.repository.GetAllPosts().OrderByDescending(p => p.PublishDate));

But, if I run this, notice that I never actually MADE an IPostRepository. This is a good time for IoC (Inversion of Control)

Add IOC

We will add not just Ninject (a popular IoC container) but an MVC3 specific package that will help us wire things up.

Install-Package Ninject.MVC3

Now, look at AppStart_NinjectMVC3.cs. Add this:

kernel.Bind<IPostRepository>().To<PostRepository>();

...and the app will run again as Ninject is using the new DependencyResolver in ASP.NET MVC 3.

// Tell ASP.NET MVC 3 to use our Ninject DI Container 
DependencyResolver.SetResolver(new NinjectServiceLocator(kernel));

Bring in a Helper for Gravatars

I'd like to add some Gravatars to my comments as they look boring.

Boring Gravatarless Comments

Again, back to the Package Management Console:

Install-Package microsoft-web-helpers

This is an interesting package, however, as it's filled with Razor Helpers that are great for using in ASP.NET Web Pages (that's the simpler stuff that you see in WebMatrix). These are effectively WebMatrix Helpers. But, since they are Razor and this is all ASP.NET, they work nicely.

@Gravatar.GetHtml(comment.Author)

Then, I just F5 in the browser...and:

Details - Windows Internet Explorer

Finally in my demo I added Windows Live Writer support using Charles Cook's XML-RPC library, much like I did in this recent blog post. That's more involved so I'll talk about that other day.

Conclusion

Things are coming together nicely with Visual Studio 2010 plus Microsoft "Unnamed Package of Web Love." There's more to come and more needed, like Database Migrations.

In the short term, the MvcScaffold package is a 0.2 prototype. I think it could be the beginnings of an interesting generic scaffolding that could be extended by the community (you could share CodeTemplate folders) and extended to do Tests, BDD, whatever. We are interested in your thoughts about how T4-based dynamic-using templates could be used along with Razor, SQLCE4, EF Code-First as well as the pile of awesome Open Source Libraries getting added to the http://www.nuget.org project daily.

Your thoughts, Dear Reader?

Related Links

About Scott

Scott Hanselman is a former professor, former Chief Architect in finance, now speaker, consultant, father, diabetic, and Microsoft employee. He is a failed stand-up comic, a cornrower, and a book author.

facebook twitter subscribe
About   Newsletter
Sponsored By
Hosting By
Dedicated Windows Server Hosting by SherWeb

This Developer's Life 1.0.7 - Audacity

November 5, '10 Comments [6] Posted in Podcast
Sponsored By

John, Alex and Miguel In this episode Rob and I talk to 3 developers who have pulled off some pretty audacious maneuvers:

  • John Resig, Creator of jQuery, decided to write yet another JavaScript framework when there were already quite a few to choose from. Not only did he succeed, he changed web development forever.
  • Alex Payne, former API Developer Lead at Twitter, talks about the (somewhat insane) move to leave Twitter, and (somewhat more insane) choice to “reinvent banking” with BankSimple.
  • Miguel de Icaza, founder of the GNOME project and creator of Mono talks about creating a clone of the .NET framework - all because he didn’t like what else was out there at the time for Linux.

    Please consider subscribing with iTunes, or Zune. Or if you have a BitTorrent client and would like to help save us bandwidth money, as well as the bragging rights of downloading legal torrents via RSS, get our Torrent Feed at ClearBits.

    You can download here - 54 minutes

    The bandwidth and other costs for this week's show were picked up by Twilio:

    Need SMS or Voice call capabilities for your application? Check out Twilio.

    … and Umbraco

    “Priceless CMS at the cost of virtually nothing”

    Also, big thanks to MaximumASP for helping us out with bandwidth for the first few shows while we got our act together!

    About Scott

    Scott Hanselman is a former professor, former Chief Architect in finance, now speaker, consultant, father, diabetic, and Microsoft employee. He is a failed stand-up comic, a cornrower, and a book author.

    facebook twitter subscribe
    About   Newsletter
    Sponsored By
    Hosting By
    Dedicated Windows Server Hosting by SherWeb

    Our Tenth Anniversary

    October 25, '10 Comments [87] Posted in Musings
    Sponsored By

    My wife and I got married three times. The first time was ten years ago today. Here's a picture of us on our first date. We were married three months later.

    Mo and Scott First Date

    Wedding One

    Mo and Scott WeddingIn a small public park in Delaware my wife and I were married. We showed up at the park to meet the judge. He said "where are your witnesses?"

    "Witnesses? Um..." I had totally forgotten we needed witnesses.

    "You're going to need two folks to sign your certificate and give their details."

    "Ok, hang on." I ran over to the jogging path at the park and stopped the first two people that ran by.

    "Hi, I know this is weird, but we're getting married and we need two witnesses. Do you have a minute?"

    We got married right there under a big tree with a judge and two joggers. Afterwards we went to Friendly's for dinner. It was glorious and wonderful.

    Wedding Two

    Officially married on paper, we moved into together and started planning our White Wedding. Everyone was there.

    We worked together in the design of the wedding. I searched the web for "Scottish Interracial Cake Topper" and we had a custom one made for the cake. It had tiny classes and a goatee as well as tiny African cloth in a design that matched Mo's hair wrap and bouquet.

    We did the whole thing on the cheap, but it looked like a million bucks. Used dress, MP3s for DJs and burned CDs for party favors. To this day we both pride ourselves on making the old look like new.

    A tiny asterisk said "Mo and Scott were married last year in a private civil ceremony." To this day I still get my anniversary dates confused.

    Wedding Three

    We went to Zimbabwe to settle lobola. One day while there I was padding into the bathroom to take a bath and when I opened the door I noticed there was a goat in the tub. I closed the door and reopened it just to check. Yep, goat. Not knowing exactly what was going on, I decided to go back to bed. A few hours later I came out, went to the tub and, no goat.

    After a bath I joined the family for breakfast. They made a big deal about giving me some special meat for breakfast. It was the bathtub goat. Unfortunately I said no thanks, and ten years later this faux pas haunts me. Pro tip: Eat what's offered to you. Of course, this was just the first of a thousand cultural goats, I mean, goofs, and I'm sure there will be a thousand more.

    It's been ten years, two boys, three houses, three jobs, three college degrees, dozens of trips, two filled passports, one new citizenship, and seven (voluntary) viewings of Love Actually.

    Ten Years Later

    Happy Tenth Anniversary, Small Wife.

    Related Links

    About Scott

    Scott Hanselman is a former professor, former Chief Architect in finance, now speaker, consultant, father, diabetic, and Microsoft employee. He is a failed stand-up comic, a cornrower, and a book author.

    facebook twitter subscribe
    About   Newsletter
    Sponsored By
    Hosting By
    Dedicated Windows Server Hosting by SherWeb

    Disclaimer: The opinions expressed herein are my own personal opinions and do not represent my employer's view in any way.