Scott Hanselman

The Weekly Source Code 53 - "Get'er Done" Edition - XML in the left hand becomes HTTP POSTs in the right hand

June 26, '10 Comments [13] Posted in Open Source | Source Code
Sponsored By

I wrote some code tonight in about ten minutes with a "Get'er Done" attitude. We all do that (I hope). It's one-time code that will solve a one-time problem. As such, it isn't always pretty, but it often is interesting.

I've got this silly website called OverheardAtHome that is a collection of silly quotes and stories that my kids (and your kids) have said around the house. It was running on DasBlog (just like this blog) for the last year or more, but the sheer workflow of populating the site was getting tiredsome. DasBlog isn't setup for screening external submissions and promoting them to posts, and I wasn't really interested in extending DasBlog in that way.

Instead, I needed to move OverheardAtHome to a hosted blogging solution, preferably a nice free one as it doesn't make any money. It's just a hobby. I like Tumblr so I figured I put it there. Tumblr has a very basic HTTP API and their native UI supports User Submissions, so it seemed like a win.

DasBlog stores its content not in a database, but rather in an XML file per day. I've got a few hundred XML files that make up the whole of the content on OverheardAtHome and it's very basic stuff.

Here's what I wrote, using the Tumblr API from CodePlex written by (I believe) Jeremy "madkidd" Hodges, who is also a Developer on Graffiti CMS, coincidentally. It's a nice little abstraction on top of HttpWebRequest that uses a little HttpHelper class from "rakker."

I figured I could use PowerShell or something script-like, but this was very fast to write. There's no error handling, but interestingly (or not), there were no errors in hundreds of posts.

using System;
using System.Linq;
using System.IO;
using System.Xml.Linq;
using System.Threading;

namespace TumblrAPI.ConsoleApp
{
class Program
{
static void Main(string[] args)
{
Console.WriteLine("Started...");

TumblrAPI.Authentication.Email = "myemail@notyours.com"; // Console.ReadLine();
TumblrAPI.Authentication.Password = "poopypants"; //Console.ReadLine();
Console.WriteLine(TumblrAPI.Authentication.Authenticate().ToString());

if (TumblrAPI.Authentication.Status == TumblrAPI.AuthenticationStatus.Valid) {
Console.WriteLine("Now make some posts...");

DirectoryInfo di = new DirectoryInfo(@"C:\overheardathome\xml");
//Get the DasBlog XML files, they are like <entry><title/><content/></entry> and stuff
FileSystemInfo[] files = di.GetFileSystemInfos("*.dayentry.xml");
var orderedFiles = files.OrderBy(f => f.Name);

XNamespace ns = "urn:newtelligence-com:dasblog:runtime:data";
foreach (FileSystemInfo file in orderedFiles)
{
XDocument xml = XDocument.Load(file.FullName);
var posts = from p in xml.Descendants(ns + "Entry") select p;

foreach (var post in posts)
{
TumblrAPI.Post.Text t = new TumblrAPI.Post.Text();
t.Title = (string)post.Element(ns + "Title");
t.Body = (string)post.Element(ns + "Content");
Thread.Sleep(500); //Tumblr will API limit me if I bash on them.
Console.WriteLine(" Response from text post: {0}", t.Publish());
}
}
}

Console.WriteLine("Done, press any key...");
Console.ReadLine();
}
}
}

Comments? What's a better pattern for left-hand/right-hand bulk crap like this, Dear Reader?

About Scott

Scott Hanselman is a former professor, former Chief Architect in finance, now speaker, consultant, father, diabetic, and Microsoft employee. He is a failed stand-up comic, a cornrower, and a book author.

facebook twitter subscribe
About   Newsletter
Sponsored By
Hosting By
Dedicated Windows Server Hosting by SherWeb

The Weekly Source Code 52 - You keep using that LINQ, I dunna think it means what you think it means.

June 18, '10 Comments [25] Posted in ASP.NET | Data | Learning .NET | LINQ | Source Code
Sponsored By

Remember good developers don't just write source code, they also READ it. You don't just become a great poet by writing lots of poems. Read and absorb as well. Do check out the Source Code category of my blog here, there is (as of today) 15 pages of posts on Source Code you can check out.

Recently my friend Jonathan Carter (OData Dude, my name for him) was working with a partner on some really weird stuff that was happening with a LINQ to SQL query. Remember that every abstraction sometimes leaks and that the whole port of an abstraction is "raise the level" so you don't have to worry about something.

Plumbing is great because it abstracts away water delivery. For all I know, there's a dude with a bucket who runs to my house when I turn on the tap. Doesn't matter to me, as long as I get water. However, sometimes something goes wrong with that dude, and I don't understand what's up with my water. This happened to JC and this partner.

In this example, we're using the AdventureWorks Sample Database to make this point. Here's some sample code the partner sent us to reproduce the weirdness.

protected virtual Customer GetByPrimaryKey(Func<customer, bool> keySelection)
{
AdventureWorksDataContext context = new AdventureWorksDataContext();

return (from r in context.Customers select r).SingleOrDefault(keySelection);
}

[TestMethod]
public void CustomerQuery_Test_01()
{
Customer customer = GetByPrimaryKey(c => c.CustomerID == 2);
}

[TestMethod]
public void CustomerQuery_Test_02()
{
AdventureWorksDataContext context = new AdventureWorksDataContext();
Customer customer = (from r in context.Customers select r).SingleOrDefault(c => c.CustomerID == 2);
}

CustomerQuery_Test_01 calls the GetByPrimaryKey method. That method takes a Func as a parameter. He's actually passing in a lamdba expression into the GetByPrimaryKey function. That makes the method reusable and is the beginning of some nice helper functions for his DAL (Data Access Layer). He's split up the query into two places. Seems reasonable, right?

Well, if you run this in Visual Studio - and in this example, I'll use the Intellitrace feature to see the actual SQL that was executed, although you can also use SQL Profiler - we see:

Wrong SQL in the Watch Window

Here's the query in text:

SELECT [t0].[CustomerID], [t0].[NameStyle], [t0].[Title], 
[t0].[FirstName], [t0].[MiddleName], [t0].[LastName],
[t0].[Suffix], [t0].[CompanyName], [t0].[SalesPerson],
[t0].[EmailAddress], [t0].[Phone], [t0].[PasswordHash],
[t0].[PasswordSalt], [t0].[rowguid], [t0].[ModifiedDate]
FROM [SalesLT].[Customer] AS [t0]

Um, where's the WHERE clause? Will LINQ to SQL kill my pets and cause me to lose my job? Does Microsoft suck? Let's take a look at the second query, called in CustomerQuery_Test_02():

SELECT [t0].[CustomerID], [t0].[NameStyle], [t0].[Title], 
[t0].[FirstName], [t0].[MiddleName], [t0].[LastName],
[t0].[Suffix], [t0].[CompanyName], [t0].[SalesPerson],
[t0].[EmailAddress], [t0].[Phone], [t0].[PasswordHash],
[t0].[PasswordSalt], [t0].[rowguid], [t0].[ModifiedDate]
FROM [SalesLT].[Customer] AS [t0]
WHERE [t0].[CustomerID] = @p0

OK, there it is, but why does the second LINQ query cause a WHERE clause to be emitted but the first doesn't? They look like basically the same code path, just one is broken up.

The first query is clearly returning ALL rows to the caller, which then has to apply the LINQ operators to do the WHERE in memory, on the caller. The second query is using the SQL Server (as it should) to do the filter, then returns WAY less data.

Here's the deal. Remember that LINQ cares about two things, IEnumerable stuff and IQueryable. The first lets you foreach over a collection, and the other includes all sorts of fun stuff that lets you query that stuff. Folks build on top of those with LINQ to SQL, LINQ to XML, LINQ to YoMomma, etc.

When you are working with something that is IQueryable; that is, the source is IQueryable, you need to make sure you are actually usually the operators for an IQueruable, otherwise you might fall back onto an undesirable result, as in this database case with IEnumerable. You don't want to return more data from the database to a caller than is absolutely necessary.

From JC, with emphasis mine:

The IQueryable version of SingleOrDefault, that takes a lambda, actually takes an Expression>, whereas the IEnumerable version, takes a Func. Hence, in the below code, the call to SingleOrDefault, is treating the query as if it was LINQ To Objects, which executes the query via L2S, then performs the SingleOrDefault on the in memory collection. If they changed the signature of GetByPrimaryKey to take an Expression>, it would work as expected.

What's a Func and what's an Expression? A Func<> (pronounced "Funk") represents a generic delegate. Like:

Func<int,int,double> divide=(x,y)=>(double)x/(double)y;
Console.WriteLine(divide(2,3));

And an Expression<> is a function definition that can be compiled and invoked at runtime. Example"

Expression<Func<int,int,double>> divideBody=(x,y)=>(double)x/(double)y;
Func<int,int,double> divide2=divideBody.Compile();
write(divide2(2,3));

So, the partner doesn't want a Func (a Func that takes a customer and returns a bool, they want a compliable Expression with a Func that takes a Customer and returns a bool. I'll have to add "using System.Linq.Expressions;" as well.

protected virtual Customer GetByPrimaryKey(Expression<Func<customer,bool>> keySelection)
{
AdventureWorksDataContext context = new AdventureWorksDataContext();

return (from r in context.Customers select r).SingleOrDefault(keySelection);

}

[TestMethod]
public void CustomerQuery_Test_01()
{
Customer customer = GetByPrimaryKey(c => c.CustomerID == 2);
}

[TestMethod]
public void CustomerQuery_Test_02()
{
AdventureWorksDataContext context = new AdventureWorksDataContext();
Customer customer = (from r in context.Customers select r).SingleOrDefault(c => c.CustomerID == 2);
}

Just changed that one line, so that GetByPrimaryKey takes a Expression> and I get the SQL I expected:

Corrected SQL in the Watch Window

Someone famous once said, "My code has no bugs, it runs exactly as I wrote it."

Layers of Abstraction are tricky, and you should always assert your assumptions and always look at the SQL that gets generated/created/executed by your DAL before you put something into production. Trust no one, except the profiler.

About Scott

Scott Hanselman is a former professor, former Chief Architect in finance, now speaker, consultant, father, diabetic, and Microsoft employee. He is a failed stand-up comic, a cornrower, and a book author.

facebook twitter subscribe
About   Newsletter
Sponsored By
Hosting By
Dedicated Windows Server Hosting by SherWeb

Extending NerdDinner: Exploring Different Database Options

May 20, '10 Comments [37] Posted in ASP.NET | ASP.NET MVC | Data | NerdDinner | Open Source | Source Code
Sponsored By

The original NerdDinner Sample was very simple. Two samples, simple, in fact. Perhaps it's the new Northwind, as it's a good way to start working with ASP.NET MVC. However, it's not a perfect sample or an idealized example on how to do many things that users want to do.

Fortunately, there's been lots of cool folks in the community who have "forked" NerdDinner and done interesting stuff with it. Each of these samples is usually focused on a specific scenario, so they won't necessarily be merged with the trunk, but they are educational nonetheless.

Four Five NerdDinners - Each Accessing Data Differently

When NerdDinner was originally done, I did it in Linq To SQL as L2S was fast, easy, had a 1:1 relationship between objects and tables, and frankly, I wasn't really feeling Entity Framework 3.5. Fast forward to .NET 4 and the Entity Framework 4 is pretty nice. The current NerdDinner v2 sample (available in the Trunk on the NerdDinner Codeplex Site, click Source Control) uses Entity Framework 4.

I heard, however, that Chris Sells was interested in exploring a ASP.NET MVC sample that accessed databases via the various ways you'll find in the wild:

  • ADO.NET Connected (DataReaders)
  • ADO.NET Disconnected (DataSets)
  • LINQ to SQL
  • LINQ to Entities (Entity Framework)

Chris worked with Nick Muhonen from Useable Concepts and Nick created four samples. Nick has posted a very in-depth article on http://msdn.com/data.

These samples are on VS2010 and ASP.NET MVC 2. Let's compare and contrast. I've also included a sample that Ayende built for to teach me NHibernate last year as the fifth sample. Big thanks to Ayende for all he does for the community, giving of his time, and for keeping Microsoft honest(ish).

ADO.NET Connected (DataReaders)

Scandalous! People still use these? Of course they do. The are wicked fast and many generated DALs (data access layers) have a DataReader at their heart.

Here's the slightly modified IDinnerRepository that was used in this first sample. Note it's not using IQueryable. I understand that it would be ideal to have a single IDinnerRepository interface and have all these samples share it, but these database access techniques differed so greatly that I'm told they gave up as it made the rest of the code smell (more) just to meet that one goal.

public interface IDinnerRepository {

//Data Access Methods

IEnumerable<Dinner> FindAllDinners();
IEnumerable<Dinner> FindByLocation(float latitude, float longitude);
IEnumerable<Dinner> FindUpcomingDinners();
Dinner GetDinner(int id);

void AddDinner(Dinner dinner);
void UpdateDinner(Dinner dinner);
void DeleteDinner(int id);

void AddDinnerRsvp(int dinnerID, RSVP rsvp);
}

You've likely seen code like this before. At least it's not concatenating the SQL manually! It could also be a sproc. The pattern remains.

public IEnumerable<Dinner> FindByLocation(float latitude, float longitude)
{
using (var connection = new SqlConnection(this.connectionString))
{
var commandText =
@"
select d.DinnerID, d.Title, d.EventDate, d.[Description], d.HostedBy,
d.ContactPhone, d.[Address], d.Country, d.Latitude, d.Longitude
from Dinners d
inner join dbo.NearestDinners(@Latitude,@Longitude) nd on
d.DinnerID = nd.DinnerID
where @CurrentDate <= d.EventDate
order by d.DinnerID

select r.RsvpID, r.DinnerID, r.AttendeeName from RSVP r
inner join Dinners d on
d.DinnerID = r.DinnerID
inner join dbo.NearestDinners(@Latitude,@Longitude) nd on
d.DinnerID = nd.DinnerID
where @CurrentDate <= d.EventDate
order by r.DinnerID, r.RsvpID
";
var command = new SqlCommand(commandText, connection);
var parameters = new[]{
new SqlParameter{ParameterName = "Latitude", DbType = DbType.Double, Value = latitude},
new SqlParameter{ParameterName = "Longitude", DbType = DbType.Double, Value = longitude},
new SqlParameter{ParameterName = "CurrentDate", DbType = DbType.Date, Value = DateTime.Now}};

command.Parameters.AddRange(parameters);

connection.Open();
return GetDinners(command);
}
}

Here's a snippet of the private method, GetDinners, that does the tearing apart of the DataReader and turning it into object(s):

private List<Dinner> GetDinners(SqlCommand command)
{
var returnDinners = new List<Dinner>();
using (var reader = command.ExecuteReader())
{
//Project first result set into a collection of Dinner Objects

while (reader.Read())
{
var dinner = new Dinner()
{
DinnerID = (int)reader["DinnerID"],
Title = (string)reader["Title"],
Description = (string)reader["Description"],
Address = (string)reader["Address"],
ContactPhone = (string)reader["ContactPhone"],
Country = (string)reader["Country"],
HostedBy = (string)reader["HostedBy"],
EventDate = (DateTime)reader["EventDate"],
Latitude = (double)reader["Latitude"],
Longitude = (double)reader["Longitude"]
};
returnDinners.Add(dinner);
}
//...

Pretty classic stuff. I generated TONS of for many years using tools like CodeSmith and T4. Generated code is best not seen. Plus, if you write this by hand, a lot can go around and it's almost always because of copy-paste errors. The compiler can't save you if half your code is written in another language tunneled inside a string.

ADO.NET Disconnected (DataSets)

I once called DataSets the spawn of Satan and destroy of all that is holy. I stand by that. ;) They have a way of leaking all over the place, as exemplified by this IDinnerRepository interface. Look away!

public interface IDinnerRepository {

//Data Access Methods
IEnumerable<NerdDinnerDataSet.DinnerRow> FindAllDinners();
IEnumerable<NerdDinnerDataSet.DinnerRow> FindByLocation(float latitude, float longitude);
IEnumerable<NerdDinnerDataSet.DinnerRow> FindUpcomingDinners();
NerdDinnerDataSet.DinnerRow GetDinner(int id);
void AddDinner(NerdDinnerDataSet.DinnerRow dinner);
void DeleteDinner(NerdDinnerDataSet.DinnerRow dinner);

void AddDinnerRsvp(NerdDinnerDataSet.DinnerRow dinner, NerdDinnerDataSet.RSVPRow rsvp);
void DeleteRsvp(NerdDinnerDataSet.RSVPRow rsvp);

// Persistence Method
void Save();
//Object factory methods

NerdDinnerDataSet.DinnerRow CreateDinnerObject();
NerdDinnerDataSet.RSVPRow CreateRsvpObject();
}

With apologies to the original creator of the Regular Expression joke, I will co-opt it for this new one:

So you've got a problem, and you've decided ADO.NET DataSets to solve it. So, you've got two problems... - Me

What does FindByLocation look like now?

public IEnumerable<NerdDinnerDataSet.DinnerRow> FindByLocation(float latitude, float longitude)
{
var now = DateTime.Now;
var dinnerTableAdapter = new DinnerTableAdapter();
var rsvpTableAdapter = new RSVPTableAdapter();

dinnerTableAdapter.FillByLocation(nerdDinnerDataSet.Dinner, latitude, longitude, now);
rsvpTableAdapter.FillByLocation(nerdDinnerDataSet.RSVP, latitude, longitude, now);
return this.nerdDinnerDataSet.Dinner;
}

The TableAdapters were created as part of the DataSetDesigner. Here' a screenshot from VS2010:

DataSet Designer

The Adapters fill the DataTables that consist of Rows. This unfortunately leaks out of our Repository into our Controller as our "Model" is now a DinnerRow. That then leaks (inappropriately) into a ViewPage of type...wait for it...System.Web.Mvc.ViewPage<NerdDinner.Models.NerdDinnerDataSet.DinnerRow>.

If you're going to use DataSets or Rows or DataTables, it's just that much more important that you use a good ViewModel projection. I personally go out of my way to not use DataSets and bump into them only in legacy code. Try to avoid them - I'd prefer the DataReader example over this one.

LINQ to SQL

Remember that LINQ to SQL is a one to one mapping between the physical tables and columns of the database and the objects it creates. Many folks prefer to use it as a DAL (Data Access Layer) that just happens to make objects, then pull the data out of the auto-generated objects into smarter business objects, such that the developer downstream never sees the generated L2S objects. Others just use them all through. For simple samples, I used to use LINQ to SQL straight, and I still do for small (< 5 page) projects, but lately I've been using EF4 as it's just as easy. Anyway, here's the now more sensible modified interface:

public interface IDinnerRepository {
IQueryable<Dinner> FindAllDinners();
IQueryable<Dinner> FindByLocation(float latitude, float longitude);
IQueryable<Dinner> FindUpcomingDinners();
Dinner GetDinner(int id);

void Add(Dinner dinner);
void Delete(Dinner dinner);

void Save();
}

And then the FindByLocation implementation:

public IQueryable<Dinner> FindByLocation(float latitude, float longitude) {
var dinners = from dinner in FindUpcomingDinners()
join i in db.NearestDinners(latitude, longitude)
on dinner.DinnerID equals i.DinnerID
select dinner;

return dinners;
}

image Notice the "NearestDinners" method there. That's a clever thing, I think. The database has a scalar-valued function called DistanceBetween for calculating the distance between two lat-longs (thanks Rob Conery!) and a table value function called NearestDinners. They look like functions from LINQ to SQL's point of view and can be included in a LINQ to SQL query as seen above.

image 

Nice and clean.

LINQ to Entities (Entity Framework)

Entity Framework 4 uses the same interface as above and the same FindByLocation:

public IQueryable<Dinner> FindByLocation(float latitude, float longitude) {
var dinners = from dinner in FindUpcomingDinners()
join i in NearestDinners(latitude, longitude)
on dinner.DinnerID equals i.DinnerID
select dinner;

return dinners;
}

The one small difference, you may notice, is that NearestDinners isn't hanging off the "db" object (the DataContext) as it was with LINQ to SQL. Instead, in order to maintain the same clean query structure, those are helper methods. One is an EdmFunction whose signature maps to that scalar function, and NearestDinner is implemented in code directly.

[EdmFunction("NerdDinnerModel.Store", "DistanceBetween")]
public static double DistanceBetween(double lat1, double long1, double lat2, double long2)
{
throw new NotImplementedException("Only call through LINQ expression");
}

public IQueryable<Dinner> NearestDinners(double latitude, double longitude)
{
return from d in db.Dinners
where DistanceBetween(latitude, longitude, d.Latitude, d.Longitude) < 100
select d;
}

Don't worry about that NotImplementedException, when the method is used in a LINQ to Entities Expression it's automatically mapped to the DistanceBetween function in the database as in the attribute.

I'd like to see better support for TVFs in EF, and I need to dig in to see if there's a better way that outlined here. Entity Framework also supports multiple databases, so you can get an Oracle Provider or a MySQL provider, etc.

So there you have four different database implementations for NerdDinner. Last, but not least, is a sample that Ayende wrote for me to teach me NHibernate, and I would be remiss to not include it in such a comparison.

NHibernate

This sample was written least year with ASP.NET MVC 2 on VS2008 using NHibernate 2.1. I'd love to see an updated version using even newer techniques.

Hibernate has the concept of a "Session" that lives for the life of a request in a Web Application. There's a config file (or a fluent configuration) that has all the properties and connection strings, (similar to EDMX files in EF or DBMLs in L2S) and this all gets setup in the Global.asax. The session is created in the BeginRequest and disposed in the EndRequest.

public MvcApplication()
{
BeginRequest += (sender, args) => CurrentSession = SessionFactory.OpenSession();
EndRequest += (sender, args) => CurrentSession.Dispose();
}

Here's the FindByLocation method. This example isn't 100% fair as this version of NHibernate doesn't support those custom functions I've been talking about. I'm going to see if the latest does and update this post. However, this does give you insight into its flexibility as it allowed inline SQL, set two parameters and returned a list of ints in a very tight single line of code.

public IQueryable<Dinner> FindByLocation(float latitude, float longitude)
{
// note that this isn't as nice as it can be, since linq for nhibernate
// doesn't support custom SQL functions right now

var matching = session.CreateSQLQuery("select DinnerID from dbo.NearestDinners(:latitude, :longitude)")
.SetParameter("longitude", longitude)
.SetParameter("latitude", latitude)
.List<int>();

return from dinner in FindUpcomingDinners()
where matching.Any(x => x == dinner.DinnerID)
select dinner;

}

A better example that lets NHibernate shine would be something more typical like:

public IQueryable<Dinner> FindUpcomingDinners()
{
return from dinner in FindAllDinners()
where dinner.EventDate >= DateTime.Now
orderby dinner.EventDate
select dinner;
}

You'll note that LINQ to NHibernate is nice and comfortable and looks just like you'd expect it to.

Just like EF and LINQ to SQL, there's a mapping file that explains how tables and columns and dataTypes map to real objects, although there isn't a visual editor as far as know. I believe there are fluent ways to express this in code, so if you're an NHibernate user, let me know alternative ways to express this and I'll update the post.

<?xml version="1.0" encoding="utf-8" ?>
<hibernate-mapping xmlns="urn:nhibernate-mapping-2.2" assembly="NerdDinner" namespace="NerdDinner.Models">
<class name="Dinner" table="Dinners" lazy="false">
<id name="DinnerID">
<generator class="identity"/>
</id>
<property name="Title"/>
<property name="EventDate"/>
<property name="Description"/>
<property name="HostedBy"/>
<property name="ContactPhone"/>
<property name="Address"/>
<property name="Country"/>
<property name="Latitude"/>
<property name="Longitude"/>
<bag name="RSVPs" cascade="all-delete-orphan" inverse="true">
<key column="DinnerID"/>
<one-to-many class="RSVP"/>
</bag>

</class>
<class name="RSVP" table="RSVP" lazy="false">
<id name="RsvpID">
<generator class="identity"/>
</id>
<property name="AttendeeName"/>
<many-to-one name="Dinner"
column="DinnerID"/>
</class>
</hibernate-mapping>

This mapping file is actually marked as an Embedded Resource, and is accessed at runtime by the NHibernate runtime. You just need to make it, and the magic happens for you. NHibernate's claim to fame is support for lots of different databases like SQL Server, Oracle, MySQL and more. There's also lots of supporting projects and libraries that orbit NHibernate to give you additional control, or different ways to express your intent.

Conclusion

There's lots of choices for Database Access on .NET. You'll run into DataReaders in older or highly tuned code, but there's no reason it can't be hidden in a Repository and still be pleasant to use. LINQ to SQL is nice, lightweight and fast and has dozens of bug fixes in .NET 4, but Entity Framework is the way they are heading going forward. Plus, Entity Framework 4 is *way* better than EF 3.5, so I'm using it for any "larger than small" projects I'm doing and I'm not having much trouble. NHibernate is very mature, actively developed and has a great community around it and it's not going anywhere.

In my opinion, if you're doing database access with .NET you should be using Entity Framework 4 or NHibernate.

Four Database-styles Sample

nddatafoursamples

NHibernate Sample

Related Links

About Scott

Scott Hanselman is a former professor, former Chief Architect in finance, now speaker, consultant, father, diabetic, and Microsoft employee. He is a failed stand-up comic, a cornrower, and a book author.

facebook twitter subscribe
About   Newsletter
Sponsored By
Hosting By
Dedicated Windows Server Hosting by SherWeb

Extending NerdDinner: Adding MEF and plugins to ASP.NET MVC

May 20, '10 Comments [16] Posted in ASP.NET | ASP.NET MVC | NerdDinner | Open Source | Source Code
Sponsored By

The original NerdDinner Sample was very simple. Two samples, simple, in fact. Perhaps it's the new Northwind, as it's a good way to start working with ASP.NET MVC. However, it's not a perfect sample or an idealized example on how to do many things that users want to do.

Fortunately, there's been lots of cool folks in the community who have "forked" NerdDinner and done interesting stuff with it. Each of these samples is usually focused on a specific scenario, so they won't necessarily be merged with the trunk, but they are educational nonetheless.

Jon Galloway and I have also added a few things to NerdDinner, taking it in a more social direction, as Jon's MVC Music Store today is a better "getting started" sample for ASP.NET MVC 2. We'll be doing a series of posts on the interesting things the community has added to NerdDinner as well as some of the ones Jon and I added and presented at Mix a few months back. Soon Jon and I will release an updated NerdDinner v2 on CodePlex (although it's been in the source code tab for weeks) with lots of fixes, new features. We'll also add many of these "one off" samples as well and host them on CodePlex.

I spoke to Microsoft Engineer Hamilton Verissimo de Oliveira, aka "Hammett" (you likely know him from the Castle Project and Monorail) about making a NerdDinner sample that included MEF (Managed Extensibility Framework) since much of MEF is built into .NET 4 now. He was kind enough to do it, but I'm just blogging it now, so thanks to Hammett for his kindness and patience.

NerdDinner on MEF

MEF lives in System.ComponentModel.Composition. Hammett's done a number of interesting things it his sample, adding Microsoft.ComponentModel.Composition.Extensions and Microsoft.ComponentModel.Composition.Extensions.Web namespaces building in some nice extension methods for common techniques as well as and implementation of IControllerFactory and a derivation of HttpApplication.

MefControllerFactory

Remember MEF is about making applications easily composable. In this sample Hammett has created his own MefControllerFactory, replacing the default controller factory that comes with ASP.NET MVC. ASP.NET MVC makes it easy to change:

protected override void Application_Start()
{
base.Application_Start();

ControllerBuilder.Current.SetControllerFactory(new MefControllerFactory(base.ScopeManager));

RegisterRoutes(RouteTable.Routes);

ViewEngines.Engines.Clear();
ViewEngines.Engines.Add(new MobileCapableWebFormViewEngine());
}

Notice his controller factory has a ScopeManager. This is a web application, and some components might be at Application Scope (create them once and hang on) and some might be at Request scope (make a new one each request).

For controllers, he's effectively recreated the default behavior of the ASP.NET MVC's controller factory, but in doing it, has given us lots of ways we can jump in an change the behavior in exotic ways by overriding CreateRootCatalog in MefhttpApplication. It's default implementation looks in /bin:

protected virtual ComposablePartCatalog CreateRootCatalog()
{
return new DirectoryCatalog(Path.Combine(AppDomain.CurrentDomain.BaseDirectory, "bin"));
}

As you probably know, ASP.NET MVC looks for classes via a convention. It looks for classes with the word "Controller" at the end that are also IController implmentations. Here's the MEF way to declare that convention using Microsoft.ComponentModel.Composition.Extensions. Note the use of scope.

[assembly: Discovery(typeof(Conventions))]

namespace NerdDinner
{
public class Conventions : ConventionDiscovery
{
public Conventions()
{
Add(PartRegistration.
Types(t => t.Name.EndsWith("Controller") && !t.IsAbstract).
Exporting((c, t) => c.
Contract(typeof (IController)).
Metadata("Name", t.Name.Substring(0, t.Name.Length - "controller".Length)).
Metadata("Mode", WebScopeMode.Request))
);
}
}
}

Pretty cool.

Controllers and their Needs

Whenever a more advanced programmer looks as the NerdDinner code they usually say they they really don't like this:

public DinnersController() : this(new DinnerRepository()) 
{
}

public DinnersController(IDinnerRepository repository)
{
dinnerRepository = repository;
}

The second constructor takes an IDinnerRepository, allowing us to make different implementations, but the default constructor says, "well, here's a concrete implementation if you don't give one." It's a slippery slope and by adding the default implementation I get to sidestep using dependency injection while making the controller testable, but I've tied my controller down with a direct dependency to the DinnerRepository. This is sometimes called "Poor Man's IoC" and many would say that this is a very poor man. That's a religious argument, but Hammett takes a stand by removing the default constructor.

public class DinnersController : Controller
{
private IDinnerRepository dinnerRepository;

public DinnersController(IDinnerRepository repository)
{
dinnerRepository = repository;
}
//...
}

So how does a DinnersController ever get an IDinnerRepository? The idea is that it's not the controllers job to worry about the how, it's only its job to want.

MEF is effectively a Dating Service for Components. Here DinnerRepository is saying it's available and it wants to meet someone who is also into "IDinnerRepository."

[Export(typeof(IDinnerRepository))]
public class DinnerRepository : NerdDinner.Models.IDinnerRepository {

That [Export] attribute is its way to saying, "I'm on the market. Matchmaker, make a match!" When MEF is asked for a Controller and it notices that it has no default constructor, as in our case, it looks at the available constructors and says, "Oh, DinnersController wants to meet someone too! I I think I know just your type." Then it creates a DinnerRepository and calls the DinnersController constructor passing it in. It injects the dependency.

Often you'll see the other components advertising their interest with an [Import] attribute, but that's not necessary in this example because all the Controllers were created via the MefControllerFactory. They don't need attributes, as they've already walked in the door of our dating service!

Other Services MEFified

Recently in a review of MVC Music Store Ayende (and others before him, as well) dissed on these line of code, which actually come with ASP.NET MVC 2 out of the box and weren't written for the sample. (Although they weren't changed)  Phil can speak to specific decisions as I wasn't involved, but many folks who are into dependency injection don't like this. This is effectively the same maneuver as shown above, written slightly differently.

public class AccountController : Controller 
{
public AccountController()
: this(null, null) {
}

public AccountController(IFormsAuthentication formsAuth, IMembershipService service)
{
FormsAuth = formsAuth ?? new FormsAuthenticationService();
MembershipService = service ?? new AccountMembershipService();
}
//...
}

Hammett's implementation just uses MEF, so:

public AccountController(IFormsAuthentication formsAuth, IMembershipService service)
{
FormsAuth = formsAuth;
MembershipService = service;
}

Again, the whole point is that the dependencies get figured out automatically, each one wearing the "I'm available for hooks ups" attributes of [Export(typeof(IFormsAuthentication))] and [Export(typeof(IMembershipService))] respectively.

All in all, MEF is a nice clean addition to an ASP.NET MVC app. Thanks to Hammett for his hard work!

Related Links

About Scott

Scott Hanselman is a former professor, former Chief Architect in finance, now speaker, consultant, father, diabetic, and Microsoft employee. He is a failed stand-up comic, a cornrower, and a book author.

facebook twitter subscribe
About   Newsletter
Sponsored By
Hosting By
Dedicated Windows Server Hosting by SherWeb

Creating an OData API for StackOverflow including XML and JSON in 30 minutes

March 28, '10 Comments [65] Posted in ASP.NET | OData | Open Source | Source Code
Sponsored By

I emailed Jeff Atwood last night a one line email. "You should make a StackOverflow API using OData." Then I realized that, as Linus says, Talk is Cheap, Show me the Code. So I created an initial prototype of a StackOverflow API using OData on an Airplane. I allocated the whole 12 hour flight. Unfortunately it took 30 minutes so I watched movies the rest of the time.

You can follow along and do this yourself if you like.

Preparation

Before I left for my flight, I downloaded two things.

First, I got Sam Saffron's "So Slow" StackOverflow SQL Server Importer. This is a little spike of Sam's that takes the 3gigs of XML Dump Files from StackOverflow's monthly dump and imports it into SQL Server.

Second, I got the StackOverflow Monthly Dump. I downloaded it with uTorrent and unzipped it in preparation for the flight.

Importing into SQL Server

I went into Visual Studio 2010 (although I could have used 2008, I like the Entity Framework improvements in 2010 enough that it made this job easier). I right clicked on the Data Connections node in the Server Explorer and created a database in SQL Express called, ahem, "StackOverflow."

 Create New SQL Server Database

Next, I opened up Sam's RecreateDB.sql file from his project in Visual Studio (I avoid using SQL Server Management Studio when I can) and connected to the ".\SQLEXPRESS" instance, selected the new StackOverflow database and hit "execute."

Recreate DB SQL inside of Visual Studio

One nit about Sam's SQL file, it creates tables that line up nicely with the dump, but it includes no referential integrity. The tables don't know about each other and there's no cardinality setup. I've overwritten the brain cells in my head that know how to do that stuff without Google Bing so I figured I'd deal with it later. You will too.

Next, I opened Sam's SoSlow application and ran it. Lovely little app that works as advertised with a gloriously intuitive user interface. I probably would have named the "Import" button something like "Release the Hounds!" but that's just me.

So Slow ... Stack Overflow database importer

At this point I have a lovely database of a few hundred megs filled with StackOverflow's public data.

image

Making a Web Project and an Entity Model

Now, from within Visual Studio I selected File | New Project | ASP.NET Web Application. Then I right clicked on the resulting project and selected Add | New Item, then clicked Data, then ADO.NET Entity Data Model.

Add New Item - StackOveflow

What's the deal with that, Hanselman? You know StackOverflow uses LINQ to SQL? Have you finally sold out and are trying to force Entity Framework on us sneakily within this cleverly disguised blog post?

No. I used EF for a few reasons. One, it's fast enough (both at runtime and at design time) in Visual Studio 2010 that I don't notice the difference anymore. Two, I knew that the lack of formal referential integrity was going to be a problem (remember I mentioned that earlier?) and since LINQ to SQL is 1:1 physical/logical and EF offers flexible mapping, I figured it be easier with EF. Thirdly, "WCF Data Services" (the data services formerly known as ADO.NET Data Services or "Astoria") maps nicely to EF.

I named it StackOverflowEntities.edmx and selected "Update Model from Database" and selected all the tables just to get started. When the designer opened, I noticed there were no reference lines, just tables in islands by themselves.

The Initial Entity Model

So I was right about there being no relationships between the tables in SQL Server. If I was a smarter person, I'd have hooked up the SQL to include these relationships, but I figured I could add them here as well as a few other things that would make our OData Service more pleasant to use.

I started by looking at Posts and thinking that if I was looking at a Post in this API, I'd want to see Comments. So, I right-clicked on a Post and click Add | Association. The dialog took me a second to understand (I'd never seen it before) be then I realized that it was creating an English sentence at the bottom, so I just focused on getting that sentence correct.

In this case, "Post can have * (Many) instances of Comment. Use Post.Comments to access the Comment instances. Comment can have 1 (One) instance of Post. Use Comment.Post to access the Post instance." was exactly what I wanted. I also already had the foreign keys properties, so I unchecked that and clicked OK.

Add Association 

That got me here in the Designer. Note the line with the 1...* and the Comments Navigation Property on Post and the Post Navigation Property on Comment. That all came from that dialog.

Posts relate to Comments

Next, I figured since I didn't have it auto-generate the foreign key properties, I'd need to map them myself. I double clicked on the Association Line. I selected Post as the Principal and mapped its Id to the PostId property in Comments.

Referential Constraint

Having figured this out, I just did the same thing a bunch more times for the obvious stuff, as seen in this diagram where Users have Badges, and Posts have Votes, etc.

A more complete StackOverflow Entity Model with associations completed

Now, let's make a service.

Creating an OData Service

Right-click on the Project in Solution Explorer and select Add | New Item | Web | WCF Data Service. I named mine Service.svc. All you technically need to do to have a full, working OData service is add a class in between the angle brackets (DataService<YourTypeHere>) and include one line for config.EntitySetAccessRule. Here's my initial minimal class. I added the SetEntitySetPageSize after I tried to get all the posts. ;)

public class Service : DataService<StackOverflowEntities>
{
// This method is called only once to initialize service-wide policies.
public static void InitializeService(DataServiceConfiguration config)
{
config.SetEntitySetAccessRule("*", EntitySetRights.AllRead);

//Set a reasonable paging site
config.SetEntitySetPageSize("*", 25);

config.DataServiceBehavior.MaxProtocolVersion = DataServiceProtocolVersion.V2;
}
}

Expanding on this class, I added caching, and an example Service Operation, as well as WCF Data Services support for JSONP. Note that the Service Operation is just an example there to show StackOverflow that they CAN have total control. Using OData doesn't mean checking a box and putting your database on the web. It means exposing specific entities with as much or as little granularity as you like. You can intercept queries, make custom behaviors (like the JSONP one), make custom Service Operations (they can include query strings, of course), and much more. OData supports JSON natively and will return JSON when an accept: header is set, but I added the JSONP support to allow cross-domain use of the service as well as allow the format parameter in the URL, which is preferred by man as it's just easier.

namespace StackOveflow
{
[JSONPSupportBehavior]
public class Service : DataService<StackOverflowEntities>
{
// This method is called only once to initialize service-wide policies.
public static void InitializeService(DataServiceConfiguration config)
{
config.SetEntitySetAccessRule("*", EntitySetRights.AllRead);

//This could be "*" and could also be ReadSingle, etc, etc.
config.SetServiceOperationAccessRule("GetPopularPosts", ServiceOperationRights.AllRead);

//Set a reasonable paging site
config.SetEntitySetPageSize("*", 25);

config.DataServiceBehavior.MaxProtocolVersion = DataServiceProtocolVersion.V2;
}

protected override void OnStartProcessingRequest(ProcessRequestArgs args)
{
base.OnStartProcessingRequest(args);
//Cache for a minute based on querystring
HttpContext context = HttpContext.Current;
HttpCachePolicy c = HttpContext.Current.Response.Cache;
c.SetCacheability(HttpCacheability.ServerAndPrivate);
c.SetExpires(HttpContext.Current.Timestamp.AddSeconds(60));
c.VaryByHeaders["Accept"] = true;
c.VaryByHeaders["Accept-Charset"] = true;
c.VaryByHeaders["Accept-Encoding"] = true;
c.VaryByParams["*"] = true;
}

[WebGet]
public IQueryable<Post> GetPopularPosts()
{
var popularPosts = (from p in this.CurrentDataSource.Posts
orderby p.ViewCount
select p).Take(20);

return popularPosts;
}
}
}

But what does this get us? So what?

Accessing StackOverflow's Data via OData

Well, if I hit http://mysite/service.svc I see this service. Note the relative HREFs.

Screenshot of an XML document describing an OData service endpoint

If I hit http://173.46.159.103/service.svc/Posts I get the posts (paged, as I mentioned). Look real close in there. Notice the <link> stuff before the content? Notice the relative href="Posts(23)"?

StackOverflow Posts in OData

Remember all those associations I set up before? Now I can see:

But that's just navigation. I can also do queries. Go download LINQPad Beta for .NET 4. Peep this. Click on Add Connection, and put in my little Orcsweb test server.

Disclaimer: This is a test server that Orcsweb may yank at any moment. Note also, that you can sign up for your own at http://www.vs2010host.com or find a host at ASP.NET or host your own OData in the cloud.

I put this in and hit OK.

LINQPad Connection String

Now I'm writing LINQ queries against StackOverflow over the web. No Twitter-style API, JSON or otherwise can do this. StackOverflow data was meant for OData. The more I mess around with this, the more I realize it's true.

LINQPad 4

This LINQ query actually turns into this URL. Again, you don't need .NET for this, it's just HTTP:

',Tags)">',Tags)">http://173.46.159.103/service.svc/Posts()?$filter=substringof('SQL',Title) or substringof('<sql-server>',Tags)

Try the same thing with an accept header of accept: application/json or just add $format=json

',Tags)&$format=json">',Tags)&$format=json">http://173.46.159.103/service.svc/Posts()?$filter=substringof('SQL',Title) or substringof('<sql-server>',Tags)&$format=json

It'll automatically return the same data as JSON or Atom, as you like.

If you've got Visual Studio, just go bust out a Console App real quick. File | New Console App, then right-click in references and hit Add Service Reference. Put in http://173.46.159.103/service.svc and hit OK.

Add Service Reference

Try something like this. I put the URIs in comments to show you there's no trickery.

class Program
{
static void Main(string[] args)
{
StackOverflowEntities so = new StackOverflowEntities(new Uri("http://173.46.159.103/service.svc"));

//{http://173.46.159.103/service.svc/Users()?$filter=substringof('Hanselman',DisplayName)}
var user = from u in so.Users
where u.DisplayName.Contains("Hanselman")
select u;

//{http://173.46.159.103/service.svc/Posts()?$filter=OwnerUserId eq 209}
var posts =
from p in so.Posts
where p.OwnerUserId == user.Single().Id
select p;

foreach (Post p in posts)
{
Console.WriteLine(p.Body);
}

Console.ReadLine();
}
}

I could keep going with examples in PHP, JavaScript, etc, but you get the point.

Conclusion

StackOverflow has always been incredibly open and generous with their data. I propose that an OData endpint would give us much more flexible access to their data than a custom XML and/or JSON API that they'll need be constantly rev'ing.

With a proprietary API, folks will rush to create StackOverflow clients in many languages, but that work is already done with OData including libraries for iPhone, PHP and Java. There's a growing list of OData SDKs that could all be used to talk to a service like this. I could load it into Excel using PowerPivot if I like as well.

Also, this service could totally be extended beyond this simple GET example. You can do complete CRUD with OData and it's not tied to .NET in anyway. TweetDeck for StackOverflow perhaps?

I propose we encourage StackOverflow to put more than the 30 minutes that I have put into it and make a proper OData service for their data, rather than a custom API. I volunteer to help. If not, we can do it ourselves with their dump data (perhaps weekly if they can step it up?) and a cloud instance.

Thoughts?

About Scott

Scott Hanselman is a former professor, former Chief Architect in finance, now speaker, consultant, father, diabetic, and Microsoft employee. He is a failed stand-up comic, a cornrower, and a book author.

facebook twitter subscribe
About   Newsletter
Sponsored By
Hosting By
Dedicated Windows Server Hosting by SherWeb
Previous Page Page 3 of 17 in the Source Code category Next Page

Disclaimer: The opinions expressed herein are my own personal opinions and do not represent my employer's view in any way.