Scott Hanselman

RFC: How FeedReaders and MacGyver report blog subscribers - Tunneled User-Agent Data

February 17, '07 Comments [3] Posted in ASP.NET
Sponsored By

Sometimes I get ever so slightly depressed that the Web is so fantastically hacked together. The way we revel in AJAX sites but forget how dizzyingly high up we are, floating in layer after layer of abstraction. IP, TCP, HTTP, UTF8/ASCII Text Encoding, HTML, XML, XHTML, CSS, ECMAScript, DOM, the list goes on...there's a lot of moving parts. I wonder how the next generation will learning all the plumbing?

They can happily drag a button from the Toolbox onto their Form and Start Programming™. I think this means I'm officially old and crusty because I'm finding myself, internally, thinking "these young punks with their Ajax and their MacGyver techniques! Assembling websites with CSS Box Hacks and Paper Clips! Feh!"

At any rate, Google Reader, an online feed aggregator whose interface I'm still slightly not digging, is now reporting their subscribers.

There's different classes of Feed Readers/Aggregators that can retrieve content two ways. There's, of course, desktop and web readers who can retrieve content directly or centrally. (These are my four classifications.)

RSS Bandit and SharpReader and NetNewsWire are actual applications that you install and run locally. They reach out from your computer directly to the feed and download it directly. FeedDemon can do this too, but is a kind of hybrid, in that if you have a NewsGator subscription it's actually getting the feed content from NewsGator, not the publisher, so in that "hybrid" (my word) mode, FeedDemon looks like an online reader.

Here's a very incomplete, but you'll-get-the-idea-it-is-just-trying-to-make-a-point table:

Reader Desktop/Web Direct/Centralized
Google Reader Web Centralized
Bloglines Web Centralized
FeedDemon Desktop Direct
(can talk to NewsGator also)
NewsGator
Online
Web Centralized
SharpReader Desktop Direct
RSS Bandit Desktop Direct
(can talk to NewsGator also)
IE7 (RSS Platform) Desktop Direct
(but shared and
centralized to the OS)

FeedBurner hosts my Feed for this site, and they have a wonderful Feed-specific Special Sauce that figures out, approximately, how many folks are subscribing/reading my site. They use lots of metrics like IP address and what not to figure out Desktop readers, and they have some algorithms to recognize that IPs change and what not.

What's interesting is how these Web and/or Centralized readers reports statistics. When a bot for one of these readers retrieves your feed, they include (or tunnel ala MacGyver), the number of subscribers in their database within the User-Agent like this. (I talked about this some two years ago):

Doesn't seem ever so slightly distasteful to you? This data is totally non-standard, and living in the HTTP Headers for User-Agent. Oleg thinks it's OK per the HTTP spec, but I say bleh.

Why hunt for older headers to stuff this data into? Four years ago Tim Bray brainstormed some ideas and thought about the URL itself, then the referrer: header, or the from: header. Why not a new one?

HTTP Headers themselves are name/value pairs, fairly well structured, but it seems that rather than forcing FeedBurner to keep tables of the various formats of the various readers and make them Regular Expression their way thought it...

Old Joke but still a Good One: So you've got a problem, and you've decided to use Regular Expressions to solve it....so, now you've got two problems...

...why not add a new HTTP Header? I mean, the blogosphere has long abandoned many of the slower standards bodies in favor of a de facto standard-building process. If enough people do it, it's standard.

RFC: Why don't the bots and online aggregators start requesting feeds like this:

GET / HTTP/1.1
Host: www.hanselman.com
User-Agent: MyFeedReadingBot
Feed-Subscriber-Count: 45

It worked for SOAP Action back in the day, why not standardize a new header now? Let's let MacGyver rest.

About Scott

Scott Hanselman is a former professor, former Chief Architect in finance, now speaker, consultant, father, diabetic, and Microsoft employee. He is a failed stand-up comic, a cornrower, and a book author.

facebook twitter subscribe
About   Newsletter
Sponsored By
Hosting By
Dedicated Windows Server Hosting by SherWeb

Adding a Timeline to DasBlog with the SIMILE Timeline Library

February 17, '07 Comments [8] Posted in ASP.NET | DasBlog | Javascript
Sponsored By

I've been exploring recently the amazing Timeline Javascript library from the SIMILE folks over at MIT.

What a joy to work with! I've not historically been a fan, by any means, of JavaScript. Of course, debugging it can be a hassle and the development process always feels very unstructured to me. Heh, probably a side-effect of the language, right?

Anyway, the API for the SIMILE Timeline is very well thought-out. It's very JavaScripty, to be clear, but it is even easier to pick up than the Google Maps API.

So easy, in fact, that I was able to add Timeline support to DasBlog over my lunch hour. Check it out at http://www.hanselman.com/blog/timeline.aspx. Try typing in the Highlight boxes, as shown in the screenshot above. Notice the highlighting in both bands of the timeline. I've checked the server-side TimelineHandler that creates the XML that Timelines consume into the head of the DasBlog tree.

Also, take a look at their "Exhibit" project. Maybe Web 2.0 will turn out something useful after all! :)

About Scott

Scott Hanselman is a former professor, former Chief Architect in finance, now speaker, consultant, father, diabetic, and Microsoft employee. He is a failed stand-up comic, a cornrower, and a book author.

facebook twitter subscribe
About   Newsletter
Sponsored By
Hosting By
Dedicated Windows Server Hosting by SherWeb

Clean up your Temp Files

February 16, '07 Comments [6] Posted in ASP.NET | Tools
Sponsored By

I was doing some tidying up on a website for a family friend recently. His ASP.NET application was taking longer and longer to start up. It also happened to use a lot of XmlSerializers, and of course, as an ASP.NET app, there's a lot of page-by-page compilation as he's not on 2.0 and doesn't use aspnet_compiler.

Additionally, there's some code that this client wrote that writes out temp files but doesn't clean up after themselves. When there's this many files in a folder, calls to GetTempFileName can block for many seconds.

Not only can "excessive temp file growth" cause pain at runtime, but it'll easily crush Explorer.exe's usefulness. It'll also become impossible to apply Attributes to the folder.

The point is, that there's a few folders that you need to watch out for file growth in.

I like to modify the options inside Microsoft Drive Cleanup using Shawn A. Van Ness's registry file. You can extensively modify what appears in the Drive Cleanup list, even adding your own file types and your own application-specific directories and files that you might want cleaned up.

Check out your own systems...drop out to a command prompt (cmd.exe) and do:

  • cd %tmp% and cd %temp% - You'll usually end up in C:\DOCUME~1\username\LOCALS~1\Temp.
    • At this point, I like to do the equivalent of a deltree and go up a directory and:
      • cd ..
      • rd Temp /s (it usually won't manage to delete the whole dir. Someone will have a file open and the final directory deletion will fail)
      • md Temp (in case it was deleted.)
  • Everything in %windir%\temp - There's lots of Perfmon counters in here, so you won't be able to delete everything. Often you can del *.* and anything that shouldn't be deleted is currently open.
  • If you are a developer, and have developed ASP.NET for years/months, clean up %windir%\Microsoft.NET\Framework\ <VERSION> \Temporary ASP.NET Files. I had 4 dozens sites in here.

Tidy up, my friends.

About Scott

Scott Hanselman is a former professor, former Chief Architect in finance, now speaker, consultant, father, diabetic, and Microsoft employee. He is a failed stand-up comic, a cornrower, and a book author.

facebook twitter subscribe
About   Newsletter
Sponsored By
Hosting By
Dedicated Windows Server Hosting by SherWeb

Xbox 360 Controller Wireless Receiver Adapter for Windows

February 16, '07 Comments [14] Posted in Gaming | Reviews
Sponsored By

(Note, I'm intentionally being a little dense here to make a point...)

BTW: If you want to skip to the point, get the updated 1.1 software here: Xbox 360 Accessories Software 1.1 32-bit for Windows Vista and bypass the excessive clicking.

I've got an Xbox 360, and I've got a PC and I've got four (4) Xbox 360 wireless controllers, so it was a natural thing when I saw the "Xbox 360 Wireless Receiver Adapter for Windows" that I'd take a look. Seems like a brilliant idea, right? A small USB dongle that lets you use your Xbox 360 Wireless Accessories on your PC. Brilliant. One stop closer to the "Games for Windows" (i.e. Vista as Game Console) dream, right?

Fabulous...on with the review.

"Setup cannot install this software because your computer does not meet the system requirements." 

Um. Yikes. OK. I got this dialog after running Setup.exe. Didn't make it very far, this is the first thing I saw. My Experience Index in Vista Ultimate is 4.2, and my graphics index is 5.6. I guess my Pentium 4 4Ghz with 2gigs of RAM and a 256M video card isn't up to the task.

I guess I'll click "Check Online" and get a new version of the software. Click.

Um...ok. The Downloads Page of microsoft.com/hardware. I have Vista...what is this page telling me Or is it asking me something? Ok, I'll click on "Game controllers..."

Ok...I figure this qualifies as game controller. Hm...as I fill this four-step page out, I'm thinking to myself...

  • I still don't know WHY my high-powered system doesn't "qualify."
  • I'm not sure why the setup program didn't just download the latest version.
  • If it couldn't download the software, why not just take me to the right page?
  • Can't they detect my operating system via my IE User Agent and figure that I'm running Vista?

Moral: Don't force your users to click and answer questions when the answer is already available in "context" for the taking. You know I have Vista. My browser said so. You know I speak English. My browser said so. You know I'm trying to install this Gaming thingie. You could have just launched a browser and included this info in the URL/QueryString.

Ok, now I click "go." Hm. Another link. I guess I click on that.

All righty...now we're getting somewhere. Download...Run...gosh this seems like a lot of work.

Make it easy to update your software, and if you ship a setup.exe, teach it to "phone home" if there's a crucial update. Considering that the 1.0 disc I have is totally useless, I can assume that someone realized at the last minute that there's a bunch of Vista machines that were going to get this error message. If the 1.0 setup was built to check for newer ones, this whole thing could have been seamless.

You know what the ironic part is? Take a look at the Program Group in Explorer after everything is installed. There's a Check For Updates icon that points to checker.exe. Why isn't this updating process integrated into Windows Update?

Alright then. I'll run Microsoft Xbox 360 Accessories Status now. Cool, I just click the button on the Receiver and the button on the Xbox 360 Controller and they are sync'ed. Slick.

Looks like the Controller just shows up like it's a joystick in Windows, which is it. I like standard.

Well, this is a great dialog, but there's no support for remapping. I tried to get the Xbox 360 Controller to work in Half-Life 2, but unless I choose to learn the obscure HL2 language for this kind of thing, it makes the controller less useful.

All the "Games for Windows" games must meet a certain level of compatibility, but that doesn't help me for older games. The newer ones must support this controller (and widescreen monitors!) along with 64-bit Vista. That's certainly something to look forward to. There's just not a lot of Games out yet (and the Games site's "Game Advisor" doesn't support FireFox!).

Hacking It

There IS, however, a tool called the Pinnacle Game Profiler that says it'll add support for a Joystick to any game through a series of remapping profiles. Unfortunately, not only does it have trouble on Vista, but it caused Vista to display this astonishingly scary pair of dialogs that I'd never seen before:

This was of course, far too dangerous to risk on my new Vista machine, as things are humming along nicely so far. Who am I to install some crazy remapping software that's not Vista savvy. Maybe one day soon they'll fix it. I hear good things about this remapper.

Until then, my Xbox 360 Controller Wireless Receiver Adapter for Windows is unused, but I have high hopes.

About Scott

Scott Hanselman is a former professor, former Chief Architect in finance, now speaker, consultant, father, diabetic, and Microsoft employee. He is a failed stand-up comic, a cornrower, and a book author.

facebook twitter subscribe
About   Newsletter
Sponsored By
Hosting By
Dedicated Windows Server Hosting by SherWeb

Exiting The Zone of Pain - Static Analysis with NDepend

February 15, '07 Comments [12] Posted in Programming | Reviews | Tools
Sponsored By

There's a great discussion going on over at Jeff Atwood's blog about Coding Smaller, modularizing properly, and refactoring often. In the comments, Jae wishes that more apps were "modularized properly," and I wholly agree.

He/she says:

I think a lot of projects are dived into too willy-nilly (especially by neophyte/ novice/ hobbyist programmers) with no pre-planning of what the thing should do and how to logically code it. [Jae]

I'll use this comment, lifted from Jeff's post, to kick of discussion here. I think that pre-planning is part of it, but there's only so much Big Design Up Front (BDUF) you can do. More and more, as I work on larger and larger projects (and product suites) I realize that refactoring effectively and often is as or more valuable as pre-planning.

There's lots of create Refactoring Tools out there, certainly Resharper and Refactor Pro leap to mind, that can help you improve your code, and to some extent, your design, with their dozens of refactorings. Both also include some code analysis tools that illustrate hotspots in your files.

<confession>
At my company, sometimes I feel less like Chief Architect, and more like Chief Debugger or Chief Code Reader. Sometimes I get to caught up in trying to read code in order to understand the big picture. This is my own failing, as I often try to use a microscope when I need a telescope.
</confession>

While trying to improve design, manage dependencies and coupling within our product suite, I've been leaning more and more on static analysis tools. Some tools are pretty UI-free, like libcheck, a tool for managing the churn of the public "surface area" of a group (or framework) of assemblies.

There's lots of tools out there for analysis of code, for many languages, all with good science behind them. I remember doing some static analysis on C++ code back in the day, but until .NET's rich meta data, doing this kind of analysis was more a parsing exercise that created Excel spreadsheets of numbers - the data just wasn't very accessible. That, combined with the fact that a lot of .NET developers now never worked in a language before .NET that enabled real solid analysis (VBScript/ASP, VB, and I don't remember a lot of MFC/C++ folks doing analysis like this) means that perhaps a majority of .NET developers haven't been introduced to the power of tools like this.

I've been spending time with NDepend more and more lately. I used it a while back when it first came out, but version 2 is out and it's giving me insight into my applications that I haven't had before.

One of the problems with applications like NDepend and static analysis apps is that they are couched in the complex or "inaccessible" language of Computer Science. Many programmers haven't gone through a formal "CompSci" program and aren't interested in an analysis application with a steep learning curve.

Like PowerShell, the first 10 minutes of NDepend is the hardest. It has to click first. There's the whole "what the hell am I looking at" process, often followed by the "screw this" declaration, followed by a quick uninstallation and a sense of "what happened."

I'm not going to even attempt to try to improve on, or replace, the good documentation that's already up on the NDepend site. One of the things that NDepend's site has lots of is screencasts, do check them out.

What I'd like to do is try to introduce some of the basics of the kinds of analyses that are possible to the average programmer or the architect who hasn't considered this kind of analysis as an important part of the development process. Additionally, I know that many of you are using FxCop as a code analysis tool. I think that NDepend intersects FxCop in some ways, but presents the data in a totally different way, and allows you to drill in so much more deeply.

I can't use Corillian's software for my example here, so I'll use DasBlog, an ASP.NET blogging engine with which I have some association, instead. As seen in the picture below, you startup the NDepend Project application and either point it at your Visual Studio Solution, or just add your assemblies automatically. Then you save the NDepend Project as an XML file, and hit the "Play" button (the green arrow pointing to the right) and NDepend launches it's console app that chews for a while.

Next, you can either view an HTML Report - you can run the reports as often as you like and NDepend will keep a history of the reports. Using a static analysis tool can be confusing because it tells you SO much information all at once...trying to sip from the fire hose can be a challenge. 

Let's start by looking at the dependency diagram that was generated by the report. These are easy to read when there is a small number of assemblies. Those of you who use Reflector Add-Ins will recognize the library style as Peli's - in fact NDepend uses the same graphics library.

This graph is really the simplest thing that NDepend can tell you - it just scratches the surface of what this tool can do. We can learn a few important things that we may or may not have known (I didn't really grok at least) about our application. 

Let's explore, for this example, the Custom Macro facilities in DasBlog and if there are design problems.

Note the "FooMacroPlugin" - it's the sample plugin that shows how to write DasBlog Custom Macros. Note that it depends on two assemblies. This exposes very clearly a lameness in the DasBlog custom Macro model - largely my fault - in that it requires a reference to two different assemblies, but Web.Core needs Runtime anyway. We can also see later that it uses concrete classes rather than Interfaces, so it's, frankly, a plugin model with problems. If DasBlog had a formal Interface assembly with defined interfaces for interacting with the system we would see different linkage.

Now that we know that a custom macro links to to different assemblies, let's take a look at exactly how linked the Macro dll is to the rest of the system. We open up Visual NDepend and are barraged with a LOT of information. This is a crazy busy interface, but in that is a great deal of power. The interface is by and large modeless, and clicking in one area changes values in all the others.

In this screenshot I've clicked on the intersection in the Dependency Matrix of the FooMacroPlugin and the Web.Core assembly. Note in the lower left corner there's Plain English generated telling me "1 method of the assembly FooMacroPlugin" is using 1 member of Web.Core." 

Looking along the horizontal I can see the number 1, then 2, indicating that Foomacro is using 1 method of the Core and 2 within the RunTime.  (The axis can be opened up by clicking the + sign to see more text) Now I'm getting a better idea of how linked my Macros are with my other two Assemblies.

I can left-click on the number 2 to see what two methods I'm using in the Runtime. A "boxes and arrows" diagram is generated telling me that my Macros (there are two in FooMacroPlugin) are calling into a property on the an Entry (a blog Entry) and getting the count of an EntryCollection.

 

Now I'm starting to see what the impact would be if I were to refactor the Macro plugin mechanism in DasBlog, and I would repeat this analysis on a more complex Custom Macro, probably from the community. Even better, I could drop in EVERY custom Macro I could find to get a really 20,000 foot view - a wide perspective - on how Macros are being used...without reading the code.

NDepend really shines in it's use of CQL - Code Query Language - an NDepend invention, but public spec. Basically it's SQL for Reflection. I can write queries using this SQL-like syntax to ask questions about assemblies, types, methods, whatever within my solution. Here's some examples:

Detect Unused Methods:

WARN IF Count > 0 IN SELECT TOP 10 METHODS WHERE MethodCa == 0 AND !IsPublic AND !IsEntryPoint AND !IsExplicitInterfaceImpl AND !IsClassConstructor AND !IsFinalizer

I ran this CQL query without the "TOP 10" qualifier on some code on one project and found 292 methods that weren't being used.

Detect Overly Complicated Methods

WARN IF Count > 0 IN SELECT TOP 10 METHODS WHERE
( NbILInstructions > 200 OR
ILCyclomaticComplexity > 50 OR
NbParameters > 5 OR
NbVariables > 8)
AND
// Here is a way to avoid to take account of generated methods.
// Another way would be to use the SELECT METHODS FROM or SELECT METHODS OUT OF facilities.
!( NameLike "InitializeComponent" OR NameLike "Generated")

Here's a great one. This gives methods that have over 200 instructions or a complexity of over 50 (REALLY complex), or have too many variables or parameters, but includes a trick to exclude both generated code and that crap that the WinForms designer creates in InitializeComponent. 

These are just some of the dozens of rules and examples that it includes. I personally find writing this kind of CQL to be WAY more intuitive and flexible than writing FxCop rules/plugins.

If you select any CQL query, it runs instantly as you select it and the Tree Map at the top updates automatically, lighting up the result of the query. The toolbox in the upper left corner shows the result of the query in a table. You can then click on any of the methods and NDepend will open Visual Studio and take you there.

The TreeMap in the diagram below acts like a "heat map" showing the hot spots discovered. The blue area on the left side of the map shows the methods that came back as a result of my query. Above the TreeMap is a combo box that indicates what metric the TreeMap is using to determine the size of the squares. In this screen shot I'm using # of IL instructions so the larger the square, the more instructions. I could use lines of code, or cyclomatic complexity, or whatever to find the "fat" parts of my application.

CQL queries can these can include "WARN" at the beginning so they can be used within a Continuous Integration build. NDepend includes a NAnt Task and an MSBUILD Task, as well as stylesheets and plugins for Cruise Control.NET so I can add constraints to keep our applications in line as changes are checked in. The rules (the CQL) is really easy to read and the editor includes intellisense. 

Another nice diagram that is produced by NDepend is the chart of Abstractness vs. Instability. It says a lot, and it's easy to take the data as a value judgement directed towards your code. As if your assembly appears in Red, you're a Bad Person.

The Y-axis indicates how Abstract your assembly is - that is, is it able to be extended without recompile. The X-axis is instability. Now, "instability" carries a value judgement in itself. They aren't saying that stable is good and instable isn't. Stable means that the public surface area of your assemblies "can't" change because there are a lot of assemblies that are dependent on it. An instable assembly has fewer, or no, dependant assemblies upstream. Take a look at Page 8 of this document on Principles of Package Design. A number between 0 and 1 is assigned for each axis based on those equations and your assembly is plotted accordingly.

Here's the trick in reading the chart:

  • If an assembly is very stable (that is, lots of folks depend on it) and it's not extensible (no abstract classes, no virtuals, etc) then you're in the lower-left quadrant of the chart in the well-named Zone of Pain.
  • If an assembly is very abstract, very extensible, but no one depends on it (it's not really being used) then it moves towards the Zone of Uselessness.

Notice how many of the assemblies are in the Zone of Pain? So many, in fact, that you can't even read the words Zone of Pain any more. If you read the assembly names, you'll see that the ones that have made their way over into that portion of the chart are most of the 3rd party assemblies we use. This makes sense because these assemblies are Stable - lots of folks depend on them - and they were designed for some specific function, not for extensibility. Over there are things like the SharpZip lib, the FreeTextBox, and SubText's Akismet library. Is it bad that there are libraries over there? It's less something to panic about then it is something to have increased awareness about. Charts like this can give you a lot of insight into how hard it would be to replace a 3rd party component if you needed, or just to extend one. Here's a screencast on doing an impact analysis on replacing a 3rd party component within NDepend itself.

We can also note that both the DasBlog Runtime and Web.Core assemblies are edging their way over into the Zone of Pain. That implies that more and more we are depending on those assemblies in unhealthy ways because they aren't well-factored for use by both the application itself and the Plugins. All this equals opportunity to possibly modify our design.

I will point out, just as a bit of interest, that when Patrick and I ran this same chart on Corillian's Voyager SDK - an SDK that has had more eyes on it, perhaps, than DasBlog - we had only one assembly in the "Zone of Pain" and none in the Zone of Uselessness. We felt OK about that.

Wrap up

I'm really just touching on the beginnings over what a good static analysis tool can do. But you have to integrate a tool like this not only into your life, but into your build process and your company's software development process. You can't just drop the tool into your word and expect instant results. If you have a fairly mature process - agile or not - and developers who are enthusiastic to this kind of very hard-numbers input, you can begin by integrating this data into your existing code reviews. Start up a working group or study group to talk about what the information means - you don't even need to take action, just talk amongst peers. After your team starts to get a sense of where the metrics are not only taking them, but when to ignore the metrics and why, then you can start adding custom queries and warnings and integrating them into your continuous build.

NOTE: One last thing, to be clear. I don't know the guy from NDepend personally. I did chat him on Skype to answer some of my more esoteric questions. I'm not paid for being a reviewer of this product, in this case I'm just a massive FanBoy, much like I am about CodeRush. Neither of them has ever given me a penny.

I'm writing this because I want folks to think about their designs, not just up front, but all through the design...before, during, after. Think about complexity, think about supportability. Realize that your code, your app, your suite is very likely more complex than can be held in your head, and if you take the time - both in your day, and scheduled into your project - to review your design, truly understand what's going on, and use your analysis to make substantive changes to improve the quality of your company's code. Tools like NDepend or one of the many code analysis tools out there like Lattix, can really give you a great deal of insight and perspective on how your applications fit together, if you just take a few hours to get over the initial "hump of difficulty."

Once I realized the depth and breadth of the information I was looking at it, I was like a kid in a candy shop. It was the first time I discovered JetBrains dotTrace profiler - "and now I can see."

Go check out NDepend. There is a FREE/ACADEMIC version that you can try out and use on your Open Source apps, and the full Pro Version is about US$400 with steep discounts if you buy in quantity. Most companies would only need 1 or 2 copies for a lead "code analyst" I think, so it's reasonably priced. Run it on your companies application and show your boss a few graphs - and explain what he's looking at - and I'm sure he or she will give you the money. ;) I recommend trying to stay out of the Zone of Pain.

About Scott

Scott Hanselman is a former professor, former Chief Architect in finance, now speaker, consultant, father, diabetic, and Microsoft employee. He is a failed stand-up comic, a cornrower, and a book author.

facebook twitter subscribe
About   Newsletter
Sponsored By
Hosting By
Dedicated Windows Server Hosting by SherWeb

Disclaimer: The opinions expressed herein are my own personal opinions and do not represent my employer's view in any way.