Scott Hanselman

Example: How to contribute a patch to an Open Source Project like DasBlog

August 2, '06 Comments [15] Posted in ASP.NET | DasBlog | Programming
Sponsored By

Lots of discussion going on around the death of NDoc and Open Source as a viable option for .NET projects. A lot of this harkens back to some of the things discussed in "Is Open Source a Crap Idea" from a a few months back.

  • I said Open Source is free like a Puppy. It requires care, feeding, and in the words of Pete Seale, if you beat it it will run away.
  • Phil says Open Source is free like a Flower. "I think of Open Source software as being like a nice set of flowers in a common space such as a courtyard. Nobody that lives around the common space owns the flowers, yet they all enjoy the presence of the flowers."
  • Jeff says Open Source is free as in Free. "The highest compliment you can pay any piece of open source software is to simply use it, because it's worth using."

From my point of view, just USING Open Source software doesn't fix bugs. Volunteers fix bugs, whether they are a member of the project or not. (Assuming you haven't got an Open Source Sugar Daddy or a company that supports your work.)

All that is required for Evil to succeed is that Good Men [and Women] do nothing.

If you have the power to fix a bug (or at least get the fix started), especially in software you're using, why not make THAT your contribution?

Jeff says:

Fine. Create a new blog entry. Walk us through, step by step, the process for contributing a fix to DasBlog. List EVERY step, starting with finding a bug.

If it's that easy, prove it.
Jeff Atwood on August 2, 2006 09:40 AM

Kind of a sassy tone that, Jeff ;) but here you go.

Note, the bug might be one line, a single char, or quite involved. It all depends. The point is, any help you can give an Open Source dev, no matter how little is a good thing. If you can give them a line number and an idea, that's contributing. Here's an ideal (and real) bug report that stops just short of including the fix. We'll fix it and submit the patch.

How to fix a bug in an Open Source Project
(Or at least one that uses Subversion or CVS and is hosted at SourceForge)

Get the Project

Install TortoiseSVN and/or TortoiseCVS, depending on which source control system the Open Source Project is using. They both work about the same, integrating into Explorer.

Make a new folder, right-click, select Checkout.

Opensource1

Enter the URL of the Source Repository and click OK. In this example, the project at Source Forge is called "dasblogce" and we are getting the "trunk" so the URL is https://svn.sourceforge.net/svnroot/dasblogce/trunk. This is how folks are getting the very latest "bleeding edge" versions of DasBlog, since we haven't released in a while.

Opensource2

Build the Project

Every project is a little different, but most have a build file, readme or some detail that says how to get started.

Assuming you have IIS installed, to build DasBlog, run the CreateDasBlogVdir.vbs file in the tools folder to make the IIS Virtual Directory for your local copy.

Open up DasBlog All.sln in Visual Studio 2003 and build. Your virtual directory is mapped to <projectdir>\newtelligence.DasBlog.Web and your binaries - since this is a web application - are in the <projectdir>\newtelligence.DasBlog.Web\bin folder.

Example: Find a Bug

So perhaps you've found a bug. Maybe you got the Yellow Screen of Death. You likely have a stack trace or a strange bit of behavior.

Most projects at SourceForge have a home page like http://sourceforge.net/projects/dasblogce. Visit ours and click the Bugs menu item. Pick one, or see if the bug you've found has already been reported.

For example, George V. Reilly reports in Bug 1397557 that our "content filter" feature for making words link as Google Searches doesn't work well. He writes:

[The Content Filter] $g(multiple words) should produce
<a href="
http://www.google.com/search?q=multiple+words">
multiple words</a>. In other words, the spaces should
be URL encoded as pluses in the URL.

I spent some time looking into a fix. It doesn't seem
to be possible with a regex. See the thread I started at
http://regexadvice.com/forums/14513/ShowPost.aspx

My suggested fix.
$g(expr) ->
<a href="
http://www.google.com/search?q=$h(${expr})">
${expr}</a>
where $h is a new kind of content filter, one that uses
a builtin function to URL encode the string.

Here he is kind enough to suggest the desired behavior. As you may have found the bug or undesirable behavior, you may have an opinion as to how to fix it.

Example: Fixing the Bug

So since this bug involves DasBlog's content filter feature, I'll search the project for "ContentFilter" using Ctrl-Shift-F (Find in Files).

Opensource4

Looking at my search, "ApplyContentFilters" looks promising in Utilities line 645.

As George points out in the bug report, we want to UrlEncoded the result, but we don't want to UrlEncode all results. Just some.

He also helpfully points me to a post on a RegularExpressions forum where he tried to get the bug fixed using pure Regex. It doesn't appear possible with just RegEx, so we'll use a System.Text.RegularExpressions.MatchEvalulator like the post suggested. He also recommends a new kind of Content Filter that would UrlEncode. In this case, it'll be necessary because we want to do this:

$g(The Quick Brown Fox)

into

<a href="http://www.google.com/search?q=The+Quick+Brown+Fox">The Quick Brown Fox</a>

Note that if we did a blanket UrlEncode on the first string, we'd get + signs in our linked text.

So, we'll take his suggestion. Here's the code for a new ApplyContentFilters function with our changes in red.

Remember, we will be submitting our changes as a DIFF file for the developers/commmiters/admins to check in, as we're assuming we don't have source 'commit' access.

private static string CustomUrlEncodingRegexReplacer(Match m)

{

    // Remove the $<char>() from around the string and encode the result.

    if (m.Value.Length < 4) throw new ArgumentOutOfRangeException("Match",m.Value,"UrlEncoded Content Filters should be in the format $u() where 'u' is any single character. Match strings must be at least 4 characters long.");

    string trimmed = m.Value.Substring(3,m.Value.Length-4);

    return HttpUtility.UrlEncode(trimmed);

}

 

public static string ApplyContentFilters( SiteConfig siteConfig, string content )

{

    if ( content == null ){return String.Empty;}

 

    foreach ( ContentFilter filter in siteConfig.ContentFilters )

    {

        if ( filter.IsRegEx )

        {

            try

            {

                if(filter.UrlEncode)

                {

                    content = Regex.Replace(content, filter.Expression, new MatchEvaluator(CustomUrlEncodingRegexReplacer), RegexOptions.Singleline);

                }

                content = Regex.Replace(content, filter.Expression,filter.MapTo,RegexOptions.Singleline);

            }

            catch

            {

            }

        }

        else

        {

            content = content.Replace( filter.Expression, filter.MapTo );

        }

    }

    return content;

}

And we'll add our new Content Filter as George suggested to the XML settings file. (Note: we don't need to edit the XML directly, we'll change the editor page in a moment)

Again, changes in red.

  <ContentFilters>
    <ContentFilter find="\$g\((?&lt;expr&gt;[\w\s\d]+)\)" replace="&lt;a href=&quot;http://www.google.com/search?q=$u(${expr})&quot;&gt;${expr}&lt;/a&gt;" isregex="true"/>
    <ContentFilter find="\$u\((?&lt;expr&gt;[\w\s\d]+)\)" replace="${expr}" isregex="true" urlencode="true"/>
...snip other filters...
  </ContentFilters>

So now after our Google filter (or dictionary filter, or any of the dozens of filters that folks use) runs, we'll be able to UrlEncode arbitrary parts of the result.

Now we'll add UrlEncode as an option to the ContentFilters object. We'll make false the default so we don't break our public interface as we don't want to break existing code or change existing behavior.

Opensource4

Content Filters don't have to be edited in the XML file directly, they are managed by a rich UI. We'll want to extend the UI quickly.

On the EditContentFilters page, I'll add a new column to allow us to edit this new option in "EditContentFilters" by copy-pasting from another column.

Opensource5

I also do copy paste of the code to update the config file by using the existing "IsRegEx" example. It's a true/false checkbox-style option, just like the new one we're adding.

Here's a test of the results. Note the + in the URL in the Status Bar.

Opensource6

Making the Patch

Now that we think the bug is fixed, we want to make a patch/diff file that will allow the developer(s) to apply easily so they can consider our patch for committing to the project. We don't want to email them a zip of our project directory. That would mean they'd have to manually diff each file and that's a hassle. Remember the goal here is easy on everyone's part. We don't want to bust our asses sending in the patch and we don't want to stress them out by sending 4 megs of code when we only changed 10 lines.

Now I'll right click in the main folder of my project in Explorer and click Create Patch.

Opensource7

From the Create Patch window I will select the files I changed:

Opensource8

In this example, it was only four files. I'll save the unified diff/patch as "contentfilterchange.patch"

Opensource9

If I look inside the patch file with Notepad2 (with it's lovely diff/patch syntax highlighting) I'll see just my few changes with deletes and adds highlighted.

Opensource10

Now, we'll log back into SourceForge and upload and attach our patch to the bug.

Opensource11

Conclusion

We've just downloaded source to an Open Source application, built it, found a bug, fixed the bug, and submitted a patch to an Open Source Project that we were not committing members of. 

About Scott

Scott Hanselman is a former professor, former Chief Architect in finance, now speaker, consultant, father, diabetic, and Microsoft employee. He is a failed stand-up comic, a cornrower, and a book author.

facebook twitter subscribe
About   Newsletter
Sponsored By
Hosting By
Dedicated Windows Server Hosting by SherWeb

The importance of P3P and a Compact Privacy Policy

August 1, '06 Comments [5] Posted in ASP.NET
Sponsored By

P3p1UPDATE: Feedburner support rocks. One thing you can say about Web 2.0, it's agile. Feedburner is curently rolling out P3P based on this post. Some interesting talk happening in the comments of this post about possibly passing on/through existing policy!

I noticed recently that a number of cookies from Feedburner were being blocked by my browser. In this case, I was running IE6 in Medium Security Mode, the default mode. They don't have a Compact Privacy Policy returned in their HTTP Headers:

GET /~s/ScottHanselman?i=
    http://www.hanselman.com/blog/foofooHTTP/1.1
Accept: */*
Referer: http://www.hanselman.com/blog/foofoo

Accept-Language: en-us
Accept-Encoding: gzip, deflate
User-Agent: Mozilla/4.0 (blah blah blah)
Host: feeds.feedburner.com
Connection: Keep-Alive
Cache-Control: no-cache

HTTP/1.1 200 OK
Date: Tue, 01 Aug 2006 07:02:46 GMT
Server: Apache/2.0.54 (Debian GNU/Linux) mod_fastcgi/2.4.2 mod_jk/1.2.15
Set-Cookie:
    fbsite=xxxxxxx-xxx-xxxx-xxxx-xxxx-xxx
Content-Length: 1809
Keep-Alive: timeout=1, max=99
Connection: Keep-Alive
Content-Type: application/x-javascript;charset=ISO-8859-1

What is Platform for Privacy Preferences (P3P)?

The P3P standard is designed to do one job and do it well - to communicate to users, simply and automatically, a Web site's stated privacy policies, and how they compare with the user's own policy preferences. Although P3P provides a technical mechanism for helping inform users about privacy policies before they release personal information, it does not provide a mechanism for ensuring sites act according to their policies.

In most cases, the first time a user visits a Web site, their browser will have to make one or two additional requests in order to locate and fetch the P3P policy. These requests may impose some minimal latency; however, the delay caused by this should usually be less than the delay from fetching a single image in a Web page. Subsequent requests to the same site will usually not incur any additional latency due to P3P, as long as the site's policy has not expired.

Currently both Internet Explorer 6 and Netscape 7 implement privacy-related features based on the P3P standard.

Nine aspects of online privacy are covered by P3P. Five detail the data being tracked by the site.

  • Who is collecting this data?
  • Exactly what information is being collected?
  • For what purposes?
  • Which information is being shared with others?
  • And who are these data recipients?

The remaining four explain the site's internal privacy policies.

  • Can users make changes in how their data is used?
  • How are disputes resolved?
  • What is the policy for retaining data?
  • And finally, where can the detailed policies be found in "human readable" form?

P3P policies aim to answer all these questions and allow the user, and the user's browser, to make decisions about content presentation and cookie acceptance based on answers to these questions.

Technical Details

P3P is a way of expressing a site’s published privacy policy using HTTP Headers. This can be expressed via an XML file pointed to in an HTTP Header.

Example

1. Client makes a GET request.

GET /index.html HTTP/1.1
Host: catalog.example.com
Accept: */*
Accept-Language: de, en
User-Agent: WonderBrowser/5.2 (RT-11)

2. Server returns content and the P3P header pointing to the policy of the resource.

HTTP/1.1 200 OK
P3P: policyref="
http://catalog.example.com/P3P/PolicyReferences.xml"
Content-Type: text/html
Content-Length: 7413
Server: CC-Galaxy/1.3.18

Alternatively, and more commonly, compact policies are summarized P3P policies that provide hints to user agents to enable the user agent to make quick, synchronous decisions about applying policy. Compact policies are a performance optimization that is optional for either user agents or servers. User agents that are unable to obtain enough information from a compact policy to make a decision according to a user's preferences SHOULD fetch the full policy

Example

1. Client makes a GET request.

GET http://news.com.com/html/js/timediff.js HTTP/1.1
Host: news.com.com
Accept: */*
Accept-Language: en-us,es;q=0.7,he;q=0.3
User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)

2. Server returns content and the P3P header including the compact policy.

HTTP/1.1 200 OK
P3P: CP="CAO DSP COR CURa ADMa DEVa PSAa PSDa IVAi IVDi CONi OUR OTRi IND PHY ONL UNI FIN COM NAV INT DEM STA"
Content-Type: text/html
Content-Length: 2248

Compact policies can be created manually but the syntax is nuanced. Most developers use a commercial web-based questionnaire like http://p3pedit.com/. These compact policies can be applied directly, often without source code changes, using Web Server’s administration tool.

How does it affect the end user?

If an end-user has set their browser, in this example IE6, to a privacy level of "High" all cookies will be blocked if the requested site doesn’t include P3P.  The user will be informed of the blocked cookie in the status bar of their browser. This is often too subtle for most users. If this site had a P3P policy available the cookie would have been allowed even though this user’s browser Privacy setting is "High."

P3p2

If your site doesn’t have a P3P policy you are virtually guaranteed calls from users unable to login. If you're running a blog with 3rd party advertising, you're likely not collecting a complete view of your users as most are blocking your cookies.

It is important to point out that Privacy options are not Security options. Cookies, used correctly, are not inherently insecure as a technology. They provide a valuable function for the end user and the developer.

P3p3

Note that if the user sets their privacy settings to "Block All Cookies" there is nothing that can be done on the server-side – they have chosen not to receive cookies.

What should I do?

Use an online questionnaire like http://p3pedit.com/ to generate a P3P Policy XML file and a Compact Policy to be applied to the site.

Use Internet Services Manager within MMC to configure Microsoft Internet Information Services (IIS) to set custom header properties to pages, virtual directories, or entire Web sites. To enable P3P custom headers using Internet Services Manager to configure IIS. (NOTE: If you don't have access to your IIS instance or your ISP doesn't want to help you out, you can also add these HTTP Headers programmatically using an HttpModule.)

1. Right-click the desired page, directory, or site, and then click Properties.
2. On the HTTP Headers tab, click Add.
3. In the Custom Header Name field, type P3P.
4. In the Custom Header Value field, enter your Compact P3P Policy and then click OK.

You can then validate your site's compliance with P3P using the W3C's online validator at http://www.w3.org/P3P/validator.html. There is a detailed deployment guide available.

About Scott

Scott Hanselman is a former professor, former Chief Architect in finance, now speaker, consultant, father, diabetic, and Microsoft employee. He is a failed stand-up comic, a cornrower, and a book author.

facebook twitter subscribe
About   Newsletter
Sponsored By
Hosting By
Dedicated Windows Server Hosting by SherWeb

Websense update - I'm not banned anymore!

July 31, '06 Comments [4] Posted in Musings
Sponsored By

A number of readers have emailed recently saying that the oppressive regime that is their IT department had banned this blog via their Websense tool because it was categorized as "Personal Web Sites; Society and Lifestyles."

I'm pleased to announce:

Thank you for writing to Websense.

The site you submitted has been reviewed.  We have made an update to the following URL in our master database to address this issue:

http://hanselman.com/blog/ - Information Technology

Categorization updates should be available in the next scheduled publication of the database.  A new database is published every business day, five days a week, Pacific Standard Time.  You should notice any updates referred to in this message within 72 hours.

Thank you for your assistance,

The Websense Database Services Staff

Yay! Welcome folks from behind the wall.

About Scott

Scott Hanselman is a former professor, former Chief Architect in finance, now speaker, consultant, father, diabetic, and Microsoft employee. He is a failed stand-up comic, a cornrower, and a book author.

facebook twitter subscribe
About   Newsletter
Sponsored By
Hosting By
Dedicated Windows Server Hosting by SherWeb

PUSHD reminder - it automatically maps network drives

July 31, '06 Comments [0] Posted in PowerShell
Sponsored By

I've blogged a few times about how cool PUSHD (and POPD) is from the cmd.exe prompt. It's one of the most powerful "DOS" commands that few folks use.

PUSHD, of course, maintains a stack of where you are in your file system. It will PUSH a directory on the stack and move you there automatically. Combined with a PROMPT variable that includes (somewhere) a $+, you'll get a very powerful way to move about.

Example from CMD>EXE:

C:\Documents and Settings\Scott>PUSHD c:\windows
C:\WINDOWS+>pushd system32
C:\WINDOWS\system32++>popd
C:\WINDOWS+>popd
C:\Documents and Settings\Scott>

One thing that I don't see a lot is PUSHD with UNC Paths, and how it will automatically map a drive for you, starting at Z: moving backwrads and will unmap them when you POPD.

Example from CMD.EXE:

C:\Documents and Settings\Scott>PUSHD \\SCOTTPC\D
Z:\+>PUSHD \\SCOTTPC\DESKTOP
Y:\++>POPD
Z:\+>POPD
C:\Documents and Settings\Scott>cd
\\scottpc\desktop
'\\scottpc\desktop'
CMD does not support UNC paths as current directories.

But, of course, I can't CD to a UNC path with CMD.EXE.

However, all these scenarios, plus CD'ing to UNC paths work within Powershell:

PS C:\Documents and Settings\Scott> pushd \\scottpc\desktop
PS Microsoft.PowerShell.Core\FileSystem::\\scottpc\desktop> cd \\scottpc\d
PS Microsoft.PowerShell.Core\FileSystem::\\scottpc\d> cd \\scottpc\desktop
PS Microsoft.PowerShell.Core\FileSystem::\\scottpc\desktop> c:
PS C:\Documents and Settings\Scott> PUSHD
\\SCOTTPC\D
PS Microsoft.PowerShell.Core\FileSystem::\\SCOTTPC\D> PUSHD \\SCOTTPC\DESKTOP
PS Microsoft.PowerShell.Core\FileSystem::\\SCOTTPC\DESKTOP> POPD
PS Microsoft.PowerShell.Core\FileSystem::\\SCOTTPC\D> POPD
PS C:\Documents and Settings\Scott> cd
\\SCOTTPC\DESKTOP

Nice stuff to know. Thanks to Ryan Carr for the reminder.

About Scott

Scott Hanselman is a former professor, former Chief Architect in finance, now speaker, consultant, father, diabetic, and Microsoft employee. He is a failed stand-up comic, a cornrower, and a book author.

facebook twitter subscribe
About   Newsletter
Sponsored By
Hosting By
Dedicated Windows Server Hosting by SherWeb

Sandcastle - Microsoft CTP of a Help CHM file generator on the tails of the death of NDoc

July 30, '06 Comments [12] Posted in PowerShell | DasBlog | Subversion | NUnit | NCover | Nant | XML | Bugs | Tools
Sponsored By

Sandcastle1Moments ago (my time) the Sandcastle CTP was released. Here's the Sandcastle Blog and here's a PowerPoint presentation on the new project. This is a very early CTP from Microsoft that supports generating documentation from any .NET language, much like NDoc.

It's great that Microsoft is paying attention to the whole "need for help files thing." However, be warned, this is uber-early stuff, and not very smooth. Actually, it's pretty darned rough. The instructions on what your batch/build/msbuild/powershell/whatever is going to need to orchestrate is here. The instructions are ghetto. Here's a slightly less ghetto Powershell script that will at least compile the example, assuming you have Powershell.

  • Assuming you have .NET 2.0 SDK and Powershell...you'll need to, of course, enable scripts via something like set-executionpolicy unrestricted
    • Note: Powershell has nothing to do with Sandcastle. I just did the script because it's wicked easy in PSH.
  • Download Sandcastle July CTP.
  • Run this Powershell script of mine to build the example: File Attachment: sandcastledoc.ps1 (1 KB)

Remember you'll need HTML Help Workshop if you're going to make CHMs (Compiled Help files). Here's the compiled example test.chm: File Attachment: Test.chm (31 KB)

Sandcastle for .NET 1.1

One note, I was able to get Sandcastle to generate help for a .NET 1.1 application, which is a very important developer scenario I hope they don't forget about. However, Sandcastle linked the 1.1 help up to the Framework 2.0 XML help for the .NET Framework BCL (Base Class Library) by default. If you change the sandcastle.config to refer to
<data files="%SystemRoot%\Microsoft.NET\Framework\v1.1.4322\*.xml" /> (line 48 in this CTP)
it appears to link up nicely for 1.1 apps even though Sandcastle uses .NET 2.0 for its reflection.

NDoc: The Death of a (great) Open Source Project

On a related note, it's going to take a while (6 months to a year?) for Microsoft to really get Sandcastle to the point where Kevin Downs got NDoc. Will this new tool be as rich and useful? Or will it be forgotten like HTML Help Workshop?

Recently Kevin Downs, the leader of NDoc, emailed a NDoc folks announcing that NDoc is dead. I was shocked to get this email, but sadly, not surprised. Here's an important part of his email:

Unfortunately, despite the almost ubiquitous use of NDoc, there has been no support for the project from the .Net developer community either financially or by development contributions. Since 1.3 was released, there have been the grand total of eleven donations to the project. In fact, were it not for Oleg Tkachenko’s kind donation of a MS MVP MSDN subscription, I would not even have a copy of VS2005 to work with!

To put this into perspective, if only roughly 1-in-10 of the those who downloaded NDoc had donated the minimum allowable amount of $5 then I could have worked on NDoc 2.0 full-time and it could have been released months ago! Now, I am not suggesting that this should have occurred, or that anyone owes me anything for the work I have done, rather I am trying to demonstrate that if the community values open-source projects then it should do *something* to support them. MS has for years acknowledged community contributions via the MVP program but there is absolutely no support for community projects.

Apparently Kevin started getting threats - yes, you heard right, threats - about a .NET 2.0 version and has been email-bombed. He's rightfully decided to bow out after a successful run.

If you're a fan of the whole N* stack, you've used NAnt, NUnit, NDoc, NCover, for years. We take for granted that these programs just work. They are fundamental. Some folks think they are our right to possess, but they forget about the real people with real lives that write this Open Source stuff in their spare time.

Hanselman Editorial Aside: It's a shame that Microsoft can't put together an organization like INETA (who already gives small stipends to folks to speak at User Groups) and gave away grants/stipends to the 20 or so .NET Open Source Projects that TRULY make a difference in measurable ways. The whole thing could be managed out of the existing INETA organization and wouldn't cost more than a few hundred grand - the price of maybe 3-4 Microsoft Engineers.

Phil makes a good point when it compares Open Source to "Source Available" with regards to Community Server. It's great that some OS products can turn into commercial apps with an OS "lite" version.

For "base of the pyramid" fundamental stuff like Build, Test, Coverage, Docs, will we pay for them? We should. Should we have given the NDoc project $5? Did NDoc help me personally and my company? Totally. Did I donate? No, and that was a mistake. I agree with Phil. Support those 5, 10, 20 truly Open Source projects with a little of your time or money.

Personally, as an Open Source project co-leader, I'd much rather folks who use DasBlog pick a bug and send me a patch (unified diff format) than give money.  I suspect that Kevin would have been happy with a dozen engineers taking on tasks and taking on bugs in their spare time.

We are blessed. This Open Source stuff is free. But it's free like a puppy. It takes years of care and feeding. You don't get to criticise a free puppy that you bring in to your home.

Goodbye Kevin and thanks for your hard work on NDoc.

About Scott

Scott Hanselman is a former professor, former Chief Architect in finance, now speaker, consultant, father, diabetic, and Microsoft employee. He is a failed stand-up comic, a cornrower, and a book author.

facebook twitter subscribe
About   Newsletter
Sponsored By
Hosting By
Dedicated Windows Server Hosting by SherWeb

Disclaimer: The opinions expressed herein are my own personal opinions and do not represent my employer's view in any way.