Scott Hanselman

Hacked! And I didn't like it - URLScan is Step Zero

August 11, '08 Comments [36] Posted in ASP.NET | ASP.NET MVC | IIS | Tools
Sponsored By

My blog was down a few days ago. I've had downtime in the minutes over the last few years, but as far as I recall, it's never been down for any significant time. Keyvan noticed that a bunch of us were attacked. Phil Haack was also, ahem, haacked.

UPDATE: To be clear, I wasn't really hacked. I was "DoS'ed" or brought down for a little bit by a distributed denial of service attack that spiked my CPU. I'm advocating that you constrain the URLs that input that get to your  application by either black-listing, or white-listing allowed content.

I host at ORCSWeb and have forever. We're in the process of making a lot of chances to my blog. I'm on an x64 machine (I've blogged about DasBlog on x64 before), but running in 32-bit AppPool We're moving my blog to a dedicated server, switching to x64, and we also were upgrading to UrlScan 3.0 which just had a Beta Release in June.

Anyway, in this crazy process, there was window of time where I didn't have UrlScan enabled on the machine. I mistakenly thought that the Ninjas wouldn't be able to catch me if I was on fire. In fact, not so.

ninjasonfire

Speaking of Ninjas, Wade Hilmo is a ninja at Microsoft who writes UrlScan.

There's a great IIS7 Request Filter for protecting against nasty attacks, but UrlScan Beta 3.0 still has the edge on the filter for the time being. Version 3.0 of UrlScan adds:

  • Support for query string scanning, including an option to scan an unescaped version of the query string.
  • Change notification for configuration (no more restarts for most settings.)
  • UrlScan can be installed as a site filter.  Different sites can have their own copy, with their own configuration.
  • Escape sequences can be used in the configuration file to express CRLF, a semicolon (normally a comment delimiter) or unprintable characters in rules.
  • Custom rules can be created to scan the URL, query string, a particular header, all headers or combination of these.  The rules can be applied based on the type of file requested.
  • Support for 64 bit IIS worker processes.

This release of UrlScan is a beta, but it's config file is backward compatible and there's a GoLive license. It's working great for me. However, to quote Wade:

"While they are effective against the current wave of automated attacks, they cannot protect against more directed attacks against a specific server."

This was a SQL Injection attack with URLs that looked like this (and some variations):

[08-11-2008 - 17:29:31] Client at 201.67.x.x: Query string length exceeded maximum allowed. Request will be rejected. Site Instance='13', QueryString= 'guid=0b93befc-3543-4bfc-ba8e-6cd340b6d9d3;DECLARE%%20@S%%20VARCHAR(4000);SET%%20@S=CAST(0x4445434C4152452...(incrediblyLONGQueryString)...220%%20AS%%20VARCHAR(4000));EXEC(@S);--', Raw URL='/blog/CommentView.aspx'

In this example, it's hitting CommentView.aspx and trying to add a bunch of T-SQL at the end, with the most evil part encoded inside a CAST() statement. It's a distributed attack with a bunch of (likely innocent) drones reaching out to be mean. In a few hour period, there were thousands of attacks for over 250 different IP addresses.

Fortunately DasBlog doesn't use a database at all, rather a bunch of XML files for storage. Unfortunately, the application was still trying to map these query strings to blog posts, and the result took my blog down.

There's really two main things to think about when dealing with user input, remembering that the URL is an input point for your application!

  1. Trust no input from the user.
  2. Constrain the input that reaches your application code as much as you can. Deny as early as possible (hardware, load-balancer, appliance, module, etc).

We need to tighten up DasBlog to more quickly reject URLs that are surely not requests for blog posts, but a tool like UrlScan allows me to easily reject obvious attacks in an way that is more efficient than letting my application code do it.

I would encourage you to take a moment and do a threat analysis on your own websites, and make sure that you ARE constraining input appropriately.

One thing to note, you can and will likely break things for a while with UrlScan, as it does constrain input and you might have valid URLs you aren't aware of. For example, UrlScan broke OpenID authentication for me as the query strings included dots, which UrlScan was denying and also the presence of the word "open" in the querystring. Other denials can happen because of keywords in the URL or length of the querystring. Be sure to test appropriately and watch the UrlScan logs of denials. You can set very blanket rules, or constrain by extension.

We always installed UrlScan on staging and production machines when I was in banking, and made them part of the testing and deployment process. In these times, having a filter installed like UrlScan is Step Zero. I will remember that in the future! Thanks to ORCSweb for answering my 2am emails and helping me fix it in near-real-time!

Technorati Tags: ,,

About Scott

Scott Hanselman is a former professor, former Chief Architect in finance, now speaker, consultant, father, diabetic, and Microsoft employee. I am a failed stand-up comic, a cornrower, and a book author.

facebook twitter subscribe
About   Newsletter
Sponsored By
Hosting By
Dedicated Windows Server Hosting by ORCS Web
Monday, August 11, 2008 11:20:51 PM UTC
Maybe time for LAMP and WordPress? ;) (Couldn't resist!)
Monday, August 11, 2008 11:26:48 PM UTC
Any way you would be willing to share your URLScan.ini file? I am having issues getting mine set up to be protective, but not overly destructive.

Thanks.
Matt
Monday, August 11, 2008 11:35:11 PM UTC
Why doesn't IIS7 include URLScan? I thought "Secure by default" was the new mantra.
Tuesday, August 12, 2008 12:09:46 AM UTC
hmmm. . . I am using BlogEngine.NET and a Cisco firewall, something tells me I might need to look into this.
Tuesday, August 12, 2008 12:17:18 AM UTC
The site I work on also got hit. Luckily parameterized queries, and TOP N queries with an explicit cast to an int meaned that the site did not get hit. Unfortunately a legacy ColdFusion app that was written over 5 years ago by some 3rd party allowed arbitrary SQL in! Including DROP TABLE commands! The exploit was a blessing and a curse because I discovered that old code and took it down, but the hackers managed to trash the database pretty good. I was able to repair everything with a glorious global search/replace across all tables/columns in the database.

The other gotcha with URLScan was that it blocks requests of a certain Content-Length. Our site has video uploads and then were hitting 404s upon completion because URLScan defaults to a 30MB Content-Length.

Scott
Tuesday, August 12, 2008 12:46:50 AM UTC
Hi Scott!

I'm running dasBlog installed in a shared hosting enviroment (as well as a lot of other dasBlog users I guess). Do you have any tips that we could use to prevent this sort of attacks?

I'm guessing that since I'm in a shared hosting enviroment I can't install UrlScan (I guess this is something my hosting provider has to do), but correct me if I'm wrong.
Tuesday, August 12, 2008 12:47:00 AM UTC
doooooood
off topic but SP1 was officially released today! Where's your post?

:)
Tuesday, August 12, 2008 2:26:04 AM UTC
Hey Doug:

http://www.techcrunch.com/2008/06/11/my-blog-was-hacked-is-yours-next-huge-wordpress-security-issues/

The type of attack we're talking about here could be attempted on any blog engine and take that engine down if the proper denial systems are not in place as Scott points out. Kudos to Scott for sharing the experience so we can all get busy checking our own vulnerabilities.

Glass houses and all that...

;-{>

-Tyler
Tuesday, August 12, 2008 2:32:36 AM UTC
Maybe time for LAMP and WordPress? ;) (Couldn't resist!) - WTF moron - I hate you idiots that think every hacked website on .NET is because it's "Microsoft" - this was a SQL injection and could potentially be used against any website - I've seen plenty of PHP sites that do ZERO querysting checking - seen plenty of unsecure MySQL also - go have another triple hamburger and diet coke
Joe C
Tuesday, August 12, 2008 2:38:28 AM UTC
sry (Couldn't resist!) LMAO
Joe C
Tuesday, August 12, 2008 3:03:35 AM UTC
A website I work on was also hacked last week. It was due to some ancient non-parameterized SQL query on our classic ASP front-end. A bit of research and 15 minutes later the attack was reversed... crafty bastards!
Tuesday, August 12, 2008 3:09:40 AM UTC
"Why doesn't IIS7 include URLScan?"

IIS7 includes virtually all the features of UrlScan 2.5; however, neither UrlScan 2.5 nor plain IIS7 provide facilities for filtering URI query strings. This is a new feature in UrlScan 3.0.

(Also, if the application running on top of IIS7 is insecure, as is the case in SQL injection attacks, there is only so much that IIS can do for it. UrlScan 3.0 provides a way of filtering URI query string based attacks but that'll only last until an attacker figures out a way to beat your filter. As Scott indicated, the ultimate solution is to fix the app's code to keep this from happening.)
Tuesday, August 12, 2008 9:07:31 AM UTC
So, you weren't really hacked, bur rather suffered a DDoS due to the hacking attempt. There is a significant difference.

Re constraining input, I don't know how UrlScan works (it's not clear from your post what you are advocating), but the way to go is white listing, not black listing.
Tuesday, August 12, 2008 9:23:17 AM UTC
I can't believe you haven't written about the SP1 release since it's here for a day or so.

Maybe it's the hack that's got you annoyed.

Still it's out there
http://www.microsoft.com/downloads/details.aspx?FamilyID=fbee1648-7106-44a7-9649-6d9f6d58056e&DisplayLang=en
Tuesday, August 12, 2008 9:36:36 AM UTC
Is it that Big Issue? Because I think Sql Injection attack is not that complex, Simply you recieve a http Request with sql Coomands. Analyze that if Key words, like "Select, Update, Delete, Varchar, exec, delete, Drop", exists in any part of GET and POST Requests.
and If there is any, Discard the request.

there is only one Problem with this Approach, that you will never be able to Submit a value, which Consists one of the Above Keywords.

PLZ Comment
Nawaz
Tuesday, August 12, 2008 9:43:50 AM UTC
Scott,

To avoid some issues with open, etc follow some tips discussed in the comments here

http://blogs.iis.net/nazim/archive/2008/06/30/using-the-new-rules-configuration-in-urlscan-v3-0-beta-part-2.aspx

about the rules for IIS.

I'll get a half decent urlscan.ini posted online at some point. Still waiting for Wade to fix urlscan 3.0 in iis 6 though as I cannot easily test in active production environments atm.

Alberto,

URLScan scans the URL for certain keywords (as the name suggests) so you need to have blacklists not whitelists.
Rovastar
Tuesday, August 12, 2008 1:12:19 PM UTC
I'm also going to move my blog from my Win 2003 VPS to my own dedicated Win 2008 x64 servers on MaximumASP to improve the quality and apply some changes. Installing UrlScan is also the first task in my TO DO list on all my dedicated servers and VPS's.
Tuesday, August 12, 2008 2:40:41 PM UTC
Hi Scott,

So it was a DDOS attack that actually took your site down for awhile?
I've been seeing these same errors in my own logs but my site is not vulnerable to SQL injection so it hasn't affected me much at least so far. My question is was the DDOS effective because it was raising errors or just because of the sheer volume of requests? I guess they haven't hit my sites as hard as yours since my sites have not been affected other than logging errors, but it raises a question that I've wondered about for a long time. I know that errors/sec is a performance metric we want to be low, but can causing these errors make it take substantially less requests to cause a denial of service than it would if no error was occurring?

I've seen so many variations of things that are apparently designed to cause ASP.NET errors, lots of viewstate errors for example, but one of the easiest ways to cause an error is apparently to make an http request for a protected file like anything with .config extension. I've seen a number of these type errors in my logs over time. I don't have url scan myself, but I guess you could block those type of requests too and keep it from causing errors.

Best Regards,

Joe Audette

ps really enjoyed your recent accessibility podcast!
Tuesday, August 12, 2008 2:56:31 PM UTC
@Douglas Karr - That's a slick solution, because we all know that the LAMP stack is immune to SQL injection attacks. If you read the post, you'd see that the blog wasn't actually hacked, just taken offline, since SQL injection doesn't work on text files.

My wife's (dotnetnuke) site wasn't taken down, but it got a lot slower due to filling up the log. I cleared the log and set it to only retain the past few days and it picked back again.

Tuesday, August 12, 2008 4:20:58 PM UTC
@Nawaz - I think this particular Injection Attack is putting everything in binary and so filtering on keywords doesn't help much. That's why a lot of folks are getting burned by it recently. This is where using LINQ comes in real handy.
Tuesday, August 12, 2008 5:32:45 PM UTC
Kevin J Baird,

You are wrong you can stop this attack:

QueryString= 'guid=0b93befc-3543-4bfc-ba8e-6cd340b6d9d3;DECLARE%%20@S%%20VARCHAR(4000);SET%%20@S=CAST(0x4445434C4152452...(incrediblyLONGQueryString)...220%%20AS%%20VARCHAR(4000));EXEC(@S);--', Raw URL='/blog/CommentView.aspx'

with urlscan 3.0

See the sql injection rules for URLscan here:

http://blogs.iis.net/nazim/archive/2008/06/30/using-the-new-rules-configuration-in-urlscan-v3-0-beta-part-2.aspx

Where the keyword CAST is used (which converts the hexstring to ascii) it is blocked (among others). In fact this one attack was stopped (in this example) by a limit to to the length of the query string that is processed before the sql injection rules.

Also you can alter the list to a more detailed on to stop 'false positives' for this.
Rovastar
Tuesday, August 12, 2008 6:10:43 PM UTC
Heh. TWO people asking about an SP1 post? You know it's out there, what would you like me to say about it? ;)
Tuesday, August 12, 2008 6:58:54 PM UTC
My company's site was also hacked. For Step0 we have now used Helicon's ISAPI-Rewrite. Similar idea as UrlScan. Is working pretty well so far.

Thanks
Ruchi
Ruchi Saini
Tuesday, August 12, 2008 7:16:50 PM UTC
I have a site that's been hit by this SQL Injection attack for several months now. What I've done as a quick fix was to put a QueryString.Contains("=CAST(0x") call in the Page_Load event for the site's single .master page and then redirect to a small static page. (I could have also returned without any processing.)

One thing that I'm doing though is logging the IP's but as you mentioned they are probably inoccent drones. However there's an interesting pattern. Two South American countries and the rest from Eastern Europe and Asia.
Tuesday, August 12, 2008 8:00:35 PM UTC
thanks a lot
Tuesday, August 12, 2008 8:22:09 PM UTC
This is a test.
test
Wednesday, August 13, 2008 12:27:30 AM UTC
@Nawaz, "Select, Delete and Update" are common words, and could be used in any blog comment - particularly, in blogs that talk about SQL :) - why should you block any http request with them? There are other (simpler) ways to avoid SQL Injection. Usually, using SqlParameters (in ADO.net) is enough. For other similar problems, know and test how your querystrings should be, set up whitelists, encode/unencode your input when needed. Just my 2 cents... :)
Wednesday, August 13, 2008 2:23:02 PM UTC
It appears that Rick Strahl (West Wind) was also recently hit by this same "attack". Not sure if he's on ORCSweb or not, but I get the impression that he's not. So it probably wasn't targeted specifically at ORCSWeb. Perhaps it was targeted at .Net bloggers.

http://www.west-wind.com/weblog/posts/447503.aspx

David Martin
Wednesday, August 13, 2008 3:12:34 PM UTC
@Dave Martin:

No, it wasn't an OrcsWeb attack as I'm hosted on my own VPS and Phil Haack is also hosted on CrystalTech.
Wednesday, August 13, 2008 7:13:48 PM UTC
Your querystrings have the same characterists of the aspbot hack which is very prevalent these days. It probably sniffed you out as an aspx site and started humping your leg over and over.
John A. Davis
Wednesday, August 13, 2008 7:40:43 PM UTC
Hi Scott,

Do you have any details on why these request were taking down dasDlog?

Were there just that many of them or is there something that dasBlog is doing with the query that was causing the problem?

Wednesday, August 13, 2008 8:16:13 PM UTC
Tony - From what I can tell, it was just massive volume.
Monday, August 18, 2008 8:21:59 PM UTC
This specific attack is one I've dealt with a few times in the past week (via urgent calls from people asking for help). The attack, if successful, results in drive-by malware being vectored through a vulnerable web site. ISAPI rewrites is one way to specifically redirect this and similar attacks.

Secure-by-default coding and configuration is the necessary approach. It can be hard to think like a bad guy though. Often people ask for a list of how they should code to avoid vulnerabilities, when perhaps a better approach would be to think about how to code to break your own system. As a rule (in my opinion) a combination of nothing-allowed-by-default input validation (with a whitelist of allowed content and types, where it's practical) and parameterized queries specifically for SQL-driven app implementations is good stuff. Neil Carpenter wrote fairly brilliantly and succinctly about parameterized queries recently - check it out. Says Neil: "To use an analogy, using input validation to stop SQL injection is like using anti-virus software to stop malware. It might work, it might not, but you’d be far better off if you actually resolved the vulnerability instead of just trying to mitigate it."
Wednesday, August 20, 2008 7:09:09 PM UTC
I'm a bit surprised to hear your blog is using XML as data storage instead of SQL Server. Is it because of caching that you get good performance out of XML, or does XML just perform well on its own compared to SQL Server?
Thursday, August 21, 2008 6:35:01 PM UTC
thanks a lot
Saturday, August 23, 2008 4:27:22 PM UTC
Visual Studio debugging note:

Once you install UrlScan 3.0 as an ISAPI filter you must ensure you have a properly configured UrlScan.ini file (found in the system/inetsrv/urlscan folder).

Ensure the UseAllowVerbs option (found at the top of the ini file) is =1 and add "DEBUG" in the [AllowVerbs] section.

You may refer to this MS Support article but be warned it improperly states adding "debug" instead of "DEBUG" to the AllowVerbs section.

(example)


UseAllowVerbs=1 ; If 1, use [AllowVerbs] section, else use the

[AllowVerbs]

;
; The verbs (aka HTTP methods) listed here are those commonly
; processed by a typical IIS server.
;
; Note that these entries are effective if "UseAllowVerbs=1"
; is set in the [Options] section above.
;

GET
HEAD
POST
DEBUG
Comments are closed.

Disclaimer: The opinions expressed herein are my own personal opinions and do not represent my employer's view in any way.