Scott Hanselman

HTTP POSTs and HTTP GETs with WebClient and C# and Faking a PostBack

December 3, '04 Comments [5] Posted in ASP.NET | ViewState | Tools
Sponsored By

A fellow emailed me wanting to screen scrape, er, ah, harvest a page that only displays the data he wants with a postback.

Remember what an HTTP GET looks like under the covers:

GET /whatever/page.aspx?param1=value&param2=value

Note that the GET includes no HTTP Body. That's important. With a POST the 'DATA' moves from the QueryString into the HTTP Body, but you can still have stuff in the QueryString.

POST /whatever/page.aspx?optionalotherparam1=value
Content-Type: application/x-www-form-urlencoded
Content-Length: 25
param1=value&param2=value

Note the Content-Type header and the Content-Length, those are important.

A POST is just the verb for when you have an HTTP document. A GET implies you got nothing.

So, in C#, here's a GET:

public static string HttpGet(string URI)
{
   System.Net.WebRequest req = System.Net.WebRequest.Create(URI);
   req.Proxy = new System.Net.WebProxy(ProxyString, true); //true means no proxy
   System.Net.WebResponse resp = req.GetResponse();
   System.IO.StreamReader sr = new System.IO.StreamReader(resp.GetResponseStream());
   return sr.ReadToEnd().Trim();
}

Here's a POST:

public static string HttpPost(string URI, string Parameters)
{
   System.Net.WebRequest req = System.Net.WebRequest.Create(URI);
   req.Proxy = new System.Net.WebProxy(ProxyString, true);
   //Add these, as we're doing a POST
   req.ContentType = "application/x-www-form-urlencoded";
   req.Method = "POST";
   //We need to count how many bytes we're sending. Post'ed Faked Forms should be name=value&
   byte [] bytes = System.Text.Encoding.ASCII.GetBytes(Parameters);
   req.ContentLength = bytes.Length;
   System.IO.Stream os = req.GetRequestStream ();
   os.Write (bytes, 0, bytes.Length); //Push it out there
   os.Close ();
   System.Net.WebResponse resp = req.GetResponse();
   if (resp== null) return null;
   System.IO.StreamReader sr = new System.IO.StreamReader(resp.GetResponseStream());
   return sr.ReadToEnd().Trim();
}

I could and should have put in more 'using' statements, but you get the gist. And, there are other ways to have done this with the BCL, but this is one.

Now, how would you fake an HTTP PostBack? Use a tool like ieHttpHeaders to watch what a real postback looks like, and well, fake it. :) Just hope they don't require unique/encrypted ViewState (via ViewStateUserKey or EnableViewStateMac) for that page, or you're out of luck.

About Scott

Scott Hanselman is a former professor, former Chief Architect in finance, now speaker, consultant, father, diabetic, and Microsoft employee. He is a failed stand-up comic, a cornrower, and a book author.

facebook twitter subscribe
About   Newsletter
Sponsored By
Hosting By
Dedicated Windows Server Hosting by ORCS Web
Friday, December 03, 2004 9:04:33 AM UTC
Whilst ieHttpHeaders is rather handy I've found the small toolbar just doesn't make it easy to read, especially with large requests (*cough* viewstate *cough*)

There's always http://www.fiddlertool.com/ which personally I find more flexible.
Friday, December 03, 2004 2:39:39 PM UTC
ieHttpHeaders - Broken Link
Friday, December 03, 2004 6:03:02 PM UTC
I fixed the link.
Friday, December 03, 2004 6:40:38 PM UTC
To postback the viewstate, it is useful to pick a landing page to hit before doing the postback, so that you can get a legit viewstate and not worry about whether the site has specific expectations of their viewstate. Basically, just pick the page that you would normally HTTP GET before you postback, and then make sure you copy the viewstate from the page to the post. You have to be careful to get the encoding right on the postback, since the viewstate is base64 and some of that isn't application/x-www-form-urlencoded compatible.
Monday, February 27, 2006 8:42:04 PM UTC
Hi,
First thank you very much for this piece of code :)
I would like to inform you System.Net.WebResponse has a Close() method, I think it should be better if you call it no?
Franck Quintana
Comments are closed.

Disclaimer: The opinions expressed herein are my own personal opinions and do not represent my employer's view in any way.