HTTP POSTs and HTTP GETs with WebClient and C# and Faking a PostBack
A fellow emailed me wanting to screen scrape, er, ah, harvest a page that only displays the data he wants with a postback.
Remember what an HTTP GET looks like under the covers:
GET /whatever/page.aspx?param1=value¶m2=value
Note that the GET includes no HTTP Body. That's important. With a POST the 'DATA' moves from the QueryString into the HTTP Body, but you can still have stuff in the QueryString.
POST /whatever/page.aspx?optionalotherparam1=value
Content-Type: application/x-www-form-urlencoded
Content-Length: 25
param1=value¶m2=value
Note the Content-Type header and the Content-Length, those are important.
A POST is just the verb for when you have an HTTP document. A GET implies you got nothing.
So, in C#, here's a GET:
public static string HttpGet(string URI)
{
System.Net.WebRequest req = System.Net.WebRequest.Create(URI);
req.Proxy = new System.Net.WebProxy(ProxyString, true); //true means no proxy
System.Net.WebResponse resp = req.GetResponse();
System.IO.StreamReader sr = new System.IO.StreamReader(resp.GetResponseStream());
return sr.ReadToEnd().Trim();
}
Here's a POST:
public static string HttpPost(string URI, string Parameters)
{
System.Net.WebRequest req = System.Net.WebRequest.Create(URI);
req.Proxy = new System.Net.WebProxy(ProxyString, true);
//Add these, as we're doing a POST
req.ContentType = "application/x-www-form-urlencoded";
req.Method = "POST";
//We need to count how many bytes we're sending. Post'ed Faked Forms should be name=value&
byte [] bytes = System.Text.Encoding.ASCII.GetBytes(Parameters);
req.ContentLength = bytes.Length;
System.IO.Stream os = req.GetRequestStream ();
os.Write (bytes, 0, bytes.Length); //Push it out there
os.Close ();
System.Net.WebResponse resp = req.GetResponse();
if (resp== null) return null;
System.IO.StreamReader sr = new System.IO.StreamReader(resp.GetResponseStream());
return sr.ReadToEnd().Trim();
}
I could and should have put in more 'using' statements, but you get the gist. And, there are other ways to have done this with the BCL, but this is one.
Now, how would you fake an HTTP PostBack? Use a tool like ieHttpHeaders to watch what a real postback looks like, and well, fake it. :) Just hope they don't require unique/encrypted ViewState (via ViewStateUserKey or EnableViewStateMac) for that page, or you're out of luck.
About Scott
Scott Hanselman is a former professor, former Chief Architect in finance, now speaker, consultant, father, diabetic, and Microsoft employee. He is a failed stand-up comic, a cornrower, and a book author.
About Newsletter
First thank you very much for this piece of code :)
I would like to inform you System.Net.WebResponse has a Close() method, I think it should be better if you call it no?
Comments are closed.
There's always http://www.fiddlertool.com/ which personally I find more flexible.