WebRequest/POST question on extracting specific data from a website - post

Question

I am trying to gather data on specific events from a company website: http://pipeline.kindermorgan.com/infoposting/notices.aspx?type=CRIT
I have worked a lot of similar websites but so far they have been pretty simple and it’s just a matter of going to the website and working with the response stream. In this case the website requires you to select a value from the first combo box (TSP/TSP Name). Without any info being passed, the URL will return the data associated with the first item in the list. I really need to be able to get the data associated with any of the items in the list.
Here is the code I have been using thus far but it fails with a Server Error 500 so I am guessing that either I did not form the POST properly or am missing some data in the post data):
For the page I have listed above I just want to get a response stream with the table of the notices for a particular TSP from the combo box (starting with Trailblazer). I know the control is “ctl00$ContentPlaceHolder1$ddlpipeline” and the value I want to send is 24. When I navigate via IE, I also have to press the “Retrieve” button.
When I look at the POST request using FireBug, I notice that there are a lot of other target/values included. I’m not sure if I need to send all of those as well and (having never done a POST before) I am not sure how to format the data in the POST to do that.
Bear with me if this request seems odd. I am more of a database person and am looking to automate a lot of the stuff we are required to look at manually every day. Any help would be greatly appreciated!
var encoding = new ASCIIEncoding();
var postData = "ctl00$ContentPlaceHolder1$ddlpipeline=24";
byte[] data = encoding.GetBytes(postData);
string RemoteURI = "http://pipeline.kindermorgan.com/infoposting/notices.aspx?type=CRIT";
var myRequest = (HttpWebRequest)WebRequest.Create(RemoteURI);
myRequest.Method = "POST";
myRequest.ContentType = "application/x-www-form-urlencoded";
myRequest.ContentLength = data.Length;
var newStream = myRequest.GetRequestStream();
newStream.Write(data, 0, data.Length);
newStream.Close();
var response = myRequest.GetResponse();
var responseStream = response.GetResponseStream();
var responseReader = new StreamReader(responseStream);

Answer 1

I actually resolved the issue and there were a number of things I found out in the process that I will share for the benefit of others who may look at this thread.
First, I had to build the POST data exactly as it appears in the POST in a browser (I used Firebug to see the POST data). This meant getting the hidden arguments as well (particularly VIEWSTATE and EVENTVALIDATION). I was able to get these by just downloading the default page source for the page (by the way, i do this in the code because it is not static for this site) and parsing out the values for the hidden fields. I then build the POST data string with any changes I may have (in my case changing the date was important but in the future I may change other things).
Now the thing that really had me stumped. I confirmed that the POST data string was exactly the same as the one sent by FireFox/FireBug through a character by character comparison and it still wouldn't work. I then remembered in a previous scraping case that I had to set the user agent.
So here is the code I ended up with:
string postData = String.Format("__EVENTTARGET=&__EVENTARGUMENT=&__LASTFOCUS="
+ "&__VIEWSTATE={0}"
+ "&ctl00%24UltraWebTree1={1}"
+ "&ctl00%24ContentPlaceHolder1%24ddlNoticeCategory={2}"
+ "&ctl00%24ContentPlaceHolder1%24ddlpipeline={3}"
+ "&ctl00%24ContentPlaceHolder1%24Button1={4}"
+ "&ctl00%24ContentPlaceHolder1%24tbDate={5}"
+ "&ctl00%24ContentPlaceHolder1%24ddlNoticeType={6}"
+ "&ctl00%24ContentPlaceHolder1%24tbSubject={7}"
+ "&ctl00%24ContentPlaceHolder1%24ddlNoticeSubType={8}"
+ "&ctl00%24ContentPlaceHolder1%24ddlOrderBy={9}"
+ "&ctl00%24ContentPlaceHolder1%24hfmode={10}"
+ "&ctl00%24ContentPlaceHolder1%24hfODSCommand={11}&ctl00%24hfPipeline={12}"
+ "&__PREVIOUSPAGE={13}&__EVENTVALIDATION={14}",
viewstate, webtree, noticecategory, pplcode,
button1, todaydate, noticetype, subject,
noticesubtype, orderby, hfmode, hfODSCommand,
hfPipeline, previouspage, eventvalidation);
var encoding = new ASCIIEncoding();
byte[] data = encoding.GetBytes(postData);
var myRequest = (HttpWebRequest)WebRequest.Create(RemoteURI);
myRequest.UserAgent = "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)" ;
myRequest.Method = "POST" ;
myRequest.ContentType = "application/x-www-form-urlencoded";
myRequest.ContentLength = data.Length;
var newStream = myRequest.GetRequestStream();
newStream.Write(data, 0, data.Length);
newStream.Close();
var myresponse = myRequest.GetResponse();
var responseStream = myresponse.GetResponseStream();
var responseReader = new StreamReader(responseStream);
string webpagesource = responseReader.ReadToEnd();
Hope this helps someone else.