I'm making a program that attempts to POST form data to the All Music Guide site, get the server's HTML response, and then parse it to extract various bits of information. I think I understand what I have to do, but I'm having trouble getting it to work.
Basically, I use a HttpWebRequest to connect to the AMG site, POST the data, and then get the response stream and read in the response. However, I randomly get one of two errors - first, I sometimes get a WebException from HttpWebRequest's CheckFinalStatus() method, with message "The underlying connection was closed - an unexpected error occured on a receive".
I've searched the net for this particular exception, and the two solutions I've seen are modifying the KeepAlive property (setting it to false), and changing the .Net configuration file "machine.config" to not use the default proxy. However, neither of these suggestions seems to have helped. In fact, changing the machine.config file to not use the system default proxy settings makes the program fail 100% of the time (my IE connection settings do not have the box "Use a proxy server" ticked).
Another problem, which occurs less frequently, is that I sometimes get back a HTML page from the server looking something like this:
<HTML><HEAD><META HTTP-EQUIV="Refresh" CONTENT=0; URL=http://www.allmusic.com/cg/amg.dll></HEAD><BODY></BODY></HTML>
The URL is the DLL to which I'm sending the form data. So does this mean that I'm supposed to repost the data? I understand that on a browser this would attempt to immediately refresh to the URL given, but does this mean that the browser must repost the data? I'm not sure whether this is standard behaviour or whether there's something wrong (because I would imagine that the server would choose to wait until it gave me proper data, instead of making me redo my query).
Here's what my code looks like:
HttpWebRequest webRequest = null;
webRequest = (HttpWebRequest) WebRequest.Create (AMG_QUALIFIED_DLL);
webRequest.KeepAlive = false;
webRequest.Timeout = 100000;
webRequest.Method = "POST";
webRequest.ContentType = "application/x-www-form-urlencoded";
// Convert the postData into a byte array
// postData is an instance variable
byte[] requestBytes = (new System.Text.ASCIIEncoding()).GetBytes (postData);
webRequest.ContentLength = requestBytes.Length;
// Write the bytes to the request stream
Stream requestStream = webRequest.GetRequestStream();
requestStream.Write (requestBytes, 0, requestBytes.Length);
requestStream.Close();
HttpWebResponse webResponse = null;
StreamReader sr = null;
try
{
webResponse = (HttpWebResponse) webRequest.GetResponse();
sr = new StreamReader (webResponse.GetResponseStream(), System.Text.Encoding.ASCII);
response = sr.ReadToEnd ();
}
catch (WebException ex)
{
Console.WriteLine (ex.Message + ex.StackTrace);
throw new WebException (ex.Message, ex.InnerException, ex.Status, ex.Response);
}
finally
{
if (sr != null)
sr.Close();
if (webResponse != null)
webResponse.Close ();
sr = null;
webResponse = null;
}
Now, what really puzzles me is this - if I use the ActiveX WebBrowser control to POST the data, it seems to work fine 100% of the time! It would appear then that there is something wrong with what I'm doing, but I'm not sure what.
Here's the WebBrowser code (it's just a slightly modified version of the MS sample code) :
object oEmpty = "";
string cPostData = "p=amg&sql="+artistNameBox.Text+"&opt1=1";
object vHeaders = "Content-Type: application/x-www-form-urlencoded" + "\n" + "\r";
object vPost = System.Text.ASCIIEncoding.ASCII.GetBytes(cPostData);
this.axWebBrowser1.Navigate ("http://www.allmusic.com/cg/amg.dll", ref oEmpty, ref oEmpty, ref vPost, ref vHeaders);
I though that perhaps this had to do with the UserAgent property of the request, and so I tried setting it to the string for IE 6 (I can't remember the string, something like "Mozilla 4.0; compatible IE6.0"?). However, that didn't seem to change things either.
Any help would be much appreciated!
