POSTing data with HttpWebRequest - problems connecting

akm · December 12, 2003

I'm making a program that attempts to POST form data to the All Music Guide site, get the server's HTML response, and then parse it to extract various bits of information. I think I understand what I have to do, but I'm having trouble getting it to work.

Basically, I use a HttpWebRequest to connect to the AMG site, POST the data, and then get the response stream and read in the response. However, I randomly get one of two errors - first, I sometimes get a WebException from HttpWebRequest's CheckFinalStatus() method, with message "The underlying connection was closed - an unexpected error occured on a receive".

I've searched the net for this particular exception, and the two solutions I've seen are modifying the KeepAlive property (setting it to false), and changing the .Net configuration file "machine.config" to not use the default proxy. However, neither of these suggestions seems to have helped. In fact, changing the machine.config file to not use the system default proxy settings makes the program fail 100% of the time (my IE connection settings do not have the box "Use a proxy server" ticked).

Another problem, which occurs less frequently, is that I sometimes get back a HTML page from the server looking something like this:

The URL is the DLL to which I'm sending the form data. So does this mean that I'm supposed to repost the data? I understand that on a browser this would attempt to immediately refresh to the URL given, but does this mean that the browser must repost the data? I'm not sure whether this is standard behaviour or whether there's something wrong (because I would imagine that the server would choose to wait until it gave me proper data, instead of making me redo my query).

Here's what my code looks like:

HttpWebRequest webRequest = null;

webRequest = (HttpWebRequest) WebRequest.Create (AMG_QUALIFIED_DLL); 

webRequest.KeepAlive = false;
webRequest.Timeout = 100000;
webRequest.Method = "POST";
webRequest.ContentType = "application/x-www-form-urlencoded";

// Convert the postData into a byte array
// postData is an instance variable
byte[] requestBytes = (new System.Text.ASCIIEncoding()).GetBytes (postData);
webRequest.ContentLength = requestBytes.Length;

// Write the bytes to the request stream
Stream requestStream = webRequest.GetRequestStream();
requestStream.Write (requestBytes, 0, requestBytes.Length);
requestStream.Close();

HttpWebResponse webResponse = null;
StreamReader sr = null;

try
{
   webResponse = (HttpWebResponse) webRequest.GetResponse(); 

   sr = new StreamReader (webResponse.GetResponseStream(), System.Text.Encoding.ASCII);
   response = sr.ReadToEnd ();
}

catch (WebException ex)
{
   Console.WriteLine (ex.Message + ex.StackTrace);
   throw new WebException (ex.Message, ex.InnerException, ex.Status, ex.Response);
}

finally
{
   if (sr != null)
       sr.Close(); 
   if (webResponse != null)
       webResponse.Close ();

   sr = null;
   webResponse = null;
}

Now, what really puzzles me is this - if I use the ActiveX WebBrowser control to POST the data, it seems to work fine 100% of the time! It would appear then that there is something wrong with what I'm doing, but I'm not sure what.

Here's the WebBrowser code (it's just a slightly modified version of the MS sample code) :

object oEmpty = "";
string cPostData = "p=amg&sql="+artistNameBox.Text+"&opt1=1";
object vHeaders = "Content-Type: application/x-www-form-urlencoded" + "\n" + "\r";
object vPost = System.Text.ASCIIEncoding.ASCII.GetBytes(cPostData);

this.axWebBrowser1.Navigate ("http://www.allmusic.com/cg/amg.dll", ref oEmpty, ref oEmpty, ref vPost, ref vHeaders);

I though that perhaps this had to do with the UserAgent property of the request, and so I tried setting it to the string for IE 6 (I can't remember the string, something like "Mozilla 4.0; compatible IE6.0"?). However, that didn't seem to change things either.

Any help would be much appreciated!

css145hs · January 24, 2004

hey, did you ever find a resolution to your problem, I would be greatly interested if you did :)

karlhaak · February 2, 2004

connection closed on webrequest

Thank you very much for your post here.

The option with keepAlive=false works excellent for me. I was experiencing the following behavior.

I was writing a windows service that would connect to a website, post some data to get a response. Everything worked fine when I ran the component as a console Application in Debug mode on my machine . Once installed as a service every second request failed.

I had to fight with an unvalid certificate for ssl, username, password and a proxyserver which gave me plenty of options to look for.

I finally tried the keepAlive option and everything works fine. I still do not understand the influence of this parameter on the failure of the connection every second time and would be interested in any ideas.

The environment I am working in is Windows 2000 with .NET 1.0.

I created an abstract of my solution as a coding example. Hope this will safe some time for somebody else.

Karl

Imports System.IO
Imports System.Text
Imports System.Threading
Imports System.Net
Imports System.Text.Encoding
Imports System.Security.Cryptography.X509Certificates

Public Class MyCertificateValidation
   Implements System.Net.ICertificatePolicy

   'This class handles problems with certificates if ssl (https) is used

   Public Function CheckValidationResult(ByVal srvPoint As ServicePoint, _
   ByVal cert As X509Certificate, ByVal request As WebRequest, ByVal problem As Integer) _
      As Boolean Implements ICertificatePolicy.CheckValidationResult
       Return True ' Accept all certificates
   End Function
End Class

Public Class RequestState
   Public request As WebRequest = Nothing
   Public requestDocument As String
End Class

Public Class HTMLRequest

   'Handler to synchronise with main thread again
   Public allDone As New ManualResetEvent(False)

   Private _requestDocument As String = "Text to send"
   Private _responseDocument As String
   Private _web As Net.HttpWebRequest

   Private Sub ReadCallback(ByVal asynchronousResult As IAsyncResult)
       'Get the information to pass on from the Async Object
       Dim myRequestState As RequestState = CType(asynchronousResult.AsyncState, RequestState)
       Dim myWebRequest2 As WebRequest = myRequestState.request
       Dim requestDocument As String = myRequestState.requestDocument

       'Get the stream to write the request to
       Dim streamResponse As Stream = myWebRequest2.EndGetRequestStream(asynchronousResult)
       Dim bytArray() As Byte = System.Text.Encoding.ASCII.GetBytes(requestDocument)
       streamResponse.Write(bytArray, 0, requestDocument.Length)
       streamResponse.Close()

       'Tell the main thread that writing to stream is complete
       allDone.Set()
   End Sub

   Public Sub retrieve()
       Dim bytOutArray() As Byte
       Dim objWebResponse As Net.WebResponse = _web.GetResponse
       Dim objStream As IO.StreamReader
       objStream = New IO.StreamReader(objWebResponse.GetResponseStream(), System.Text.ASCIIEncoding.ASCII)
       _responseDocument = objStream.ReadToEnd()
       'close all streams
       objWebResponse.Close()
       objStream.Close()
   End Sub

   Public Sub send()
       'Prepare Object to pass into asynchron call
       Dim myRequestState As New RequestState
       myRequestState.request = _web
       myRequestState.requestDocument = _requestDocument
       Dim r As IAsyncResult = CType(_web.BeginGetRequestStream(AddressOf ReadCallback, myRequestState), IAsyncResult)

       'Wait till asychrone is done ## allDone.Set() ##
       allDone.WaitOne()
   End Sub

   Public Sub getRequest()

       'Configure connection

       'in case of problem with certificate
       ServicePointManager.CertificatePolicy = New MyCertificateValidation

       'Create request object
       _web = CType(WebRequest.Create("https://www.somedomain.com"), HttpWebRequest)
       With _web
           .ContentType = "application/x-www-form-urlencoded"
           .Method = "POST"
           .KeepAlive = False
           .ContentLength = _requestDocument.Length
           .Timeout = 10000

           'in case of proxyserver
           .Proxy = New WebProxy("xxx.proxyserver.com", 8080)

           'in case of required password
           Dim cache As CredentialCache = New CredentialCache
           cache.Add(New Uri("https://www.somedomain.com/"), "Basic", New NetworkCredential("Username", "password"))
           .Credentials = cache
       End With

       send() 'send the request
       retrieve() 'get the data
   End Sub
End Class

Edited February 3, 2004 by karlhaak

karlhaak · February 3, 2004

connection closed on webrequest Microsoft article

I finally found the Microsoft article that applies to the problem.

It can be found under the following location

http://support.microsoft.com/default.aspx?scid=kb%3Ben-us%3B819450

There is also some good advice in

Jan Tielens Blog

Hope this will help someone who stumbles over the same problem.

Karl

aric · July 29, 2004

I have hit the same problem...

I am trying to implement a financial gateway system, which uses WSE 1.0 SP1, and the sample code provided makes an SSL connection to their server. So, I have to find a way to override the System.Net.ICertificatePolicy so as to always approve SSL certs.

I am running ASP.NET (VB) .NET version 1.1, and am having a hard time getting your code to work, and was wondering if you had any suggestions.

I put your class above my class (in a code-behind page which inherits the page) thusly:

Public Class MyCertificateValidation
   Implements System.Net.ICertificatePolicy

   'This class handles problems with certificates if ssl (https) is used

   Public Function CheckValidationResult(ByVal srvPoint As ServicePoint, _
   ByVal cert As X509Certificate, ByVal request As WebRequest, ByVal problem As Integer) _
      As Boolean Implements ICertificatePolicy.CheckValidationResult
       Return True ' Accept all certificates
   End Function
End Class

Public Class thepage

   ' Inherit the page
   Inherits Page
.
.
.

Everything looks good, but I get this error:

Compilation Error

Description: An error occurred during the compilation of a resource required to service this request. Please review the following specific error details and modify your source code appropriately.

Compiler Error Message: BC30149: 'MyCertificateValidation' must implement 'Overridable Function CheckValidationResult(srvPoint As ServicePoint, certificate As Security.Cryptography.X509Certificates.X509Certificate, request As WebRequest, certificateProblem As Integer) As Boolean' for interface 'System.Net.ICertificatePolicy'.

Source Error:

Line 26: 
Line 27: Public Class MyCertificateValidation
Line 28:     Implements System.Net.ICertificatePolicy
Line 29: 
Line 30:     'This class handles problems with certificates if ssl (https) is used

However, it clearly DOES overright it... and I am stumped. :confused:

Any ideas anyone?

Thanks in advance.

karlhaak · July 30, 2004

Quick suggestions for: Compiler Error Message: BC30149

Hello Aric,

I cannot see any problem at first glance. Here are some quick things I would like to check.

1) If you programm with visual studio, it will mark the class as invalid if not all declared interfaces have been implemented. Can you check that.

2) Put the class in an extra file.

3) check if the namespace

Imports System.Security.Cryptography.X509Certificates

has been imported.

4) check if the namespace

Imports System.Net

has been imported.

hope one of these will help

Karl

mary92028 · January 23, 2008

Karl - you and Jan helped me with my SSL problem in VB.net. MUCH APPRECIATED.

Mary

Sign In

POSTing data with HttpWebRequest - problems connecting

Recommended Posts

akm

css145hs

karlhaak

karlhaak

aric

karlhaak

mary92028

Join the conversation

Browse

Activity