HTTPWebRequest Hanging

C

calpha

Newcomer
Joined
Nov 13, 2003
Messages
7
I've got a page scraper running----however, i run the pages to scrape from a database, and the scraping works for the first two lookups (one to login and get the auth cookie, and the second to scrape the particular page).

However, from the third try on, my httpwebrequest times out. I assume there's something not being closed, but I am really not sure.

here's my code
Visual Basic: 
Function GetRequestStream(ByVal sURL As String) As String
        Dim objRequest As HttpWebRequest = WebRequest.Create(sURL)
        Dim objResponse As HttpWebResponse
        Dim myWriter As StreamWriter
        Dim sr As StreamReader
        Dim objCookie As CookieContainer
        Dim strRead As String
        With objRequest
            .Method = "PROPFIND"
            .CookieContainer = Me.ckCont
            Try
                myWriter = New StreamWriter(.GetRequestStream())
            Catch ex As Exception
                RaiseEvent ErrorMessage(ex)
                Return Nothing
            End Try
        End With

        Try
            objResponse = objRequest.GetResponse()
            sr = New StreamReader(objResponse.GetResponseStream())
            strRead = sr.ReadToEnd()
            sr.Close()
            objResponse.Close()
            Return strRead

        Catch ex As Exception
            RaiseEvent ErrorMessage(ex)
            Return Nothing
        End Try

    End Function
obviously I'm just returning a string----but this function is called on every iteration through my loop.
 
while its great if that works, why is that the solution, I would really appreciate a technical explanation, there are very tangible benefits to keepalive=true, and I cannot blindly turn it off to fix a problem

(hehe, is it clear I'm having a similar problem) :)
 
Yah, this post is a bit old----
I got it working, but I don't understand why. It has to do with explicitlly closing the http response streams.

This code works without fault, and does not time out or hang.

Code: 
Function GetRequestStream(ByVal sURL As String) As String
        Dim objRequest As HttpWebRequest = WebRequest.Create(sURL)
        Dim objResponse As HttpWebResponse
        Dim myWriter As StreamWriter
        Dim sr As StreamReader
        Dim objCookie As CookieContainer
        Dim strRead As String
        With objRequest
            .Method = "PROPFIND"
            .CookieContainer = Me.ckCont
            Dim RequestStream As Stream
            Try
                RequestStream = .GetRequestStream()
                myWriter = New StreamWriter(RequestStream)
            Catch ex As Exception
                RaiseEvent ErrorMessage(ex)
                Return Nothing
            Finally
                myWriter.Close()
                RequestStream.Close()

            End Try
        End With

        Try
            objResponse = objRequest.GetResponse()
            sr = New StreamReader(objResponse.GetResponseStream())
            strRead = sr.ReadToEnd()
            sr.Close()
            objResponse.Close()
            Return strRead


        Catch ex As Exception
            RaiseEvent ErrorMessage(ex)
            Return Nothing
        End Try
 
thanks for the reply, yes it was a while ago.

I just found the problem I was having, I had to turn sendchunked to false. now I have no problems, I was already closing the response objects when using them. I have keepalive true, and unsafeconnectionsharing (not exact name) true.

me->server: POST request
me->server: post body
server->me: 200 OK
server->me: 400 Bad Request

it was issuing two responses for 1 request, dont know whats up with that. maybe an IIS bug??

it was happenning on random things too, like the 4th request in the "keepalive" connection, or the 6th. but always after a binary post payload.

in any case, thanks for your response
 
css145hs said:
while its great if that works, why is that the solution, I would really appreciate a technical explanation, there are very tangible benefits to keepalive=true, and I cannot blindly turn it off to fix a problem

(hehe, is it clear I'm having a similar problem) :)
Click to expand...

it's note THE solution, but it was A soultion for a similar problem I had while retrieveing data using the WebClient class. WebClient sends KeepAlives = true and there is no way to modify this and occasionaly, I would receive the "The operation has timed-out." exception.
 
Back
Top