HttpWebRequest timing out?

[krinpit]

Hi,

 

I need to read and parse a webpage to obtain certain information from it on a daily basis, in a VB.NET environment. I can successfully initialise the request and can receive some information from the webpage in question, but it's as if it sometimes times out and only returns to me what I managed to obtain from the stream while it was alive. Sometimes I manage to get only around 500 of the lines into my arraylist at the end. Please see my code and see if I'm missing something...

 

       Dim arrBuff As New ArrayList() ' Results go in here
       Dim myHttpWebRequest As HttpWebRequest
       Dim myHttpWebResponse As HttpWebResponse
       Dim receiveStream As Stream
       Dim encode As Encoding
       Dim sr As IO.StreamReader

       Try
           myHttpWebRequest = CType(WebRequest.Create(strURL), HttpWebRequest)
           myHttpWebResponse = CType(myHttpWebRequest.GetResponse(), HttpWebResponse)
           receiveStream = myHttpWebResponse.GetResponseStream()
           encode = System.Text.Encoding.GetEncoding("utf-8")
           sr = New StreamReader(receiveStream, encode)

           Do Until sr.Peek = -1
               strLine = sr.ReadLine
               arrBuff.Add(strLine)
           Loop
       Catch ex As System.Net.WebException
           MsgBox(ex.Message)
       Finally
           sr.Close()
           myHttpWebResponse.Close()
       End Try

[krinpit]

Well, the good news is that I've solved my problem. (I've posted the new code below).

 

The bad news is, I had to use an ugly work around, and it doesn't diminish my curiousity as to why the previous method didn't work. :confused:

 

The URL in question is:

 

http://www.msci.com/eqpages/body_DM.S.8.html

 

   Public Function readWebPage(ByVal strURL As String) As ArrayList

       Dim arrBuff As ArrayList
       Dim sr As IO.StreamReader

       Dim hwRequest As HttpWebRequest
       Dim hwResponse As HttpWebResponse
       Dim receiveStream As Stream

       Try
           hwRequest = CType(WebRequest.Create(strURL), HttpWebRequest)
           hwRequest.Timeout = 999999999
           hwResponse = CType(hwRequest.GetResponse(), HttpWebResponse)
           receiveStream = hwResponse.GetResponseStream()
           sr = New StreamReader(receiveStream)

           Dim sLongStr As String = sr.ReadToEnd() 'Read stream into a string, all in one go
           arrBuff = MultiLineStringToArrayList(sLongStr)
       Catch ex As System.Net.WebException
           MsgBox(ex.Message)
       Catch ex As Exception
           MsgBox(ex.Message)
       Finally
           If Not (sr Is Nothing) Then sr.Close()
           If Not (hwResponse Is Nothing) Then hwResponse.Close()
           hwRequest = Nothing
       End Try

       Return arrBuff
   End Function

   Private Function MultiLineStringToArrayList(ByVal sLongStr As String) As ArrayList
       '~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
       'This Function divides a string into a new string for every line
       '~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
       sLongStr = sLongStr.Replace(vbCrLf, vbLf) ' Replace any Carriage returns with a normal LineFeed
       Return New ArrayList(sLongStr.Split(vbLf))  ' Array with a line per element
   End Function