Extract HTML from a redirected page

[HD666]

Hello,

 

I am using visual basic 2005. I found on the web the following function that extracts HTML from webpages. It is very useful but unfortunately it does not work with redirected pages. That is, when I put in it a URL of a redirect page it gives me nothing or error. I added to it ".AllowAutoRedirect = True" but still it did not work. I wonder how to make it work for redirected pages.

 

I appreciate the help.

Public Function GetPageHTML(ByVal URL As String, _
     Optional ByVal TimeoutSeconds As Integer = 10) _
    As String
       ' Retrieves the HTML from the specified URL,
       ' using a default timeout of 10 seconds
       Dim objRequest As Net.HttpWebRequest
       Dim objResponse As Net.HttpWebResponse
       Dim objStreamReceive As System.IO.Stream
       Dim objEncoding As System.Text.Encoding
       Dim objStreamRead As System.IO.StreamReader

       Try
           ' Setup our Web request
           objRequest = Net.WebRequest.Create(URL)
           objRequest.Method = "GET"
           objRequest.KeepAlive = True
           objRequest.AllowAutoRedirect = True
           objRequest.Timeout = TimeoutSeconds * 1000
           ' Retrieve data from request
           objResponse = objRequest.GetResponse()
           objStreamReceive = objResponse.GetResponseStream
           objEncoding = System.Text.Encoding.GetEncoding( _
               "utf-8")
           objStreamRead = New System.IO.StreamReader( _
               objStreamReceive, objEncoding)
           ' Set function return value
           GetPageHTML = objStreamRead.ReadToEnd()
           ' Check if available, then close response
           If Not objResponse Is Nothing Then
               objResponse.Close()
           End If
       Catch
          Return "error"
       End Try
   End Function

[haydenw]

Hi,

Pages that redirects sends a http header back to the client showing the location of the page being redirected to. Using VB 2008 i found the following code will give you this location:

 

objresponse.ResponseUri.AbsoluteUri

 

You Could then do the following:

 

If not url=objresponse.ResponseUri.AbsoluteUri then
return GetPageHTML(objresponse.ResponseUri.AbsoluteUri)
End IF

 

Hope this helps.