Search through a text file

[chris58950]

I need to program a way to search through a log (text) file for a specific word, should I load the file into a string variable, then use the InStr method to search for a specific word? This file can be very large at times (2-30 MB). Is there a limitation on how much data can fit into a string? What would be the best way to do this? Thanks for your help.

[Volte]

You could either

 

1) Open the file (with the StreamReader) and use the StreamReader's ReadToEnd() method to put it into a String. Then use the String's IndexOf() function to find the substring.

 

or

 

2) Open the file (StreamReader again) and read each line individually, using the StreamReader.ReadLine() method inside a loop. When you read in one line, search the one line for the substring (using IndexOf, as in the first point). If it's there, you can stop the loop; if it's not, you keep looping. If you get to the end of the file, then you know the substring is not part of the file.

[Derek Stone]

I'd read in far more than one line at a time if the file even had a chance of exceeding a few hundred kilobytes, which you said it will. In other words I wouldn't use ReadToEnd() or ReadLine(), since they are slower than frozen molasses. Additionally, using ReadToEnd() (or ReadLine() on a file with few or no carriage returns/line feeds) would cause a huge memory allocation to take place, spiking a very small application up to 150MB+ if a 30MB file was loaded.

 

The following is an alternative that searches an entire 60MB file in under a second (in my tests):

 

Dim bufferSize As Integer = 102400
Dim file As StreamReader = New StreamReader(New FileStream("D:\Desktop\Log.txt", _
   FileMode.Open, FileAccess.Read, FileShare.Read), _
   System.Text.Encoding.Unicode, False, bufferSize)
Dim wordToFind As String = "Pencil"
Dim buffer(bufferSize * 2) As Char, counter As Integer

While file.Peek >= 0
   counter += 1
   file.Read(buffer, bufferSize, bufferSize)

   Dim content As String = New String(buffer)

   Dim location As Integer = content.IndexOf(wordToFind, bufferSize - wordToFind.Length)

   If location > 0 Then
       'MessageBox.Show("Text found at: " _
           '& ((counter - 1) * bufferSize + (location - bufferSize)).ToString(), _
           'Application.ProductName)
       'Exit While
   End If

   Array.Copy(buffer, bufferSize, buffer, 0, bufferSize)
End While

file.Close()

[chris58950]

Searcg Through a Text File

 

Hi Derek,

 

I modified your code slightly for my needs, but I'm having a problem. When the code gets to the IF statement for checking if the location > 0, the location variable has a value of &HFFFFFFFFFFF. I'm not sure how this can happen if the data type of this variable is integer. I thought it would throw an exception (unless this is a reference type). In any case, it doesn't work. Please help. Thank You

 

----------------------

 

If file.Exists(strLogPath & strFileName) Then

Try

 

Dim file As StreamReader = New StreamReader(New FileStream(strLogPath & strFileName, _

FileMode.Open, FileAccess.Read, FileShare.Read), _

System.Text.Encoding.Unicode, False, bufferSize)

Dim wordToFind As String = "FAIL"

Dim buffer(bufferSize * 2) As Char, counter As Integer

 

While file.Peek >= 0

counter += 1

file.Read(buffer, bufferSize, bufferSize)

 

Dim content As String = New String(buffer)

 

Dim location As Integer = content.IndexOf(wordToFind, bufferSize - wordToFind.Length)

 

If location > 0 Then

'MessageBox.Show("Text found at: " _

' & ((counter - 1) * bufferSize + (location - bufferSize)).ToString(), _

'Application.ProductName)

'Exit While

Return "Errors Encountered"

Exit While

ElseIf location = 0 Then

Return "No Errors Found"

End If

 

Array.Copy(buffer, bufferSize, buffer, 0, bufferSize)

End While

 

file.Close()

 

Catch ex As Exception

MsgBox("Could not search through log file. This is what went wrong: " & ex.GetBaseException.Message)

End Try

 

End If