Reading large textfiles fast

[themster]

I want to read textfiles as large as 100000 lines and I want it go as fast as possible.

 

I have tried using StreamReader.ReadLine and that works fine for textfiles with 20000 lines which takes approx. 3-4 sec.

 

When I use the same code on a textfile with 40000 lines it doesn't take 6-8 sec but 20-25 sec.

 

Why isn't the time proportional to the number of lines and is there any other faster way to read files?

[divil]

It depends if you really want to read it line by line. The fastest way will be to allocate a buffer of the right size to begin with, and just fill it with the contents, reading large chunks at once.

 

If you really need it line by line you could then split this data on newlines, it would probably end up faster than reading the file line by line in the first place. You'll be looking at less than a second probably.

[themster]

I found the following code but the buffer doesn't return anything.

 

Dim myStreamReader As StreamReader = New StreamReader( _
   New FileStream("C:\somefile.txt", FileMode.Open), _
   System.Text.Encoding.Unicode, False, 4096)
Dim cBuffer() As Char

ReDim cBuffer(4096)

Do While myStreamReader.ReadBlock(cBuffer, 0, cBuffer.Length - 1) > 0
   'Buffer filled and available for use
Loop

myStreamReader.Close() 

[vbMarkO]

themster,

 

Did you ever find what you were needing to allow you to read large textfiles.

 

I am needing the same thing, in fact I came accross the same code and wrote the author of it but he hasnt responded to me as of yet.

 

Because the buffer didnt return anything but jibberish for me.... so I figure I was accessing it wrong or something.

 

Just the same, I have a textfile I need to loop through but a I need a faster way of doing it.

 

I had a pretty good sugestion or at least it sounded good.

 

I was told I should load the textfile intoa dataset then get the info I need out of the dataset, they said it would be much faster.

 

However, all this did was send me on yet another search for how to do it. As all they gave was a sugestion and not an example how.

 

I would be interested in knowing what solution you came up with

 

vbMarkO