Performance question VB .NET /C/C++/VB6/C#

[vokuit00]

Hi,

 

i have a general question about a project that has just begun. I should read lines out of a file. A line (in ASCII format) looks like this:

 

timestamp(Double) channel(Byte) ID(Integer) mode(String with 2 characters) lenght(Byte) data(max. 8 Byte)

 

I should read in this lines, parse and visualize them.

 

My first trials with a file of 60.000 lines were a bit disillusioning. I tried to visualize them in a listbox-control and it needs 10 min. to load and visualize them. The next trial with a listview-control was about one minute. But yesterday i got the information that the maximum file size was 1,500,000 lines.

 

Now my naive question is, how to handle such large amount of data in VB .NET? Or should i use C/C++ for a better performance?

 

My first thougts were to implement a class in VB .NET in which i "fill" in the data after parsing the lines with RegEx. After that i could put this class in an Array or a Collection or a database???

 

I must visualize this data on three kinds. First as a kind of treeview, second in an ActiveX-component like an oszilloskope and third in a table. The user must manipulate some parts of this data.

 

I hope you can excuse my bad English and that you can give me some recommendation for working on this large amount of data.

 

Many Thanks

 

Volker

[Denaes]

Post the code you're using to read the ASCII data.

 

.Net provides some easy ways to access ASCII, which by default are slower. So unless you've looked into better ASCII performance - you're not getting it. It doesn't help that most VB users are still able to use the old VB6 function which is really slow.

 

Post your code and someone can advise you on how to kick it up a notch.

[vokuit00]

Some kind of code

 

Hi,

 

this morning i tried some tests:

 

Dim start As Integer

Dim ende As Integer

Dim anzahl As Integer

Dim tmpString As String

start = System.Environment.TickCount()

ListView1.BeginUpdate()

While (readstream.Peek() <> -1)

tmpString = readstream.ReadLine()

ListView1.Items.Add(tmpString)

anzahl += 1

End While

ListView1.EndUpdate()

ende = System.Environment.TickCount()

MessageBox.Show("Es dauerte: " & CType(ende - start, String) & " für " & anzahl.ToString & " Zeilen")

 

With 68000 lines and the usage of a listview it needs 11 sec. With a listbox it need 137 sec.

 

Greetings

 

Volker

[Denaes]

Don't have VB.Net available on this machine, but I can see one area or two to trim some fat.

 

Unless you post the code with your connection ("readstream") people cannot look into helping with that.

 

 

As it is, you can remove the string tmpString. Unless you cut out code where it does something else - this is a bottleneck. Not saying it's your problem, but Strings are fairly slow and you're copying it an extra time here.

 

Dim start As Integer
     Dim ende As Integer
     Dim anzahl As Integer
     'Dim tmpString As String
     start = System.Environment.TickCount()
     ListView1.BeginUpdate()
     While (readstream.Peek() <> -1)
        'tmpString = readstream.ReadLine()
        'ListView1.Items.Add(tmpString)
ListView1.Items.Add(readstream.ReadLine())
        anzahl += 1
     End While
     ListView1.EndUpdate()
     ende = System.Environment.TickCount()
     MessageBox.Show("Es dauerte: " & CType(ende - start, String) & " für " & anzahl.ToString & " Zeilen")

 

 

I'm honestly not sure if this is slowing things down:

 

While (readstream.Peek() <> -1)

 

I used to use something along the lines of:

 

While (not readstream.EoF())

 

Again, I don't have VS open until I get home tonite (probobly another 10 hours) so I can't really test anything - just give advice.

 

I'm not entirely sure why your listbox is taking so long to populate - what is your code for the listbox?

[vokuit00]

Answer:

 

Is use exactly the same code for the ListBox. Only replace ListView1 with ListBox1.

 

Thanks

 

Volker

[penfold69]

If the ascii data is in a relatively simple format of CSV or Tab Delimited, then you may be able to make use of the ODBC driver, and fill a datatable with it.

 

I doubt if it will actually result in much of an ACTUAL improved speed, but it might make it more manageable to thread it, so that it is perceived to be faster.

 

I've worked on importing and exporting CSV/TAB files a bit recently, and I've found the ODBC driver is the most flexible to work with.

 

Of course, if your data is in true binary form, then it probably wont work for you.

 

B.

[Nerseus]

Does the listbox have sorting turned on? That might account for its extreme slowness. If you're retrieving from a database, put the "ORDER BY" there and turn sorting off in your listbox/combobox to save time.

 

I can't see any reason to fill a listbox with more than a few dozen entries, maybe a few hundred if really needed. Anything more and you need a different solution.

 

-ner

[Denaes]

I can't see any reason to fill a listbox with more than a few dozen entries, maybe a few hundred if really needed. Anything more and you need a different solution.

 

-ner

 

What? You don't want to scroll through 1.5 million items?

 

Correct me if I'm wrong, wouldn't DataTable(or DataSet) work better for storing and retreiving this information? Not scrollable, but it's searchable.

[Joe Mamma]

I dont understand. . .

 

you talk asciii, but then talk byte data.

is the file delimited/fixed width (ASCII)? or structured:

Double,Byte,Integer,char(2),Byte, Byte(8) - therefore binary?

 

which ever it is makes a difference as to how to approach it

[donnacha]

Can you store the info in the file as xml, then you will be able to load the data straight into a datasat in 1 go and then bind this dataset to whatever controls you want. This would be the most efficent way to do it.

[Denaes]

Can you store the info in the file as xml' date=' then you will be able to load the data straight into a datasat in 1 go and then bind this dataset to whatever controls you want. This would be the most efficent way to do it.[/quote']

 

More efficient, if you just want the data in a control, would be a datareader, loading each record.

 

If you want to keep the data in memory, then the dataset is just fine.

 

still you normally don't want to display 1.5 million records at one time.

[michael_hk]

I don't think anyone will like to have so many items in a listbox....

[Derek Stone]

XML will never be performant with the exception of binary XML (see W3C XML Binary Characterization Group). Suggesting XML as a solution to a performance problem is usually, if not always, misguided.

[donnacha]

One basic question for you, why do you not use a database instead of a text file. It might make life easier for you. Are you restricted byhow the data is put into this file.