awyeah

Thank you for the valueable help. Will try it out and let you know. :)

I have included some explanations and reduced the code to see what I am doing. Then maybe you can have an idea of how I can fire multiple threads using this type of mechanism. 'Go through each in file list For Each curFile As String In rmr_files 'Open file and read all data Dim RmrData() As String = IO.File.ReadAllLines(curFile) 'For each line in file, read data and process For Each curLine As String In RmrData 'Do some data processing here ......................................... ......................................... 'Write processed data to a new text file - USE APPEND IO.File.AppendAllText(app_dir & "\" & customer_name & ".txt", writetofile) 'Move to the next line in the file till some criteria is met Next curLine 'Move to the next file in the list Next curFile End Sub Hope this helps. I have already included the Option Explicit for the data type. Just didn't paste it since the code was getting too long. Here it is now, the rest of it. Option Strict Off Option Explicit On Public Class Form1 Dim app_dir As String 'Location to application directory Dim tempfiles_dir As String 'Location to \TempFiles directory Dim customer_id_file As String 'Location to file \customerids.txt Dim rmr_file_list As String 'Location to rmrfiles.txt file Dim rmr_files() As String 'Array containing directories and rmr data file names Dim count_rmrfiles As Long 'Number of rmr data files Dim current_rmrfile As Integer 'Current file being read Dim customerids As Collections.Generic.List(Of String) Dim idFileName As String = customer_id_file Dim words() As String Dim countchar1 As String Dim countchar2 As String Private Sub Form1_Load(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles MyBase.Load Dim th As New System.Threading.Thread(AddressOf ReadRMRDataFileIntoTextFiles) 'Set directories and files app_dir = Application.StartupPath tempfiles_dir = app_dir & "\TempFiles\" rmr_file_list = app_dir & "\TempFiles\loadprofilefiles.txt" customer_id_file = app_dir & "\customerids.txt" 'Read file names from a text file into an array.. this only takes a sec or so.. Call ReadLoadProfileFilesIntoArray() ProgressBar1.Minimum = 0 ProgressBar1.Maximum = count_rmrfiles ProgressBar1.Value = 0 'Start reading rmr data files th.Start() End Sub Private Sub UpdateProgressBar() If Me.InvokeRequired Then Me.Invoke(New MethodInvoker(AddressOf UpdateProgressBar)) Else ProgressBar1.Value = current_rmrfile End If End Sub As for the CountOccurrences sub I found it somewhere on the Internet, done by someone. I also found RegExp doing the same thing, however I found it regular expression matching is slower.

Dear all, I have created a code in VB.NET to read data from text files. Data is read from a list of files, where each file is changed using a for loop. Data is written to a new text files. Each file is read one by one and written in the same way. Now, my speed of execution is very slow. I am using a Quadcore processor with only 20-30% of CPU utilization when my code runs. Is there anyway I can increase the speed of reading and writing? To read only 125 files it takes 10 minutes or more, which is very slow indeed, because in the end I need to read thousands of files and write them. Each file is approxiamately 30-50kb. Here is my code. Public Sub ReadRMRDataFileIntoTextFiles() 'Read in the customerids once up front Dim customerids As Collections.Generic.List(Of String) Dim idFileName As String = customer_id_file If IO.File.Exists(idFileName) Then customerids = IO.File.ReadAllLines(idFileName).ToList() Else customerids = New Collections.Generic.List(Of String)() End If 'now process files current_rmrfile = 0 For Each curFile As String In rmr_files Dim customer_name As String customer_name = "" 'Open rmr data file Dim RmrData() As String = IO.File.ReadAllLines(curFile) For Each curLine As String In RmrData 'RemoveEmptyEntires option takes care of Pack() and Trim() 'If line has proper data inside to be read If InStr(curLine, "METER") > 0 Then If InStr(curLine, ":") > 0 Then Dim newcurLine() As String = curLine.Replace(" ", "").Split(":") customer_name = Trim(newcurLine(1)) 'If customer already added in list - do not add 'Else if customer not added - add into list If Not customerids.Contains(customer_name) Then customerids.Add(customer_name) IO.File.AppendAllText(idFileName, customer_name & vbCrLf) End If ElseIf InStr(curLine, "=") > 0 Then Dim newcurLine() As String = curLine.Replace(" ", "").Split("=") customer_name = Trim(newcurLine(1)) 'If customer already added in list - do not add 'Else if customer not added - add into list If Not customerids.Contains(customer_name) Then customerids.Add(customer_name) IO.File.AppendAllText(idFileName, customer_name & vbCrLf) End If End If End If 'Split and Join string to apply "Trim" and "Pack" words = curLine.Trim(" ").Split(vbTab) 'Count occurences of string countchar1 = CountOccurrences(curLine, "/", False) countchar2 = CountOccurrences(curLine, ":", False) 'If data has started, then read it If countchar1 = 2 And countchar2 = 1 And words.Length >= 1 Then 'Get data from line Dim trimwords As String = String.Join(" ", words) Dim datewrite As String = trimwords.Substring(0, 10) Dim timewrite As String = trimwords.Substring(11, 5) Dim kwhwrite As String = words(1) 'Splitting date Dim day_write As String = datewrite.Substring(3, 2) Dim month_write As String = datewrite.Substring(0, 2) Dim year_write As String = datewrite.Substring(6, 4) datewrite = String.Format("{0}-{1}-{2}", day_write, month_write, year_write) ''''Time If timewrite = "24:00" Then timewrite = "00:00:00" Else timewrite = String.Format("{0}:{1}", timewrite, "00") End If Dim writetofile As String = String.Format("{0},{1},{2}", datewrite, timewrite, kwhwrite & vbCrLf) IO.File.AppendAllText(app_dir & "\" & customer_name & ".txt", writetofile) Else 'If data has not yet started, skip the initial lines Continue For End If Next curLine current_rmrfile = current_rmrfile + 1 UpdateProgressBar() Next curFile System.Threading.Thread.Sleep(3000) Me.Close() End Sub Function CountOccurrences(ByVal p_strStringToCheck, ByVal p_strSubString, ByVal p_boolCaseSensitive) Dim arrstrTemp Dim strBase, strToFind If p_boolCaseSensitive Then strBase = p_strStringToCheck strToFind = p_strSubString Else strBase = LCase(p_strStringToCheck) strToFind = LCase(p_strSubString) End If arrstrTemp = Split(strBase, strToFind) CountOccurrences = UBound(arrstrTemp) End Function One of the sample data files to read. Service Point ID=060430_00001587 AKAUN=601011 METER=28509864 DATE/TIME=01/05/2009 00:00 TO 30/06/2009 00:00 A= KWH IMPORT B= KWH EXPORT C= KVARH IMPORT D= KVARH IMPORT DATE TIME A B C D 05/01/2009 00:30 74 50 0 0 05/01/2009 01:00 77 61 0 0 05/01/2009 01:30 76 62 0 0 05/01/2009 02:00 77 60 0 0 05/01/2009 02:30 76 61 0 0 05/01/2009 03:00 76 61 0 0 05/01/2009 03:30 77 62 0 0 05/01/2009 04:00 76 61 0 0 05/01/2009 04:30 76 51 0 0 05/01/2009 05:00 73 49 0 0 05/01/2009 05:30 75 50 0 0 05/01/2009 06:00 74 50 0 0 05/01/2009 06:30 74 49 0 0 05/01/2009 07:00 75 50 0 0 05/01/2009 07:30 73 48 0 0 05/01/2009 08:00 74 50 0 0 05/01/2009 08:30 76 62 0 0 05/01/2009 09:00 72 59 0 0 05/01/2009 09:30 71 59 0 0 All help is appreciated.

Thanks, everything works well now, after I have implemented the changes you have mentioned. Just one thing.. In the "Pack" function when I Dim "words" as String rather than Object, I get the following errors and it doesn't compile: Error 1 Value of type '1-dimensional array of String' cannot be converted to 'String'. C:\Users\jawad\Desktop\WindowsApplication1\WindowsApplication1\Form1.vb 418 17 WindowsApplication1 Error 2 Value of type 'String' cannot be converted to 'System.Array'. C:\Users\jawad\Desktop\WindowsApplication1\WindowsApplication1\Form1.vb 419 24 WindowsApplication1 Error 3 Value of type 'String' cannot be converted to 'System.Array'. C:\Users\jawad\Desktop\WindowsApplication1\WindowsApplication1\Form1.vb 419 41 WindowsApplication1

Thanks for all these wonderful suggestions. I have implemented the streamwriter and am in process of doing the rest as now. The "stripout" function basically removes certain characters from a string. Similar to replace, the string to be replaced is "" (null) so nothing is replaced and occurrences of that character in the entire string are deleted. Any faster method for the "replace" or stripout function?? Replace takes one of the most longest time to execute as so I saw. Also I join (concat) strings in a very terrible way I see. Any better way to achieve this?? datewrite = "" & day_Renamed & "-" & month_Renamed & "-" & year_Renamed & "" writetofile = "" & datewrite & "," & timewrite & "," & kwhwrite & ""

Thanks you for your expert advise. I will use the streamwriter as you have mentioned. Yes I use Left(), Right() and Mid() functions in VB6, not sure how I can correctly replace those Microsoft.VisualBasic.Strings with the methods of the string class. Any tips for replacing these? I have also considered to remove "DoEvents" now, since in VB6 the GUI might not respond, but in .NET it seems to run a bit more faster now. As per your request, here are the three functions, returnContents, Pack and StripOut. 'Read all data in the text file into array Public Function returnContents(ByVal strFile As String) As String Dim filenum As Short filenum = FreeFile() FileOpen(filenum, strFile, OpenMode.Input) returnContents = InputString(1, LOF(filenum)) FileClose(filenum) End Function 'Remove extra white spaces in string Public Function Pack(ByRef str_Renamed As String) As String Dim words As Object Dim X As Integer Dim temp As String words = Split(str_Renamed, " ") For X = LBound(words) To UBound(words) If words(X) <> "" Then temp = temp & " " & words(X) End If Next X Pack = temp End Function Public Function FileExists(ByRef OrigFile As String) As Object Dim fs As Object fs = CreateObject("Scripting.FileSystemObject") FileExists = fs.FileExists(OrigFile) End Function Public Function StripOut(ByRef From As String, ByRef What As String) As String Dim i As Short StripOut = From For i = 1 To Len(What) StripOut = Replace(StripOut, Mid(What, i, 1), "") Next i End Function

Dear all, I am trying to complete a project. My task is to read large TXT files (300 files, 2.2 GB in total) read the customers inside, perform some data calculation and write each customer data to a separate txt file. I made a code in VB6, it runs fine, however it takes 8 days to run on a quadcore processor, utilizing 20-30% of the CPU. Now I upgraded the code and am running on VB.NET 2008, the latest version. The speed still remains the same, and the CPU usage is also about the same. Anyway I can make this process faster. I have a quadcore PC with 8GB of ram, possible to maybe make VB utilize all the CPU say 100%? so it executes atleast 2 to 3 times faster. Here is my code in VB.NET 2008: Option Strict Off Option Explicit On Imports VB = Microsoft.VisualBasic Public Class Form1 Inherits System.Windows.Forms.Form Dim rmr_files() As String 'Array containing directories and rmr data file names Public Sub Button1_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles Button1.Click Call ReadRMRFileNamesIntoArray 'Read RMR data files Call ReadRMRDataFileIntoTextFiles End Sub Public Sub ReadRMRDataFileIntoTextFiles() Dim str_Renamed As String Dim str2 As String Dim sJoin As String Dim row As Integer Dim customer_name As String Dim line As Integer Dim line2 As Integer Dim count As Integer Dim count2 As Integer Dim countdata1 As Integer Dim countdata2 As Integer Dim writetofile As String Dim writecustid As String Dim custrecidfile As Boolean Dim custpresent As Boolean Dim datewrite As String Dim timewrite As String Dim kwhwrite As String Dim tempdate As String Dim temptime As String Dim tempkwh As String Dim custfileexists As Boolean Dim origfilepath As String Dim i As Integer Dim sArray As Object Dim custArray As Object Dim myObjFs1 As Scripting.FileSystemObject Dim objWrite1 As Object Dim myObjFs2 As Scripting.FileSystemObject Dim objWrite2 As Object Dim col As Integer Dim tempArray As Object Dim myObjFs3 As Scripting.FileSystemObject Dim objWrite3 As Object Dim year_Renamed As String Dim month_Renamed As String Dim day_Renamed As String Dim splitrmrdata1() As String Dim splitrmrdata2() As String Dim customerids() As String For count = LBound(rmr_files) To UBound(rmr_files) 'Open rmr data file and begin to read splitrmrdata1 = Split(returnContents(rmr_files(count)), vbNewLine) row = 1 line = 1 For countdata1 = LBound(splitrmrdata1) To UBound(splitrmrdata1) str_Renamed = Trim(Pack(StripOut(splitrmrdata1(countdata1), """"))) 'Split string into separate words and characters sArray = Split(str_Renamed, " ") For i = LBound(sArray) To UBound(sArray) sArray(i) = """" & sArray(i) & """" Next 'Join back array to convert into csv format sJoin = Join(sArray, ",") If UCase(Mid(sJoin, 2, 8)) = "RECORDER" Then sJoin = Replace(sJoin, "RECORDER"",""ID", "RECORDER ID") End If 'New customer found If InStr(sJoin, "RECORDER") <> 0 Then row = 1 End If '''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''' 'Open new file 'Get the name of the customer from second line after the "RECORDER" line If row = 1 Then splitrmrdata2 = Split(returnContents(rmr_files(count)), vbNewLine) line2 = 1 For countdata2 = LBound(splitrmrdata2) To UBound(splitrmrdata2) str2 = Trim(Pack(StripOut(splitrmrdata2(countdata2), """"))) If line2 = line + 1 Then 'Split string into separate words and characters custArray = Split(str2, " ") 'Get the name of customer (Recorder ID) customer_name = custArray(0) Exit For End If line2 = line2 + 1 System.Windows.Forms.Application.DoEvents() Next End If '''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''' 'Check if customer recorder id file exists custrecidfile = FileExists(my.Application.Info.DirectoryPath & "\customerids.txt") 'If file does not exist - create the file and add the first customer id If custrecidfile = False Then myObjFs1 = New Scripting.FileSystemObject 'Create an empty text file myObjFs1 = CreateObject("Scripting.FileSystemObject") objWrite1 = myObjFs1.CreateTextFile(my.Application.Info.DirectoryPath & "\customerids.txt") 'Write to the text file and close it objWrite1.WriteLine (customer_name) objWrite1.Close() End If 'Get contents of customer recorder id file list customerids = Split(returnContents(my.Application.Info.DirectoryPath & "\customerids.txt"), vbNewLine) 'Check if customer present in customer id list or not custpresent = False For count2 = LBound(customerids) To UBound(customerids) If StrComp(customerids(count2), Trim(Pack(customer_name)), 1) = 0 Then custpresent = True Exit For End If System.Windows.Forms.Application.DoEvents() Next 'If customer already added in list - do not add 'Else if customer not added - add into list If custpresent = False Then myObjFs2 = New Scripting.FileSystemObject 'Create an empty text file myObjFs2 = CreateObject("Scripting.FileSystemObject") objWrite2 = myObjFs2.OpenTextFile(my.Application.Info.DirectoryPath & "\customerids.txt", Scripting.IOMode.ForAppending, True) 'Write to the text file and close it objWrite2.WriteLine (customer_name) objWrite2.Close() End If 'If line is not empty, only then proceed If sJoin <> "" Then 'If row does not contain names like RECORDER ID, DATE, HOUR etc the continue If row <> 1 Then 'Write data into text file rows tempArray = Split(sJoin, ",") tempdate = tempArray(1) temptime = tempArray(2) tempkwh = tempArray(6) tempdate = Trim(Pack(Replace(tempdate, """", ""))) temptime = Trim(Pack(Replace(temptime, """", ""))) tempkwh = Trim(Pack(Replace(tempkwh, """", ""))) 'Splitting date into proper format 'Splitting date into: dd/mm/yy day_Renamed = Microsoft.VisualBasic.Strings.Left(tempdate, 2) month_Renamed = Microsoft.VisualBasic.Strings.Mid(tempdate, 3, 2) year_Renamed = Microsoft.VisualBasic.Strings.Right(tempdate, 2) 'Adjust dd/mm/yy to dd-mm-yyyy If CDbl(Microsoft.VisualBasic.Strings.Left(year_Renamed, 1)) = 8 Or CDbl(Microsoft.VisualBasic.Strings.Left(year_Renamed, 1)) = 9 Then year_Renamed = "19" & year_Renamed & "" ElseIf CDbl(Microsoft.VisualBasic.Strings.Left(year_Renamed, 1)) = 0 Or CDbl(Microsoft.VisualBasic.Strings.Left(year_Renamed, 1)) = 1 Then year_Renamed = "20" & year_Renamed & "" End If 'Set date format: dd-mm-yyyy datewrite = "" & day_Renamed & "-" & month_Renamed & "-" & year_Renamed & "" timewrite = temptime kwhwrite = tempkwh 'If file does not exist create it 'If file exists - open it, write to it and close it. origfilepath = my.Application.Info.DirectoryPath & "\" & customer_name & ".txt" myObjFs3 = New Scripting.FileSystemObject custfileexists = FileExists(origfilepath) 'If temp file does not exist, create empty text file If custfileexists = False Then myObjFs3 = CreateObject("Scripting.FileSystemObject") objWrite3 = myObjFs3.CreateTextFile(origfilepath) Else myObjFs3 = CreateObject("Scripting.FileSystemObject") objWrite3 = myObjFs3.OpenTextFile(origfilepath, Scripting.IOMode.ForAppending, True) End If 'Write to text file and close it writetofile = "" & datewrite & "," & timewrite & "," & kwhwrite & "" objWrite3.WriteLine (writetofile) objWrite3.Close() End If End If 'Increment the row row = row + 1 line = line + 1 System.Windows.Forms.Application.DoEvents() Next System.Windows.Forms.Application.DoEvents() Next End Sub

Dear all, This is part of the code for my Masters project which I wrote. I am using this code to run in VB to read data from approximately huge text files which contain raw data. Approximately 300 files each 6-7 MB; total 2.26GB text file data (each text file has like 30,000+ lines). The problem is using this code my data reads very slowly (30 days approximately to read all text files into excel worksheet), and I need to speed up the process. I was thinking if I convert to VB.NET 2005 it will execute faster. Option Explicit Dim rmr_files() As String 'Array containing directories and rmr data file names Dim rmr_folder_name As String 'Name of first folder of RMR data Dim rmr_file_list As String 'RMR data file names text file Dim rmr_data_file As String 'RMR excel data file name Dim sheet_names() As String 'Sheet names in excel RMR Data file Dim sheet_index() As Long 'Index number of excel sheets Dim found_sheet_index As Long 'Index number of found sheet Public Sub ReadFirstRMRDataFileIntoExcel() Dim filenum As Long Dim filenum2 As Long Dim str As String Dim str2 As String Dim sJoin As String Dim row As Long Dim remove_quotes As String Dim customer_name As String Dim line As Long Dim line2 As Long Dim count As Long Dim fileno As Long Dim filechange As Boolean Dim customersheet As Boolean Dim xlApp As Excel.Application Dim xlWb As Excel.Workbook Dim xlWs As Excel.Worksheet 'Set excel objects Set xlApp = CreateObject("Excel.Application") Set xlWb = xlApp.Workbooks.Open(rmr_data_file) xlApp.Visible = False xlApp.DisplayAlerts = False 'Change file indication fileno = 1 filechange = False '''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''' 'Read only the files in the first folder of the RMR data For count = LBound(rmr_files) To UBound(rmr_files) 'If the rmr data folder is the first folder then proceed only If InStr(rmr_files(count), rmr_folder_name) <> 0 Then 'Open rmr data file and begin to read filenum = FreeFile Open "" & rmr_files(count) & "" For Input As filenum row = 1 line = 1 Do While Not EOF(filenum) Line Input #filenum, str str = Trim(Pack(StripOut(str, """"))) 'Split string into separate words and characters Dim i As Long Dim sArray As Variant sArray = Split(str, " ") For i = LBound(sArray) To UBound(sArray) sArray(i) = """" & sArray(i) & """" Next 'Join back array to convert into csv format sJoin = Join(sArray, ",") If UCase(Mid(sJoin, 2, 8)) = "RECORDER" Then sJoin = Replace(sJoin, "RECORDER"",""ID", "RECORDER ID") End If 'New customer found If InStr(sJoin, "RECORDER") <> 0 Then row = 1 End If '''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''' 'Open new file 'Get the name of the customer from second line after the "RECORDER" line If row = 1 Then filenum2 = FreeFile Open "" & rmr_files(count) & "" For Input As filenum2 line2 = 1 Do While Not EOF(filenum2) Line Input #filenum2, str2 str2 = Trim(Pack(StripOut(str2, """"))) If line2 = line + 1 Then 'Split string into separate words and characters Dim custArray As Variant custArray = Split(str2, " ") 'Get the name of customer customer_name = custArray(0) Exit Do End If line2 = line2 + 1 Loop Close #filenum2 End If '''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''' '''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''' 'Check if customer sheet already added or not in excel file customersheet = False For Each xlWs In xlWb.Worksheets If Trim(customer_name) = Trim(xlWs.Name) Then customersheet = True Exit For End If Next ''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''' 'IF CUSTOMER DOES NOT EXIST IN EXCEL WORKBOOK THEN ONLY WRITE TO EXCEL SHEET If customersheet = False Then 'If it is the first customer in first file then If line = 1 And xlWb.Worksheets(1).Name <> customer_name And fileno = 1 Then xlWb.Worksheets(1).Name = customer_name Set xlWs = xlWb.Sheets(1) End If 'If new customer create new worksheet and write to it If row = 1 And line <> 1 And filechange = False Then 'Add new sheet for customers in the same txt file xlWb.Worksheets.Add xlWb.Worksheets(1).Name = customer_name Set xlWs = xlWb.Sheets(1) ElseIf row = 1 And line = 1 And filechange = True Then 'For new text files add new sheet for new customers xlWb.Worksheets.Add xlWb.Worksheets(1).Name = customer_name Set xlWs = xlWb.Sheets(1) 'File not going to change to the next until for loop increments filechange = False End If End If 'Write data into excel row Dim col As Long Dim tempArray As Variant tempArray = Split(sJoin, ",") For col = LBound(tempArray) To UBound(tempArray) remove_quotes = Trim(Pack(Replace(tempArray(col), """", ""))) 'Splitting date into proper format If col = 1 And row <> 1 Then Dim dateleft As String Dim datemid As String Dim dateright As String 'Splitting date into: dd/mm/yy dateleft = Left(remove_quotes, 2) datemid = Mid(remove_quotes, 3, 2) dateright = Right(remove_quotes, 2) 'Adjust dd/mm/yy to dd/mm/yyyy If Left(dateright, 1) = 7 Or Left(dateright, 1) = 8 Or Left(dateright, 1) = 9 Then dateright = "19" & dateright & "" ElseIf Left(dateright, 1) = 0 Or Left(dateright, 1) = 1 Or Left(dateright, 1) = 2 Then dateright = "20" & dateright & "" End If 'Set format: dd/mm/yyyy xlWs.Cells(row, col + 1).Value = "" & dateleft & "/" & datemid & "/" & dateright & "" Else 'Add entry without splitting xlWs.Cells(row, col + 1).Value = remove_quotes End If Next 'Increment the row row = row + 1 line = line + 1 DoEvents Loop Close #filenum 'Save workbook before opening new data file xlWb.SaveAs rmr_data_file End If 'Increase the file number read fileno = fileno + 1 'Check if file changed or not (new file or reading old file still) filechange = True DoEvents Next 'Save workbook and close excel xlWb.SaveAs rmr_data_file xlApp.Quit Set xlWs = Nothing Set xlWb = Nothing Set xlApp = Nothing End Sub Can anyone give me pointers on how to convert this into VB.NET, or if anyone of you can assist me, that would be a big help. I have no idea about VB.NET programming and am new, since I know most of the VB functions are obsolete in .NET 2005. regards, awyeah