Speeding up serialization?

[Arokh]

[Posting frenzy he he]

 

I've been using serialization for some time now and

it is perfect for easily storing data.

 

But some issue have come up:

Currently I have ~3000 entires (with each ~7.5kb) with file information packed into an array,

which is de- serialized upon opening/closing the program.

 

For deserialzing it almost takes 8 seconds which is already too long for my purposes and the entries will continue to grow.

 

Here is the Class I use for storing Fileinformation:

    <Serializable()> Class FileClass
       Public FilePath As New FilePathClass
       Public ID(1) As Long
       Public Data As New DataClass

       <System.Serializable()> Class DataClass
           Public DataContents As New Dictionary(Of String, Object)

           Default Public Property Data(ByVal Path As String) As Object
               Get
               End Get
               Set(ByVal Value As Object)
               End Set
           End Property
       End Class
   End Class

Most of the data is contained by DataContents in DataClass.

 

I'm using the Binaryserialization since the XML one doesn't work (the dictionary object isn't supported, if I understood the error corretly).

 

So is there some way to speed it up,

or is serialzation not meant to store greater amounts of data?

If not, what alternative ways to store it are there?

 

My Serialization Methods:

    Private Shared Function DeSerializeObj(ByVal FilePath As String) As Object
       Dim SettingFileStream As IO.Stream = IO.File.OpenRead(FilePath)
       Dim SettingLoader As New Runtime.Serialization.Formatters.Binary.BinaryFormatter

       Try
           Return SettingLoader.Deserialize(SettingFileStream)

       Catch ex As Exception
           MsgBox("Coudn't load " & FilePath & ". Specific settings reseted.")
           Return Nothing
       Finally
           SettingFileStream.Close()
       End Try
   End Function
   Private Shared Sub SerializeObj(ByVal Obj As Object, ByVal FilePath As String)
       If Not My.Computer.FileSystem.DirectoryExists(My.Computer.FileSystem.GetParentPath(FilePath)) Then
           My.Computer.FileSystem.CreateDirectory(My.Computer.FileSystem.GetParentPath(FilePath))
       End If

       Dim SettingFileStream As IO.Stream = IO.File.Create(FilePath)
       Dim SettingSaver As New Runtime.Serialization.Formatters.Binary.BinaryFormatter

       SettingSaver.Serialize(SettingFileStream, Obj)
       SettingFileStream.Close()
   End Sub

[MrPaul]

Serialization is slow because it is so easy to use

 

The reason serialization is so easy to use is that the .NET serialization process embeds additional information with the underlying 'useful' data which tells the deserializer what it is deserializing. This will include information about the types of object being deserialized, and may involve the data being encoded. All of this adds overhead.

 

To reduce this additional overhead, you can write your own bespoke code for reading and writing data. This will involve explicitly writing each object field to the Stream when you save, and reading each field from the Stream when you load.

[PlausiblyDamp]

How long is it taking to serialize the data? Is this a similar amount of time compared to deserializing the data?

 

In your data class what are you storing in the dictionary - object can incur overheads in terms of run time casting etc.

[JumpyNET]

I tried serialization in one of my projects too, but it was too slow so I ended up looping through all the items and saving them a little like this:

   Public Sub SaveRecording(ByVal FilePath As String)

       'Ensure that the target does not exist.
       System.IO.File.Delete(FilePath)

       'Declare those variables!
       Dim sFileName As String = FilePath
       Dim fs As New System.IO.FileStream(sFileName, _
           System.IO.FileMode.Create, System.IO.FileAccess.Write, System.IO.FileShare.ReadWrite)
       Dim Compresser As New ICSharpCode.SharpZipLib.BZip2.BZip2OutputStream(fs)

       'Write in some text 
       AddText(Compresser, "BLAA BLAA BLAA")
       AddText(Compresser, Environment.NewLine)

       'Write something else
       Dim TmpBytes As List(Of Byte) = New List(Of Byte)
       TmpBytes.Add(32)
       TmpBytes.Add(13)
       Compresser.Write(TmpBytes.ToArray, 0, TmpBytes.Count)

       'Close everything
       Compresser.Close()
       fs.Close()

   End Sub


   Private Sub AddText(ByVal fs As ICSharpCode.SharpZipLib.BZip2.BZip2OutputStream, ByVal value As String)
       Dim info As Byte() = New System.Text.UTF8Encoding(True).GetBytes(value)
       fs.Write(info, 0, info.Length)
   End Sub

 

The data I needed to save was some times more than 100 MB uncompressed, so compressing greatly reduced the time it took to save the file.

 

By loopin through the items your self you can easily detect the saving progress. Deserialization also seems to crash if you change your application's or some variable's name.

[Arokh]

Sorry for replying so late.

 

The dictionary DataContents stores mostly Strings,

but I use it more like a "treeview".

The dictionary can contains other dictionaries,

providing something like a folder structure.

Which is why it takes so long, I guess.

 

I guess Ithe only choice I have is the way data is stored in DataContents,

to speed things up, I just hoped/hope for an easier way.

 

 

How long is it taking to serialize the data?

Mhh, it also takes some time but not as long as deserializing.