Rather than rely on GC handles and a lot of interop code why don't you just serialize to a byte array?
'For a structure
<Serializable()> _
Structure SampleStruct
Public i As Integer
Public s As String
'etc.
End Structure
'the following should work...
Dim ms As New IO.MemoryStream()
Dim s As New System.Runtime.Serialization.Formatters.Binary.BinaryFormatter()
'any old sample structure / data
Dim test As SampleStruct
test.i = 1000
test.s = "Hello World"
s.Serialize(ms, test)
'following just removes any padding as allocated by ms
Dim b(ms.Position) As Byte
Array.Copy(ms.GetBuffer, b, ms.Position)
'b() now contains a serialized version of test
ms.Close()
If you are using something like this then just declare an enum of all posible structure types and send that as the first byte / integer of the packet and deserialize the correct structure based on it's value at the other end.