Colors are serializable, so you should have no trouble. Are you remembering to tell the XML Serializer that you want to serialize it? For example, you usually make an array of all the types you want to serialize:
Dim serTypes() As Type = {GetType(String), GetType(Drawing.Color), GetType(MyClass)}
Dim xmlSer As New Xml.Serialization.XmlSerializer(GetType(Integer), serTypes)
' ^ The first param is the type you wish to serialize, and the second
' param is an array of other types you also wish to serialize. With
' the array above, you can serialize String, Color, MyClass, plus
' Integer.
'do the serialization here