A Gentle Introduction to Serialization for Python

Originally posted on machinelearningmastery.

Serialization refers to the process of converting a data object (e.g. Python objects, Tensorflow models) into a format that allows us store or transmit the data and then recreate the object when needed using the reverse process of deserialization.

There are different formats for serialization of data, such as JSON, XML, HDF5, Python’s pickle, for different purposes. JSON for instance returns a human-readable string form while Python’s pickle library can return a byte array.

In this post, you will discover how to use two common serialization libraries in Python to serialize data objects (namely pickle and HDF5) such as dictionaries and Tensorflow models in Python for storage and transmission

After completing this tutorial, you will know:

  • Serialization libraries in Python such as pickle and h5py
  • Serializing objects such as dictionaries and Tensorflow models in Python
  • How to use serialization for memoization to reduce function calls

Let’s get started!


The tutorial is divided into four parts, they are:

  • What is serialization and why do we serialize?
  • Using Python’s pickle library
  • Using HDF5 in Python
  • Comparison between different serialization methods

What is serialization and why should we care?

Think about storing an integer, how would you store that in a file or transmit it? That’s easy! We can just write the integer to a file and store or transmit that file.

But now, what if we think about storing a Python object (e.g. a Python dictionary or a Pandas DataFrame), which has a complex structure and many attributes (e.g., columns and index of the DataFrame, and the data type of each column)? How would you store it as a file or transmit it to another computer?

This is where serialization comes in!

Serialization is the process of converting the object into a format that can be stored or transmitted. After transmitting or storing the serialized data, we are able to later reconstruct the object and obtain the exact same structure/object, which makes it really convenient for us to continue using the stored object later on instead of reconstructing the object from scratch.

In Python, there are many different formats for serialization available. One common example for hash maps (Python dictionaries) which works across many languages is the JSON file format which is human-readable and allows us to store the dictionary and recreate it with the same structure. But JSON can only store basic structures such as list and dictionary, and it can only keep strings and numbers. We cannot ask JSON to remember the data type (e.g., numpy float32 vs float64). It also cannot distinguish between Python tuples and lists.

More powerful serialization formats exists. In the following, we will explore two common serialization libraries in Python, namely pickle and h5py.

Using Python’s pickle library

The pickle module is part of the Python standard library and implements methods to serialize (pickling) and deserialize (unpickling) Python objects.

To get started with pickle, import it in Python

afterwards, to serialize a Python object such as a dictionary and store the byte stream as a file, we can use pickle’s dump() method.

and the byte stream representing test_dict is now stored in the file “test.pickle”!

To recover the original object, we read the serialized byte stream from the file using pickle’s load() method.

Warning: Only unpickle data from sources you trust as it is possible to for arbitrary malicious code to be executed during the unpickling process.

Put them together, the following code helps you to verify pickle can recover the same object:

Besides writing the serialized object into a pickle file, we can also obtain the object serialized as a bytes-array type in Python using pickle’s dumps() function:

Similarly, we can use pickle’s loads method to convert from a bytes-array type back to the original object

One useful thing about pickle is that it can serialize almost any Python object, including user-defined ones, such as the following

The code above will print the following

Note that the print statement in the class’ constructor is not executed at the time pickle.loads() is invoked. Because it reconstructed the object, not recreated it.

Pickle can even serialize Python functions since functions are first class objects in Python:

Therefore, we can make use of pickle to save our work. For example, a trained model from Keras or scikit-learn can be serialized by pickle and load it later instead of re-train the model every time we use it. The following is to show you how we can build a LeNet5 model to recognize the MNIST handwritten digits using Keras, then serialize the trained model using pickle. Afterwards, we can reconstruct the model without training it again and it should produce exactly the same result as the original model:

The above code will produce the output as follows, note that the evaluation scores from the original and reconstructed models are tied out perfectly at the last two lines:

While pickle is a powerful library, it still does have its own limitations to what can be pickled. For example, live connections such as database connections and opened file handles cannot be pickled. This issue arises because reconstructing these objects requires pickle to re-establish the connection with the database/file which is something pickle cannot do for you (because it needs appropriate credentials and is out of scope of what pickle is intended for).

Using HDF5 in Python

Hierarchical Data Format 5 (HDF5) is a binary data format. The h5py package is a Python library that provides an interface to the HDF5 format. From h5py docs, HDF5 “lets you store huge amounts of numerical data, and easily manipulate that data from Numpy.”

What HDF5 can do better than other serialization format is that it stores data in a file system like hierarchy. You can store multiple objects or dataset in HDF5 like saving multiple files in the file system. You can also read a particular dataset from HDF5 like reading one file from the file system without concerning the other. If you’re using pickle for this, you will need to read and write everything each time you load or create the pickle file. Hence HDF5 is advantageous for huge amount of data that can’t fit entirely into memory.

To get started with h5py, you first need to install the h5py library, which you can do using

Or if you are using a conda environment

We can then get started with creating our first dataset!

This creates a new dataset in the file test.hdf5 named “test_dataset”, with a shape of (100, ) and a type int32. h5py datasets follow a Numpy syntax so you can do slicing, retrieval, get shape, etc. similar to Numpy arrays

To retrieve at a specific index,

To get a slice from index 0 to index 10 of dataset,

If you initialized the h5py file object outside of a with statement, remember to close the file as well!

To read from a previously created HDF5 file, you can open the file in “r” for read mode or “r+” for read/write mode,

To organize your HDF5 file, you can use groups,

Another way to create groups and files is by specifying the path to the dataset you want to create and h5py will create the groups on that path as well (if they don’t exist),

The two snippets of code both create group1 if it has not be created previously, and then a dataset1 within group1.

HDF5 in Tensorflow

To save a model in Tensorflow Keras using HDF5 format, we can use the save() function of the model with a filename having extension .h5, like the following:

To load the stored HDF5 model, we can also use the function from Keras directly:

One reason we don’t want to use pickle for a Keras model is that we need a more flexible format that does not tie to a particular version of Keras. If we upgraded our Tensorflow version, the model object may change and pickle may fail to give us a working model. Another reason is to keep only the essential data for our model. For example, if we check the HDF5 file my_model.h5 created in the above, we see these are stored:

Hence Keras selected only the data that are essential to reconstruct the model. A trained model will contain more datasets, namely, there are /optimizer_weights/ besides /model_weights/. Keras will reconstruct the model and restore the weights appropriately to give us a model that function the same.

Comparing between different serialization methods

In the above we saw how pickle and h5py can help serialize our Python data.

We can use pickle to serialize almost any Python object, including user-defined ones and functions. But pickle is not language agnostic. You cannot unpickle it outside Python. There are even 6 versions of pickle developed so far and older Python may not be able to consume the newer version of pickle data.

On the contrary, HDF5 is cross platform and works well with other language such as Java and C++. In Python, the h5py library implemented the Numpy interface to make it easier to manipulate the data. The data can be accessed in different language because HDF5 format supports only the Numpy data types such as float and strings. We cannot store arbitrary objects such as a Python function into HDF5.

Further reading

This section provides more resources on the topic if you are looking to go deeper.





In this post, you discovered what serialization is and how to use libraries in Python to serialize Python objects such as dictionaries and Tensorflow Keras models. You have also learnt the advantages and disadvantages of two Python libraries for serialization (pickle, h5py).

Specifically, you learned:

  • what is serialization and why it is useful
  • how to get started with pickle and h5py serialization libraries in Python
  • pros and cons of different serialization methods


Source: machinelearningmastery