Some Common Gotchas in Python

As a new Python programmer, I ran into three common language gotchas: passing variables by reference, unbound local errors, and mutable default arguments. Hopefully this post will help shed some light on these gotchas and save you the time I spent debugging them. However, what I gained by attempting to understand these gotchas was a wonderful opportunity to take a deep dive into a new language and explore some of its under-the-hood behavior.


Passing By Reference

 
I encountered the pass-by-reference gotcha when I first tried to play the game loop. My game was able to ask the user for input, set the size of the board, create a board, and accurately create a human and computer player. However, when I tried to place the first move, my move was not marked in the one square I had picked, but in three squares.

My game was marking the entire column.

In other words, instead of looking like the image on the left, my board looked like the image on the right:

x 2 3
4 5 6
7 8 9
x 2 3
x 5 6
x 8 9

After some research and debugging, I discovered that this behavior was thanks to Python’s pass by reference characteristic. My board had been created using the following function:

[[None * row_size] * row_size]  

Here when creating a board with a row_size of 3, one might expect to get 9 distinct None objects. However, when building the board using this function in Python, we start off with None, which is Python's null equivalent. None is an object (for more about None see the section directly below). The above line of code creates three distinct None objects while resolving the code within the inner set of square brackets, thereby producing a single row of distinct elements. It then duplicates that result twice more in order to resolve the code in the outer square brackets, leading to a total of three rows, each row having the same references as the previous one. Which is why changing one None cell caused a change in all three cells in the same column.
 

A Brief Look At None In Python

  • None is an object—a class. Not a primitive type, such as int, or True and False.
  • In Python 3.x, the type object was changed from type 'NoneType' to class 'NoneType'. However the behaviour of None has remained the same from 2.x to 3.x.
  • Because None is an object, we cannot use it to check if a variable exists. It is a value/object, not an operator used to check a condition.
     
     
    In this REPL—Click on the Play button to run tests, or toggle lines 24 and 25, and then run the code to view the Pass-by-Reference gotcha
     

Python Scope

 
To discuss Unbound Local Errors, I thought it might be helpful to touch upon Python scoping first. Scoping in Python follows the LEGB Rule: Local -> Enclosed
-> Global -> Built-in.
Here we work our way from the inside out, from local to built-in.

posts/2019-05-07-some-common-gotchas-in-python/legb.png
  • Local scope refers to a variable that is defined within a function body. A variable is always checked for in local scope first.
    In the repl below, if we toggle the lines of code so that line 8 is off, we'll see that our outer function executes and prints the local version of variable my_var, returning a value of my inner/local variable.
  • Enclosed scope is created when a function wraps another function—here the outer function is the enclosing scope. Back in our repl, we can now comment line 7 and uncomment line 8. On line 8, the nonlocal keyword will now point Python to using the variable initialized on line 5, in the outer enclosing scope. Thus we will see my enclosed variable printed to the console.
  • Global—as we can imagine, variables assigned at the top-level of a module file are global in scope. line 13 of the repl prints my_var from line 2, as it is unable to access the various other instantiations of my_var that are all within enclosed or local scopes. Similar to the nonlocal keyword, we can use the global keyword to initialize a variable to the global scope from an enclosed or local context.
  • Built-ins—no surprises here either, built-ins refer to names preassigned in Python's built-in names module, such as Math, open, range, and SyntaxError.


    In this REPL—Click on the Play button to run tests, or toggle lines 15, 24, and 25, and then run the code to learn more about how Python's LEGB scoping works
     
     



Unbound Local Error

 
Let's say we define a variable my_var in the global scope and set it to 5. If we then create a function and attempt to assign a new value to the same variable, but within the function's local context, we'll get an UnboundLocalError returned to us. The above error occurs because, when we make an assignment to a variable in scope, that variable becomes local to that scope and shadows any similarly named variable in the outer scope. In the context of the repl below, on line 7 we are in effect initializing a new variable my_var to a value of += 1, and an error is generated. This error is particularly common when working with lists, as shown in the last example in the repl, and one way to solve accessing a variable declared in the global context is to refer to it as global, as seen in the method on line 10 of the repl.  
 
In this REPL—Click on the Play button to run tests, or edit the code in functions bar and foo2 to play with Unbound Local Errors
 
 

 

Mutable Default Arguments

For my second bug, a gotcha showed up in my implementation of the Minimax algorithm. My code was displaying some strange behavior.

x x o
o 5 o
x o x
x o x
x o x
o 8 9
x o x
4 o 6
7 8 9

My first test was set up to check that Minimax chose the only open spot, in this case a 5.

After that, the next test checked if, when given a choice between two open spots, Minimax selected the winning move as opposed to a blocking move. In this case the open spots were 8 and 9, counting from a 1-indexed array.

A third test checked whether, when given a choice of multiple open spots, Minimax still chose the winning move. Here the open spots in the 1-indexed array included 4, 6, 7, 8, 9.

Strangely enough, 5 kept showing up as the selected move in all three of these tests, even though 5 was only a valid option in my very first test, and not a valid option in the other two mentioned. It looked like the open spot from my first test was somehow persisting into my next two tests.

Here my mentor alerted me to read up on Python Gotchas, of which Mutable Default Arguments1 was one.

According to the article,

A new list is created once when the function is defined, and the same list is used in each successive call.

Python’s default arguments are evaluated once when the function is defined, not each time the function is called (like it is in say, Ruby). This means that if you use a mutable default argument and mutate it, you will and have mutated that object for all future calls to the function as well.

 
Here was the offending line itself:

def minimax(self, game, depth=0, scores_map={}):  

The same goes for any default arguments in a function call that are set to mutable data types in Python. A quick search for mutable types in Python tells us:

Objects of built-in types like int, float, bool, str, tuple, and unicode are immutable. Objects of built-in types like list, set, and dict are mutable. Custom classes are generally mutable.

A table to illustrate might be simpler, see below.2

My scores map was being mutated with a 5 the first time it was called and then it held on to the 5 for all future calls to that function as well.

To avoid this common gotcha, the trick was to adjust my code to initialize the scores map variable to None instead of an empty dictionary in the function call, and then to check if scores_map needed to be initialized later within the body of the function:

def minimax(self, game, depth=0, scores_map=None):
  if not scores_map:  
    scores_map = {}   



In this REPL—Click on the Play button to run tests, or toggle the multiple tests on each of the map and array methods to view how this quirk is triggered

   


 

Mutable & Immutable Types In Python

Class Description Immutable Mutable
bool Boolean value x
int Integer (arbitrary magnitude) x
float floating-point number x
list Mutable sequence of objects x
tuple Immutable sequence of objects x
str Character string x
set Unordered set of distinct objects x
frozenset Immutable form of set class x
dict Dictionary/Associative mapping x


  1. Reitz, K [online]. Available at: https://docs.python-guide.org/writing/gotchas/ [Accessed 15 Oct. 2018] 

  2. Mohan, M [online] Available at: https://medium.com/@meghamohan/mutable-and-immutable-side-of-python-c2145cf72747
    [Accessed 15 Oct. 2018] 

Shibani Mookerjee, Software Crafter

Shibani Mookerjee is a Software Crafter at 8th Light in NYC.

Interested in 8th Light's services? Let's talk.

Contact Us