--- jupytext: formats: ipynb,md:myst text_representation: extension: .md format_name: myst format_version: 0.13 jupytext_version: 1.11.5 kernelspec: display_name: Python 3 (ipykernel) language: python name: python3 --- ```{code-cell} ipython3 :tags: [remove-cell] from ndslib.config import jupyter_startup jupyter_startup() ``` (python)= # A brief introduction to Python Researchers in neuroimaging use different programming languages to perform data analysis. In this book, we chose to focus on the **Python** programming language. However, as we mentioned in {numref}`about`, this book isn't meant to be a general introduction to programming; out of necessity, we're going to assume that you've had some prior experience writing code in one or more other programming languages (or possibly even Python itself). That said, we'll be *briefly* reviewing key programming concepts as we go, so you don't need to have had *much* prior experience. As long as you've encountered, say, variables, functions, and for-loops before, you should be just fine. And if you haven't yet, we recommend some resources at the end of this section, to get you up to speed. Conversely, if you've been programming in other languages for years, you should be able to breeze through this section very quickly, though we might still recommend paying attention to the last few sub-sections, which talk about some deeper ideas underlying the Python language. ## What is Python? Let's start by talking a bit about what Python is and why we chose it for this book. First of all, it is a programming language, which means it's a set of rules for writing instructions that a computer can understand and follow. Of course, that alone doesn't make Python special; there are hundreds of other programming languages out there! But Python isn't just *any* programming language; by many rankings, it's currently (as we write this in 2022) the world's single most popular language. And it's particularly dominant in one of this book's two core areas of focus: data science. We also happen to think it's by far the best choice for doing serious work in the book's other core area of focus--neuroimaging, and it is the language that we use most often for our neuroimaging work. But you don't have to take our word for that right now; hopefully, you'll be convinced of it as we work through this book. Why do so many people like Python? There are many answers to that, but here are a few important ones. First, Python is a *high-level, interpreted* programming language. This means that, in contrast to low-level, compiled languages like C, C++, Java, etc., Python features a high level of abstraction. A lot of the things you have to worry about in many low-level languages (e.g., memory allocation, garbage collection, etc.) are done for you automatically in Python. Second, Python's syntax is readable and (relatively speaking) easy to learn. As you'll see shortly, many Python operators are ordinary English words. Python imposes certain rules on code structure that most languages don't. This may be a bit annoying at first, but it also makes it easier to read other people's code once you acclimate. One of the consequences of these design considerations is that mathematical ideas translate into code in a way that does not obscure the math. That means that mathematical equations implemented in Python look a lot like the same equations written on paper. This is useful in data science applications, where ideas are expressed in mathematical terms. Third, Python is a *general-purpose* language. In contrast to many other dynamic programming languages designed to serve specific niche uses, Python is well-suited for a wide range of applications. It features a comprehensive standard library (i.e., the functionality available out-of-the-box when you install Python) and an enormous ecosystem of third-party packages. It also supports multiple programming paradigms to varying extents (object-oriented, functional, etc.). Consequently, Python is used in many areas of software development. Very few other languages can boast that they have some of the best libraries implemented in *any* language for tasks as diverse as, say, scientific computing and back-end web development. Lastly, there's the sheer size of the Python community. While Python undeniably has many attractive features, we wouldn't want to argue that it's a *better* overall programming language than anything else. To some degree, it's probably true that Python's popularity is an accident of history. If we could randomly re-run the last two decades, we might be extolling the virtues of Haskell (or Julia, or Ruby, or...) instead of Python. So we're not saying Python is *intrinsically* the world's greatest language. But there's no denying that there are immense benefits to using a language that so many other people use. For most tasks you might want to take on if you're doing data science and/or neuroimaging data analysis, finding good libraries, documentation, help, and collaborators is simply going to be much easier if you work in Python than if you work in almost any other language. Importantly, the set of tools in Python specifically for the analysis of neuroimaging data have rapidly evolved and matured in the last couple of decades, and they have gained substantial popularity through their use in research (we'll dive into these tools in {numref}`nipy`). The Neuroimaging in Python ecosystem also has a strong ethos of producing high-quality open-source software, which means that the Python tools that we will describe in this book should be accessible to anyone. More broadly, Python has been adopted across many different research fields and is one of the most commonly used programming languages in the analysis of data across a broad range of domains, including astronomy, geosciences, natural language processing, and so on. For researchers who are thinking of applying their skills in the industry outside of academia, Python is very popular in the industry, with many entry positions in industry data science calling out specifically experience programming in Python as a desired skill. With that basic sales pitch out of the way, let's dive right into the Python language. We'll spend the rest of this chapter working through core programming concepts and showing you how they're implemented in Python. It should go without saying that this can only hope to be a cursory overview; there just isn't time and space to provide a full introduction to the language! But we'll introduce many more features and concepts throughout the rest of the book, and we also end each chapter with a curated list of additional resources you can look into if you want to learn much more. ## Variables and basic types It's common to introduce programming by first talking about variables, and we won't break with this convention. A variable, as you probably already know, is a store of data that can take on different values (hence the name *variable*), and is typically associated with a fixed name. ### Declaring variables ```{eval-rst} .. index:: single: Variable assignment ``` In Python, we declare a variable by writing its name and then assigning it a value with the equal (`=`) sign: ```{code-cell} ipython3 my_favorite_variable = 3 ``` Notice that when we initialize a variable, we don't declare its *type* anywhere. If you're familiar with *statically typed* languages like C++ or Java, you're probably used to having to specify what type of data a variable holds when you create it. For example, you might write `int my_favorite_number = 3` to indicate that the variable is an integer. In Python, we don't need to do this. Python is *dynamically typed*, meaning that the type of each variable will be determined on the fly, once we start executing our program. It also means we can change the type of the variable on the fly, without anything bad happening. For example, by over-writing it with a character string value, instead of the integer value that was once stored in this variable: ```{code-cell} ipython3 my_favorite_variable = "zzzzzzz" ``` ### Printing variables ```{eval-rst} .. index:: single: print function ``` We can examine the contents of a variable at any time using the built-in `print()` function: ```{code-cell} ipython3 print(my_favorite_variable) ``` If we're working in an interactive environment like a Jupyter notebook, we may not even need to call `print()`, as we'll automatically get the output of the last line evaluated by the Python interpreter: ```{code-cell} ipython3 # this line won't be printed, because it isn't the last line in the notebook cell to be evaluated "this line won't be printed" # but this one will my_favorite_variable ``` As you can see in this example, the hash sign (`#`) is used for comments. That means that any text that is after a `#` is ignored by the Python interpreter and not evaluated. ### Built-in types +++ All general-purpose programming languages provide the programmer with different *types* of variables—things like strings, integers, Booleans, and so on. These are the most basic building blocks a program is made up of. Python is no different and provides us with several [built-in types](https://docs.python.org/3/library/stdtypes.html)](https://docs.python.org/3/library/stdtypes.html). Let's take a quick look at some of these. #### Integers An integer is a numerical data type that can only take on finite whole numbers as its value. For example: ```{code-cell} ipython3 number_of_subjects = 20 number_of_timepoints = 1000 number_of_scans = 10 ``` Any time we see a number written somewhere in Python code, and it's composed only of digits (no decimals, quotes, etc.), we know we're dealing with an integer. In Python, integers support all of the standard arithmetic operators you're familiar with--addition, subtraction, multiplication, etc. For example, we can multiply the two variables we just defined: ```{code-cell} ipython3 number_of_subjects * number_of_timepoints ``` Or divide one integer by another: ```{code-cell} ipython3 number_of_timepoints / number_of_scans ``` Notice that the result of the above division is *not* itself an integer! The decimal point in the result gives away that the result is of a different type -- a *float*. +++ #### Floats A float (short for *floating point number*) is a numerical data type used to represent real numbers. As we just saw, floats are identified in Python by the presence of a decimal. ```{code-cell} ipython3 roughly_pi = 3.14 mean_participant_age = 24.201843727 ``` All of the standard arithmetic operators work on floats just like they do on ints: ```{code-cell} ipython3 print(roughly_pi * 2) ``` We can also freely combine ints and floats in most operations: ```{code-cell} ipython3 print(0.001 * 10000 + 1) ``` Observe that the output is of type `float`, even though the value is a whole number, and hence could in principle have been stored as an `int` without any loss of information. This is a general rule in Python: arithmetic operations involving a mix of `int` and `float` operands will almost always return a `float`. Some operations will return a `float` even if all operands are `int`s, as we saw above in the case of division. (python_ex1)= #### Exercise The Python built-in `type()` function reports to you the type of a variable that is passed to it. Use the `type` function to verify that `number_of_subjects * number_of_timepoints` is a Python integer, while `number_of_timepoints / number_of_scans` is not. Why do you think that Python changes the result of a division into a variable of type float? #### Strings A string is a sequence of characters. In Python, we define strings by enclosing zero or more characters inside a pair of quotes (either single or double quotes work equally well, so you can use whichever you prefer; just make sure the opening and closing quotes match!). ```{code-cell} ipython3 country = "Madagascar" ex_planet = 'Pluto' ``` Python has very rich built-in functionality for working with strings. Let's look at some of the things we can do. We can calculate the length of a string: ```{code-cell} ipython3 len(country) ``` Or convert it to uppercase (try also `.lower()` and `.capitalize()`): ```{code-cell} ipython3 country.upper() ``` We can count the number of occurrences of a substring (in this case, a single letter `a`): ```{code-cell} ipython3 country.count("a") ``` Or replace a matching substring with another substring: ```{code-cell} ipython3 country.replace("car", "truck") ``` One thing that you might notice in the above examples is that they seem to use two different syntaxes. In the first example, it looks like `len()` is a *function* that takes a string as its parameter (or *argument*). By contrast, the last 3 examples use a different "dot" notation, where the function comes after the string (as in `country.upper()`). If you find this puzzling, don't worry! We'll talk about the distinction in much more detail below. (python_ex2)= #### Exercise Write code to count how many times the combination "li" appears in the string "supercalifragilisticexpialidocious". Assign this value into a new variable named `number_of_li` and print its value. #### Booleans ```{eval-rst} .. index:: single: Booleans ``` Booleans operate pretty much the same in Python as in other languages; the main thing to recognize is that they can only take on the values `True` or `False`. Not `true` or `false`, not `"true"` or `"false"`. The only values a boolean can take on in Python are `True` and `False`, written exactly that way. For example: ```{code-cell} ipython3 enjoying_book = True ``` One of the ways that boolean values are typically generated in Python programs is through logical or comparison operations. For example, we can ask whether the length of a given string is greater than a particular integer: ```{code-cell} ipython3 is_longer_than_2 = len("apple") > 2 print(is_longer_than_2) ``` Or whether the product of the first two numbers below equals the third... ```{code-cell} ipython3 is_the_product = 719 * 1.0002 == 2000 print(is_the_product) ``` Or, we might want to know whether the conjunction of several sub-expressions is `True` or `False`: ```{code-cell} ipython3 ("car" in country) and (len("apple") > 2) and (15 / 2 > 7) ``` This last example, simple as it is, illustrates a nice feature of Python: its syntax is more readable than that of most other programming languages. In the above example, we ask if the substring `"car"` is contained in the string `country` using the English language word `in`. Similarly, Python's logical conjunction operator is the English word `and`. This means that we can often quickly figure out -- or at least, vaguely intuit -- what a piece of Python code does. (python_ex3)= #### Exercise Some integer values are equivalent to the Python Boolean values. Use the equality (`==`) operator to find integers that are equivalent to `True` and that are equivalent to `False`. #### None ```{eval-rst} .. index:: single: None ``` In addition to these usual suspects, Python also has a type called `None`. `None` is special and indicates that no value has been assigned to a variable. It's roughly equivalent to the `null` value found in many other languages. ```{code-cell} ipython3 name = None ``` Note: `None` and `False` are *not* the same thing! ```{code-cell} ipython3 name == False ``` Also, assigning the value `None` to a variable is not the same as not defining the variable in the first place. Instead, a variable that is set to `None` is something that we can point to in our program without raising an error but doesn't carry any particular value. These are subtle but important points, and in later chapters, we'll use code where the difference becomes important. ## Collections Most code we're going to want to write in Python will require more than just integers, floats, strings, and booleans. We're going to need more complex data structures, or *collections*, that can hold other objects (like strings, integers, etc.) and enable us to easily manipulate them in various ways. Python provides built-in support for many common data structures, and others can be found in modules that come installed together with the language itself -- the so-called "standard library" (e.g., in the [collections](https://docs.python.org/3/library/collections.html) module). ```{eval-rst} .. index:: single: Python standard library ``` ### Lists ```{eval-rst} .. index:: single: List ``` Lists are the most common collection we'll work with in Python. A list is a *heterogeneous* collection of objects. By heterogeneous, we mean that a list can contain elements of different types. It doesn't have to contain only strings or only integers; it can contain a mix of the two, as well as all kinds of other types. #### List initialization To create a new list, we enclose one or more values between square brackets (`[` and `]`). Elements are separated by commas. Here is how we initialize a list containing 4 elements of different types (an integer, a float, and two strings). ```{code-cell} ipython3 random_stuff = [11, "apple", 7.14, "banana"] ``` (listindexing)= #### List indexing ```{eval-rst} .. index:: single: Indexing (lists) ``` Lists are *ordered* collections, by which we mean that a list retains a memory of the position each of its elements was inserted in. The order of elements won't change unless we explicitly change it. This allows us to access individual elements in the list directly, by specifying their position in the collection, or *index*. To access the $i^{th}$ element in a list, we enclose the index $i$ in square brackets. Note that Python uses 0-based indexing (i.e., the first element in the sequence has index 0), and not 1 as in some other data-centric languages (Julia, R, etc.). For example, it means that the following operation returns the second item in the list and not the first. ```{code-cell} ipython3 random_stuff[1] ``` Many bitter wars have been fought on the internet over whether 0-based or 1-based indexing is better. We're not here to take a philosophical stand on this issue; the fact of the matter is that Python indexing is 0-based, and that's not going to change. So whether you like it or not, you'll need to make your peace with the idea that indexing starts from 0 while you're reading this book. (python_ex4)= #### Exercise Indexing with negative numbers: in addition to indexing from the beginning of the list, we can index from the end of the list using negative numbers (e.g., `random_stuff[-1]`). Experiment indexing into the list `random_stuff` with negative numbers. What is the negative number index for the last item in the list? What is the negative number index for the first item in the list? Can you write code that would use a negative number to index the first item in the list, without advance knowledge of its length? #### List slicing ```{eval-rst} .. index:: single: Slicing (lists) ``` Indexing is nice, but what if we want to pull more than one element at a time out of our list? Can we easily retrieve only part of a list? The answer is yes! We can *slice* a list, and get back another list that contains multiple contiguous elements of the original list, using the colon (`:`) operator. ```{code-cell} ipython3 random_stuff[1:3] ``` In the list-slicing syntax, the number before the colon indicates the start position, and the number after the colon indicates the end position. Note that the start is inclusive and the end is exclusive. That is, in the above example, we get back the 2nd and 3rd elements in the list, but *not* the 4th. If it helps, you can read the `1:3` syntax as saying *I want all the elements in the list starting at index `1` and stopping just before index `3`*. +++ #### Assigning values to list elements Lists are *mutable* objects, meaning that they can be modified after they've been created. In particular, we very often want to replace a particular list value with a different value. To overwrite an element at a given index, we assign a value to it, using the same indexing syntax we saw above: ```{code-cell} ipython3 print("Value of first element before re-assignment:", random_stuff[0]) random_stuff[0] = "eleventy" print("Value of first element after re-assignment:", random_stuff[0]) ``` #### Appending to a list It's also very common to keep appending variables to an ever-growing list. We can add a single element to a list via the `.append()` function (notice again that we are calling a function using the 'dot' notation, we promise that we'll come back to that later!). ```{code-cell} ipython3 random_stuff.append(88) print(random_stuff) ``` (python_ex5)= #### Exercise There are several ways to combine lists, including the `append` function you saw above, as well as the `extend` method. You can also add lists together using the addition (`+`) operator. Given the following two lists: `list1 = [1, 2, 3]` `list2 = [4, 5, 6]` How would you create a new list called `list3` that has the items: `[6, 5, 1, 2, 3]`, with as few operations as possible and only using indexing operations and functions associated with the list (hint: you can look up these functions in the Python [online documentation for lists](https://docs.python.org/3/tutorial/datastructures.html#more-on-lists)) ### Dictionaries (dict) ```{eval-rst} .. index:: single: Dictionary ``` Dictionaries are another extremely common data structure in Python. A dictionary (or `dict`) is a mapping from keys to values; we can think of it as a set of key/value pairs, where the keys have to be unique (but the values don't). Many other languages have structures analogous to Python's dictionaries, though they're usually called something like *associative arrays*, *hash tables*, or *maps*. #### Dictionary initialization We initialize a dictionary by specifying comma-delimited key/value pairs inside curly braces. Keys and values are separated by a colon. It looks like this: ```{code-cell} ipython3 fruit_prices = { "apple": 0.65, "mango": 1.5, "strawberry": "$3/lb", "durian": "unavailable", 5: "just to make a point" } ``` Notice that both the keys and the values can be heterogeneously typed (observe the last pair, where the key is an integer). #### Accessing values in a dictionary In contrast to lists, you can't access values stored in a dictionary directly by their serial position. Instead, values in a dictionary are accessed by their key. The syntax is identical to that used for list indexing. We specify the key whose corresponding value we'd like to retrieve in between square brackets: ```{code-cell} ipython3 fruit_prices['mango'] ``` And again, the following example would fail, raising a `KeyError` telling us there's no such key in the dictionary: ```{code-cell} ipython3 fruit_prices[0] ``` However, the reason the above key failed is *not* that integers are invalid keys. To prove that, consider the following: ```{code-cell} ipython3 fruit_prices[5] ``` Superficially, it might look like we're requesting the 6th element in the dictionary and getting back a valid value. But that is not what is actually happening here. If it's not clear to you why `fruit_prices[0]` fails while `fruit_prices[5]` succeeds, go back and look at the code we used to create the `fruit_prices` dictionary. Carefully inspect the keys and make sure you understand what's going on. #### Updating a dictionary Updating a dictionary uses the same `[]`-based syntax as accessing values, except we now make an explicit assignment. For example, we can add a new entry for the `ananas` key: ```{code-cell} ipython3 fruit_prices["ananas"] = 0.5 ``` Or over-write the value for the `mango` key: ```{code-cell} ipython3 fruit_prices["mango"] = 2.25 ``` And then look at the dict again: ```{code-cell} ipython3 print(fruit_prices) ``` (python_ex6)= #### Exercise Add another fruit to the dictionary. This fruit should have several different values associated with it, organized as a list. How can you access the second item in this list in one single call? ### Tuples ```{eval-rst} .. index:: single: Tuple ``` The last widely-used Python collection we'll discuss here (though there are many other more esoteric ones) is the *tuple*. Tuples are very similar to lists in Python. The main difference between lists and tuples is that lists are *mutable*, meaning, they can change after initialization. Tuples are *immutable*; once a tuple has been created, it can no longer be modified. We initialize a tuple in much the same way as a list, except we use parentheses (round brackets) instead of square brackets: ```{code-cell} ipython3 my_tuple = ("a", 12, 4.4) ``` Just to drive home the immutability of tuples, let's try replacing a value and see what happens: ```{code-cell} ipython3 my_tuple[1] = 999 ``` Our attempt to modify the tuple raises an error. Fortunately, we can easily convert any tuple to a list, after which we can modify it to our heart's content. ```{code-cell} ipython3 converted_from_tuple = list(my_tuple) converted_from_tuple[1] = 999 print(converted_from_tuple) ``` In practice, you can use a list almost anywhere you can use a tuple, though there are some important exceptions. One that you can already appreciate is that a tuple can be used as a key to a dictionary, but a list can't: ```{code-cell} ipython3 dict_with_sequence_keys = {my_tuple : "Access this value using a tuple!"} ``` ```{code-cell} ipython3 dict_with_sequence_keys[converted_from_tuple] = "This will not work" ``` Admittedly, the error that this produces is a bit cryptic, but it relates directly to the fact that a mutable object is considered a bit unreliable because elements within it can change without notice. ## Everything in Python is an object ```{eval-rst} .. index:: single: Object-oriented programming ``` Our discussion so far might give off the impression that some data types in Python are basic or special in some way. It's natural to think, for example, that strings, integers, and booleans are "primitive" data types —- i.e., that they're built into the core of the language, behave in special ways, and can't be duplicated, or modified. And this is true in many other programming languages. For example, in Java, there are exactly 8 primitive data types. If you get bored of them, you're out of luck. You can't just create new ones -— say, a new type of string that behaves just like the primitive strings, but adds some additional functionality you think would be kind of cool to have. Python is different: it doesn't *really* have any primitive data types. Python is a deeply *object-oriented* programming language, and in Python, *everything is an object*. Strings are objects, integers are objects, booleans are objects. So are lists. So are dictionaries. *Everything* is an object in Python. We'll spend more time talking about what objects are, and the deeper implications of everything being an object, at the end of this chapter. For now, let's focus on some of the practical implications for the way we write code. ### The dot notation ```{eval-rst} .. index:: single: dot notation ``` Let's start with the dot (`.`) notation we use to indicate that we're accessing data or functionality inside an object. You've probably already noticed that there are two kinds of constructions we've been using in our code to do things with variables. There's the functional syntax, where we pass an object as an argument to a function: ```{code-cell} ipython3 len([2, 4, 1, 9]) ``` And then there's the object-oriented syntax that uses the dot notation, which we saw when looking at some of the functionality implemented in strings: ```{code-cell} ipython3 phrase = "aPpLeS ArE delICIous" phrase.lower() ``` If you have some experience in another object-oriented programming language, the dot syntax will be old hat to you. But if you've mostly worked in data-centric languages (e.g., R or Matlab), you might find it puzzling at first. What's happening in the above example is that we're calling a function attached to this object (this is called a "method" of the object) `lower()` *on* the `phrase` string itself. You can think of the dot operator `.` as expressing a relationship of belonging, or roughly translating as "look inside of". So, when we write `phrase.lower()`, we're essentially saying, "call the `lower()` method that's contained inside of `phrase`". (We're being a bit sloppy here for the sake of simplicity, but that's the gist of it.) Note that `lower()` works on strings, but, unlike functions like `len()` and `round()`, it isn't a built-in function in Python. We can't just call `lower()` directly: ```{code-cell} ipython3 lower("TrY to LoWer ThIs!") ``` Instead, it needs to be called via an instance that contains this function, as we did above with `phrase`. +++ Neither is `lower()` a method that's available on *all* objects. For example, this won't work: ```{code-cell} ipython3 num = 6 num.lower() ``` Integers, as it happens, don't have a method called `lower()`. And neither do most other types. But strings do. And what the `lower()` method does, when called from a string, is return a lower-cased version of the string to which it is attached. But that functionality is a feature of the string type itself, and *not* of the Python language in general. Later, we'll see how we go about defining new types (or classes), and specifying what methods they have. For the moment, the main point to take away is that almost all functionality in Python is going to be accessed via objects. The dot notation is ubiquitous in Python, so you'll need to get used to it quickly if you're used to a purely functional syntax. +++ #### Inspecting objects One implication of everything being an object in Python is that we can always find out exactly what data an object contains, and what methods it implements, by inspecting it in various ways. We won't look very far under the hood of objects in this chapter, but it's worth knowing about a couple of ways of interrogating objects that can make your life easier. First, you can always see the type of an object with the built-in `type()` function, which you also saw before: ```{code-cell} ipython3 msg = "Hello World!" type(msg) ``` Second, the built-in `dir()` function will show you all of the methods implemented on an object, as well as *static attributes*, which are variables stored within the object. Be warned that this will often be a long list and that some of the attribute names you see (mainly those that start and end with two underscores) will look a little wonky. We'll talk about those briefly later. ```{code-cell} ipython3 dir(msg) ``` That's a pretty long list! Any name in that list is available to you as an *attribute* of the object (e.g., `my_var.values()`, `my_var.__class__`, etc.), meaning that you can access it and call it (if it is a function) using the dot notation. Notice that the list contains all of the string methods we experimented with earlier (including `lower`), as well as many others. (python_ex7)= #### Exercise Find the methods associated with "`int`" objects. Are they different from the methods associated with "`float`" objects? ## Control flow Like nearly every other programming language, Python has several core language constructs that allow us to control the flow of our code -- the order in which functions get called and expressions are evaluated. The two most common ones are conditionals (if-then statements) and for-loops. ### Conditionals ```{eval-rst} .. index:: single: if statements ``` ```{eval-rst} .. index:: single: elif statements ``` ```{eval-rst} .. index:: single: else statements ``` Conditional (or if-then) statements allow our code to branch -— meaning, we can execute different chunks of code depending on which of two or more conditions is met. For example: ```{code-cell} ipython3 mango = 0.2 if mango < 0.5: print("Mangoes are super cheap; get a bunch of them!") elif mango < 1.0: print("Get one mango from the store.") else: print("Meh. I don't really even like mangoes.") ``` The printed statement will vary depending on the value assigned to the `mango` variable. Try changing that value and see what happens when you re-run the code. Notice that there are three statements in the above code: `if`, `elif` (which in Python stands for "else if"), and `else`. Only the first of these (i.e., `if`) is strictly necessary; the `elif` and `else` statements are optional. (python_ex8)= #### Exercise There can be arbitrarily many `elif` statements. Try adding another one to the code above that executes only in the case that mangos are more expensive than 2.0 and less expensive than 5.0. (forloop)= ### Loops ```{eval-rst} .. index:: single: for loops ``` For-loops allow us to iterate (or loop) over the elements of a collection (e.g., a list) and perform the same operation(s) on each one. As with most things Python, the syntax is quite straightforward and readable: ```{code-cell} ipython3 for elem in random_stuff: print(elem) ``` Here we loop over the elements in the `random_stuff` list. In each iteration (i.e., for each element), we assign the value to the temporary variable `elem`, which only exists within the scope of the `for` statement (i.e., `elem` won't exist once the for-loop is done executing). We can then do whatever we like with `elem`. In this case, we just print its value. #### Looping over a range While we can often loop directly over the elements in an array (as in the above example), it's also very common to loop over a range of integer indices, which we can then use to access data stored in some sequential form in one or more collections. To facilitate this type of thing, we can use Python's built-in `range()` function, which produces a sequence of integers starting from `0` and stopping before the passed value: ```{code-cell} ipython3 num_elems = len(random_stuff) for i in range(num_elems): val = random_stuff[i] print(f"Element {i}: {val}") ``` (python_ex9)= #### Exercise The content that was printed in each iteration of the loop in the last example is formatted using a so-called "f-string". This is a way to compose strings that change based on information from the code surrounding them. An f-string is a string that has the letter "f" before it, as in this example, and it can contain segments enclosed by curly braces (`{` and `}`) that contain Python statements. In this case, the Python statements in each curly bracket are variable names, and the values of the variables at that point in the code are inserted into the string, but you could also insert small calculations that produce a result that then gets inserted into the string at that location. As an exercise, rewrite the code above so that in each iteration through the loop the value of `i` and the value of `i` squared are both printed. Hint: powers of Python numbers are calculated using the `**` operator. ### Nested control flow We can also nest conditionals and for-loops inside one another (as well as inside other compound statements). For example, we can loop over the elements of `random_stuff`, as above, but keeping only the elements that meet some condition—e.g., only those elements that are strings: ```{code-cell} ipython3 # create an empty list to hold the filtered values strings_only = [] # loop over the random_stuff list for elem in random_stuff: # if the current element is a string... if isinstance(elem, str): # ...then append the value to strings_only strings_only.append(elem) print("Only the string values:", strings_only) ``` ### Comprehensions ```{eval-rst} .. index:: single: List comprehension ``` In Python, for-loops can also be written in a more compact way known as a *list comprehension* (there are also dictionary comprehensions, but we'll leave that to you to look up as an exercise). List comprehensions are just a bit of *syntactic sugar* -— meaning, they're just a different way of writing the same code but don't change the meaning in any way. Here's the list comprehension version of the for-loop we wrote above: ```{code-cell} ipython3 p = [print(elem) for elem in random_stuff] ``` We can also embed conditional statements inside list comprehensions. Here's a much more compact way of writing the string-filtering snippet we wrote above: ```{code-cell} ipython3 strings_only = [elem for elem in random_stuff if isinstance(elem, str)] print("Only the string values:", strings_only) ``` List comprehensions can save you quite a bit of typing once you get used to reading them, and you may eventually even find them clearer to read. It's also possible to nest list comprehensions (equivalent to for-loops within for-loops), though that power should be used sparingly, as nested list comprehensions can be difficult to understand. +++ (python_ex10)= #### Exercise Using a comprehension, create a list, where each element is a tuple. The first element in each tuple should be the index of the element in `random_stuff` and the second element of the tuple should be its square. ### Whitespace is syntactically significant One thing you might have noticed when reading the conditional statements and for-loops above is that we always seem to indent our code inside these statements. This isn't a matter of choice; Python is a bit of an odd duck among programming languages, in that it imposes strong rules about how whitespace can be used (i.e., whitespace is *syntactically significant*). This can take a bit of getting used to, but once you do, it has important benefits: there's less variability in coding style across different Python programmers, and reading other people's code is often much easier than it is in languages without syntactically significant whitespace. The main rule you need to be aware of is that whenever you enter a *compound statement* (which includes for-loops and conditionals, but also function and class definitions, as we'll see below), you have to increase the indentation of your code. When you exit the compound statement, you then decrease the indentation by the same amount. The exact amount you indent each time is technically up to you. But it's strongly recommended that you use the same convention everyone else does (described in the Python style guide, known as [PEP8](https://peps.python.org/pep-0008/)), which is to always indent or dedent by 4 spaces. Here's what this looks like in a block with multiple nested conditionals: ```{eval-rst} .. index:: single: PEP8 ``` ```{code-cell} ipython3 num = 800 if num > 500: if num < 900: if num > 700: print("Great number.") else: print("Terrible number.") ``` (python_ex11)= #### Exercise Modify the above snippet so that you (a) consistently use a different amount of indentation (for example, 3 spaces), and (b) break Python by using invalid indentation. ## Namespaces and imports Python is a high-level, dynamic programming language, which people often associated with flexibility and lack of precision (e.g., as we've already seen, you don't have to declare the type of your variables when you initialize them in Python). But in some ways, Python is much more of a stickler than most other dynamic languages about the way Python developers write their code. We just saw that Python is very serious about how you indent your code. Another thing that's characteristic of Python is that it takes *namespacing* very seriously. If you're used to languages like, say, R or MATLAB, you might expect to have hundreds of different functions available to call as soon as you fire up an interactive prompt. By contrast, in Python, the *built-in namespace* -— the set of functions you can invoke when you start running Python -— is [very small](https://docs.python.org/3/library/functions.html). This is by design: Python expects you to carefully manage the code you use, and it's particularly serious about making sure you maintain orderly namespaces. In practice, this means that any time you want to use some functionality that is not built-in and immediately available, you need to explicitly *import* it from whatever module it's currently in, via an `import` statement. This can initially look strange, but once you get used to it, you'll find that it substantially increases code clarity and almost completely eliminates naming conflicts and confusion about where some functionality came from. ### Importing a module ```{eval-rst} .. index:: single: import ``` ```{eval-rst} .. index:: single: Namespace ``` Conventionally, all import statements in a Python file are consolidated at the very top (though there are some situations where this isn't possible). Here's what the most basic usage of `import` looks like: ```{code-cell} ipython3 import json dummy = {'a': 1, 'b': 5} json.dumps(dummy) ``` ```{eval-rst} .. index:: single: Python standard library ``` In this case, we begin by importing a module from Python's *standard library*--i.e., the set of libraries that come bundled with the Python interpreter in every standard installation. Once we import a module, we can invoke any of the functions located inside it, using the dot syntax you see above. In this case, we import the `json` module, which provides tools for converting to and from the JSON format. JSON, which stands for *JavaScript Object Notation*, is a widely-used text-based data representation format. In the above example, we take a Python dictionary and convert (or "dump") it to a JSON string by calling the `dumps()` function. Note that if we hadn't explicitly imported `json`, the `json.dumps()` call would have failed, because `json` would be undefined in our namespace. You can also try directly calling `dumps()` alone (without the `json` prefix) to verify that there is no such function available to you in Python's root namespace. ### Importing from a module Importing a module by name gives us access to all of its internal attributes. But sometimes we only need to call a single function inside a module, and we might not want to have to type the full module name every time we use that function. In that case, we can import *from* the module: ```{code-cell} ipython3 # defaultdict is a dictionary that has default values for new keys. from collections import defaultdict # When initializing a defaultdict, we specify the default type of new values. test_dict = defaultdict(int) # this would fail with a normal dict, but with a defaultdict, # a new key with the default value is created upon first access. test_dict['made_up_key'] ``` In this case, we import `defaultdict` directly *from* the `collections` module *into* our current namespace. This makes `defaultdict` available for our use. Note that `collections` itself is *not* available to us unless we explicitly import it (i.e., if we run `import collections`): ```{code-cell} ipython3 import collections another_test_dict = collections.defaultdict(int) ``` ### Renaming variables at import time Sometimes the module or function we want to import has an unwieldy name. Python's import statements allow us to rename the variable we're importing on the fly using the `as` keyword: ```{code-cell} ipython3 from collections import defaultdict as dd float_test_dict = dd(float) ``` For many commonly used packages, that are strong conventions about naming abbreviations, and you should make sure to respect these in your code. For example, it's standard to see `import numpy as np` and `import pandas as pd` (both are libraries that you will learn about in {numref}`numpy` and {numref}`pandas`, respectively). Your code will still work fine if you use other variable names, but other programmers will have a slightly more difficult time understanding what you're doing. So be kind to others and respect the conventions. ## Functions ```{eval-rst} .. index:: single: Functions ``` Python would be of limited use to us if we could only run our code linearly from top to bottom. Fortunately, as in almost every other modern programming language, Python has *functions*: blocks of code that only run when explicitly called. Some of these are built into the language itself (or contained in the standard library's many modules we can import from, as we saw above): ```{code-cell} ipython3 approx_pi = 3.141592 round(approx_pi, 2) ``` Here we use the `round()` function to round a float to the desired precision (2 decimal places). The `round()` function happens to be one of the few dozen "built-ins" included in the root Python namespace out of the box, but we can easily define our own functions, which we can then call just like the built-in ones. Functions are defined like this: ```{code-cell} ipython3 def print_useless_message(): print("This is a fairly useless message.") ``` Here, we're defining a new function called `print_useless_message`, which, as you might expect, can print a fairly useless message. Notice that nothing happens when we run the above block of code. That's because all we've done is *define* the function; we haven't yet *called* or *invoked* it. We can do that like this: ```{code-cell} ipython3 print_useless_message() ``` ### Function arguments and return values Functions can accept *arguments* (or parameters) that alter their behavior. When we called `round()` above, we passed two arguments: the float we wanted to round, and the number of decimal places we wanted to round it to. The first argument is mandatory in the case of `round()`; if we try calling `round()` without any arguments (feel free to give it a shot), we'll get an error. This should make intuitive sense to you because it would be pretty strange to try to round no value at all. Functions can also explicitly *return* values to the user. To do this, we have to explicitly end our function with a `return` statement, followed by the variable(s) we want to return. If a function doesn't explicitly end with a `return` statement, then the special value `None` we encountered earlier will be returned. Let's illustrate the use of arguments by writing a small function that takes a single float as input, adds Gaussian noise (generated by the standard library's `random` module), and returns the result. ```{code-cell} ipython3 import random def add_noise(x, mu, sd): """Adds gaussian noise to the input. Parameters ---------- x : number The number to add noise to. mu : float The mean of the gaussian noise distribution. sd : float The standard deviation of the noise distribution. Returns ------- float """ noise = random.normalvariate(mu, sd) return (x + noise) ``` The `add_noise()` function has three required parameters: The first (`x`) is the number we want to add noise to. The second (`mu`) is the mean of the Gaussian distribution to sample from. The third (`sd) is the distribution's standard deviation. Notice that we've documented the function's behavior inside the function definition itself using what's called a *[docstring](https://www.python.org/dev/peps/pep-0257/)*. This a good habit to get into, as good documentation is essential if you expect other people to be able to use the code you write (including yourself in the future). In this case, the docstring indicates to the user what the expected type of each argument is, what the argument means, and what the function returns. In case you are wondering why it is organized in just this way, that is because we are following the conventions of docstrings established by the numpy project (and described in the [numpy docstring guide](https://numpydoc.readthedocs.io/en/latest/format.html)). ```{eval-rst} .. index:: single: Software documentation ``` ```{eval-rst} .. index:: single: docstring ``` Now that we've defined our noise-adding function, we can start calling it. Note that because we're sampling randomly from a distribution, we'll get a different output every time we re-run the function, even if the inputs are the same. ```{code-cell} ipython3 add_noise(4, 1, 2) ``` (python_ex12)= #### Exercise Based on the function definition provided above, define a new function that produces a sample of `n` numbers each of which is `x` with Gaussian noise of mean `mu` and standard deviation `std` added to it. The return value should be a list of length `n`, itself a parameter to the function. ### Function arguments ```{eval-rst} .. index:: single: Positional arguments ``` ```{eval-rst} .. index:: single: Key-word arguments ``` Python functions can have two kinds of arguments: *positional* arguments, and *keyword* (or *named*) arguments. #### Positional arguments Positional arguments, as their name suggests, are defined by position, and they *must* be passed when the function is called. The values passed inside the parentheses are mapped one-to-one onto the arguments, as we saw above for `add_noise()`. That is, inside the `add_noise()` function, the first value is referenced by `x`, the second by `mu`, and so on. If the caller fails to pass the right number of arguments (either too few or too many), an error will be generated: ```{code-cell} ipython3 add_noise(7) ``` In this case, the call to the function fails because the function has 3 positional arguments, and we only pass one #### Keyword arguments Keyword arguments are arguments that are assigned a default value in the function *signature* (i.e., the top line of the function definition, that looks like `def my_function(...)`). Unlike positional arguments, keyword arguments are optional: if the caller doesn't pass a value for the keyword argument, the corresponding variable will still be available inside the function, but it will have whatever value is defined as the default in the signature. To see how this works, let's rewrite our `add_noise()` function so that the parameters of the gaussian distribution are now optional: ```{code-cell} ipython3 def add_noise_with_defaults(x, mu=0, sd=1): """Adds gaussian noise to the input. Parameters ---------- x : number The number to add noise to. mu : float, optional The mean of the gaussian noise distribution. Default: 0 sd : float, optional The standard deviation of the noise distribution. Default: 1 Returns ------- float """ noise = random.normalvariate(mu, sd) return x + noise ``` This looks very similar, but we can now call the function without filling in `mu` or `sd`. If we don't pass in those values explicitly, the function will internally use the defaults (i.e., `0` in the case of `mu`, and `1` in the case of `sd`). Now, when we call this function with only one argument: ```{code-cell} ipython3 add_noise_with_defaults(10) ``` Keyword arguments don't have to be filled in order, as long as we explicitly name them. For example, we can specify a value for `sd` but not for `mu`: ```{code-cell} ipython3 # we specify x and sd, but not mu add_noise_with_defaults(5, sd=100) ``` Note that if we didn't specify the name of the argument (i.e., if we called `add_noise_with_defaults(5, 100)`, the function would still work, but the second value we pass would be interpreted as `mu` rather than `sd` because that's the order they were introduced in the function definition. It's also worth noting that we can always explicitly name *any* of our arguments, including positional ones. This is extremely handy in cases where we're calling functions whose argument names we remember, but where we don't necessarily remember the exact order of the arguments. For example, suppose we remember that `add_noise()` takes the three arguments `x`, `mu`, and `sd`, but we don't remember if `x` comes before or after the distribution parameters. We can guarantee that we get the result we expect by explicitly specifying all the argument names: ```{code-cell} ipython3 add_noise(mu=1, sd=2, x=100) ``` To summarize, functions let us define a piece of code that can be called as needed and reused. We can define a default behavior and override it as necessary. There is a bit more to it, of course. For a few more details, you can go into more depth in {numref}`argument_unpacking`, or skip forward to {numref}`classes`. (argument_unpacking)= #### Argument unpacking with \*args and \*\*kwargs** ```{eval-rst} .. index:: single: Argument unpacking ``` It sometimes happens that a function needs to be able to accept an unknown number of arguments. A very common scenario like this is where we've written a "wrapper" function that takes some input, does some operation that relies on only some of the arguments the user passed, and then hands off the rest of the arguments to a different function. For example, suppose we want to write an `arg_printer()` function that we can use to produce a standardized display of the positional and keyword arguments used when calling some other *arbitrary* function. Python handles this scenario elegantly via special `*args` and `**kwargs` syntax in function signatures, also known as *argument unpacking*. The best way to understand what `*args` and `**kwargs` do is to see them in action. Here's an example: ```{code-cell} ipython3 def arg_printer(func, *args, **kwargs): """ A wrapper that takes any other function plus arguments to pass to that function. The arguments are printed out before the function is called and the result returned. Parameters ---------- func : callable The function to call with the passed args. args : list, optional List of arguments to pass into func. kwargs : dict, optional Dict of keyword arguments to pass into func. Returns ------- The result of func() when called with the passed arguments. """ print("Calling function:", func.__name__) print("Positional arguments:", args) print("Keyword arguments:", kwargs) return func(*args, **kwargs) ``` This may seem a bit mysterious, and there are parts we won't explain right now (e.g., `func.__name__`). But try experimenting a bit with calling this function, and things may start to click. Here's an example to get you rolling ```{code-cell} ipython3 arg_printer(add_noise, 17, mu=0, sd=5) ``` What's happening here is that the first argument to `arg_printer()` is the `add_noise()` function we defined earlier. Remember: everything in Python is secretly an object--even functions! You can pass functions as arguments to other functions too; it's no different than passing a string or a list. A key point to note, however, is that what we're passing in is, in a sense, the *definition* of the function. Notice how we didn't add parentheses to `add_noise` when we passed it to `arg_printer()`? That's because we don't want to call `add_noise` yet; we're leaving it to the `arg_printer()` function to do that internally. All the other arguments to `arg_printer()` after the first one are arguments that we actually want to pass onto the `add_noise()` function when it's called internally by `arg_printer()`. The first thing `arg_printer()` does is print out the name of the function we just gave it, as well as all of the positional and keyword arguments we passed in. Once it's done that, it calls the function we passed in (`add_noise()`) and passes along all the arguments. If the above doesn't make sense, don't worry! As we mentioned above, we're moving quickly, and the concepts from here on out start to get quite a bit denser. A good way to explore these ideas a bit better is to write your own code and experiment with things until they start to make sense. (python_ex13)= #### Exercise Replace `add_noise` with built-in functions like `min`, `len`, or `list`. What other arguments do you need to change? (classes)= ## Classes The material we've covered so far in this chapter provides a brief overview of the most essential concepts in Python and should be sufficient to get you started reading and writing code in the language. If you're impatient to start working with scientific computing libraries and playing with neuroimaging data, you could stop here and move on to the next chapter. That said, there are some concepts we haven't talked about yet that are quite central to understanding how Python really works under the hood, and you'll probably get more out of this book if you understand them well. They are, however, quite a bit more difficult conceptually. In particular, the object-oriented programming (OOP) paradigm, and Python's internal data model (which revolves around something called *magic methods*), are not very intuitive, and it can take some work to wrap your head around them if you haven't seen OOP before. We'll do our best to introduce these concepts here, but don't be surprised or alarmed if they don't immediately make sense to you -- that's completely normal! You'll probably find that much of this material starts to "click" as you work your way through the various examples in this book and start to write your own Python analysis code. ```{eval-rst} .. index:: single: Object-oriented programming ``` ```{eval-rst} .. index:: single: Classes ``` We've said several times now that everything in Python is actually an object. We're now in a position to unpack that statement. What does it mean to say that something is an object? How do objects get defined? And how do we specify what an object should do when a certain operation is applied to it? To answer these questions, we need to introduce the notion of *classes* -- and, more generally, the *object-oriented programming* (OOP) paradigm. ### What is a class? A class is, in a sense, a kind of template for an object. You can think of it as a specification or a set of instructions that determine what an object of a given kind can do. In a sense, it's very close in meaning to what we've already been referring to as the *type* of an object. There *is* technically a difference between types and classes in Python, but it's quite subtle, and in day-to-day usage, you can use the terms interchangeably and nobody is going to yell at you. ### Defining classes So a class is a kind of template; okay, what does it look like? Well, minimally, it looks like very little. Here's a fully functional class definition: ```{code-cell} ipython3 class Circle: pass ``` That's it! We've defined a new Python class. In case you're wondering, the `pass` statement does nothing -- it is used as a placeholder that tells the Python interpreter it shouldn't expect any more code to follow. ### Creating instances ```{eval-rst} .. index:: single: Class instance ``` You might think that this empty `Circle` class definition isn't very interesting because we obviously can't *do* anything with it. But that isn't entirely true. We can already *instantiate* this class if we like -- which is to say, we can create new objects whose behavior is defined by the `Circle` class. A good way to think about it is in terms of an "X-is-a-particular-Y" relationship. If we were to draw 3 different circles on a piece of paper, we could say that each one is an *instance* of a circle, but none of them would be the actual *definition* of a circle. That should give you an intuitive sense of the distinction between a class definition and instances of that class. The syntax for creating an instance in Python is simple: ```{code-cell} ipython3 my_circle = Circle() ``` That's it again! We now have a new `my_circle` variable on our hands, which is an instance of class `Circle`. If you don't believe that, it's easy to prove: ```{code-cell} ipython3 type(my_circle) ``` In case you are wondering, the reason that this appears to be a `__main__.Circle` object is that while we are running the code in this notebook, it is defined within a namespace (yes, namespaces again) called `__main__`, and the `type` function recognizes this fact. You can learn more about this particular namespace in the [Python documentation](https://docs.python.org/3/library/__main__.html) #### A note on nomenclature You may have already noticed the naming convention we've used throughout this tutorial: our variable names are always composed of lower-case characters, with words separated by underscores. This is called *snake_case*. You'll also note that class names are capitalized (technically, they're in *CamelCase*). Both of these are standard conventions in the Python community, and you should get in the habit of following both. ### Making it do things The `Circle` class definition we wrote above was perfectly valid, but not terribly useful. It didn't define any new behavior, so any instances of the class we created wouldn't do anything more than base objects in Python can do (which isn't very much). Let's fix that by filling in the class a bit. ```{code-cell} ipython3 from math import pi class Circle: def __init__(self, radius): self.radius = radius def area(self): return pi * self.radius**2 ``` There's not much code to see here, but conceptually, a lot is going on. Let's walk through this piece by piece. We start by importing the variable `pi` from the standard library `math` module. Then, we start defining the class. First, observe that we've defined what looks like two new functions inside the class. Technically, these are *methods* and not functions, because they're *bound* to a particular object. But the principle is the same: they're chunks of code that take arguments, do some stuff, and then (possibly) return something to the caller. You'll also note that both methods have `self` as their first argument. This is a requirement: all instance methods have to take a reference to the current instance (conventionally named `self`) as their first argument (there are also *class* methods and *static* methods, which behave differently, but we won't cover those). This reference is what will allow us to easily *maintain state* (that is, to store information that can vary over time) inside the instance. Now let's walk through each of the defined methods. First, there's `__init__()`. The leading and trailing double underscores indicate that this is a special kind of method called a *magic* method; we'll talk about those a bit more later. For the moment, we just have to know that `__init__()` is a special method that gets called whenever we create a new instance of the class. So, when we write a line like `my_circle = Circle()`, what happens under the hood is that the `__init__()` method of `Circle` gets executed. ```{eval-rst} .. index:: single: __init__ ``` Observe that, in this case, `__init__()` takes a single argument (other than `self`, that is): a `radius` argument. And further, the only thing that happens inside `__init__()` is that we store the value of the input argument `radius` in an *instance attribute* called `radius`. We do this by assigning to `self.radius`. Remember: `self` is a reference to the current instance, so the newly-created instance that's returned by `Circle()` will have that `radius` value set in the `.radius` attribute. This code should make this a bit clearer: ```{code-cell} ipython3 my_circle = Circle(4) print(my_circle.radius) ``` Next, let's look at the `area()` method. This one takes no arguments (again, `self` is passed automatically; we don't need to, and shouldn't, pass it ourselves). That means we can just call it and see what happens. Let's do that: ```{code-cell} ipython3 my_circle.area() ``` When we call `area()`, what we get back is the area of our circle—based on the radius stored in the instance at that moment. Note that this area is only computed when we call `area()`, and isn't computed in advance. This means that if the circle's radius changes, so too will the result of `area()`: ```{code-cell} ipython3 my_circle.radius = 9 print(my_circle.area()) ``` (python_ex14)= #### Exercises 1. Add to the implementation of `Circle` another method that calculates the circumference of the circle. 2. Implement a class called `Square` that has a single attribute `.side`, and two methods for `area` and `circumference`. ### Magic methods ```{eval-rst} .. index:: single: Magic methods ``` There's a *lot* more we could say about how classes work in Python, and about object-oriented programming in general, but this is just a brief introduction, so we have to be picky. Let's introduce just one other big concept: *magic methods*. The concepts we'll discuss in this last section are fairly advanced, and aren't usually discussed in introductions to Python -- so, as we've said several times now, don't worry if they don't immediately click for you. We promise you'll still be able to get through the rest of the book just fine! The reason we decided to cover magic methods here is that, once you *do* understand them well, you'll have a substantially deeper grasp on how Python works, and you'll probably see patterns and make connections that you otherwise wouldn't. Magic methods of objects, as we've seen a couple of times now, start and end with a double underscore: `__init__`, `__getattr__`, `__new__`, and so on. As their names suggest, these methods are magic -— at least in the sense that they appear to add some magic to a class's behavior. We've already talked about `__init__`, which is a magic method that gets called any time we create a new instance of a class. But there are many others. The key to understanding how magic methods work is to recognize that they're usually called implicitly when a certain operation is applied to an object -— even if it doesn't look like the magic method and the operation being applied have anything to do with each other (that's what makes them magic!). Remember how we said earlier that *everything in Python is an object*? Well, now we're going to explore one of the deeper implications of that observation, which is that *all operators in Python are just cleverly-disguised method calls*. That means that when we write even an expression as seemingly basic as `4 * 3` in Python, it's implicitly converted to a call to a magic method on the first operand (`4`), with the second operand (`3`) being passed in as an argument. This is a bit hard to explain abstractly, so let's dive into an example. Start with this naive arithmetic operation: ```{code-cell} ipython3 4 * 3 ``` No surprises there. But here's an equivalent way to write the same line, which makes clearer what's happening under the hood when we multiply one number by another: ```{code-cell} ipython3 # 4 is a number, so we have to wrap it in parentheses to prevent a syntax error. # but we wouldn't have to do this for other types of variables (e.g., strings). (4).__mul__(3) ``` Remember the dot notation? Here, `__mul__` is a (magic) method implemented in the integer class. When Python evaluates the expression `4 * 3`, it calls `__mul__` on the first integer, and hands it the second one as an argument. See, we weren't messing around when we said *everything is an object in Python*. Even something as seemingly basic as the multiplication operator is an alias to a method called on an integer object. ### The semantics of * Once we recognize that Python's `*` operator is just an alias to the `__mul__` magic method, we might start to wonder if this is *always* true. Does every valid occurrence of `*` in Python code imply that the object just before the `*` must be an instance of a class that implements the `__mul__` method? The answer is yes! The result of an expression that includes the `*` operator (and for that matter, every other operator in Python, including things like `==` and `&`) is entirely dependent on the receiver object's implementation of `__mul__`. Just to make it clear how far-reaching the implications of this principle are, let's look at how a couple of other built-in Python types deal with the `*` operator. Let's start with strings. What do you think will happen when we multiply a string object by 2? ```{code-cell} ipython3 "apple" * 2 ``` There's a good chance this was *not* the behavior you expected. Many people intuitively expect an expression like `"apple" * 2` to produce an error, because we don't normally think of strings as a kind of thing that can be multiplied. But remember: in Python, the multiplication operator is just an alias for a `__mul__` call. And there's no particular reason a string class *shouldn't* implement the `__mul__` method; why not define *some* behavior for it, even if it's counterintuitive? That way users have a super easy way to repeat strings if that's what they want to do. What about a list? ```{code-cell} ipython3 random_stuff * 3 ``` List multiplication behaves a lot like string multiplication: the result is that the list is repeated $n$ times. What about dictionary multiplication? ```{code-cell} ipython3 fruit_prices * 2 ``` Finally, we encounter an outright failure! It appears Python dictionaries can't be multiplied. This presumably means that the `dict` class doesn't implement `__mul__` (you can verify this for yourself by inspecting any dictionary using `dir()`). ### Other magic methods Most of the magic methods in Python do something very much like what we saw for the multiplication operator. Consider the following operators: `+`, `&`, `/`, `%`, and `<`. These map, respectively, onto the magic methods `__add__`, `__and__`, `__truediv__`, `__mod__`, and `__lt__`. Many others follow the same pattern. There are also magic methods that are tied to built-in functions rather than operators (for example, when you call `len(obj)`, that's equivalent to calling `obj.__len__`), or that are triggered by certain events (for example, `__getattr__` is called when a requested attribute isn't found in an object). In practice, you won't need to know much about magic methods unless you start writing a lot of classes of your own (but you can find a full description of all the magic methods; you can peruse the [official Python docs for that](https://docs.python.org/3/reference/datamodel.html).). We spent a lot of time talking about them mainly because they're a good way to convey some deep insights about the data model at the core of the Python language. ### Hungry Circles Let's come full circle now (awful pun intended) and revisit the `Circle` class we defined earlier. The last thing we'll do in this chapter is to add a magic method to our `Circle` class. This will nicely tie together a lot of different threads we've covered. What we're going to do is give instances of class `Circle` the ability to "eat" other circles. When given Python code like this: ```python c1 = Circle(4) c2 = Circle(2) c1 * c2 ``` we want the first circle to "grow" its radius by exactly the amount required for its new area to equal the sum of the two circles' previous areas. Here's our updated implementation: ```{code-cell} ipython3 from math import pi, sqrt class Circle: def __init__(self, radius): self.radius = radius def __mul__(self, prey): new_area = self.area() + prey.area() self.radius = sqrt(new_area / pi) def area(self): return pi * self.radius**2 ``` The only change here is the addition of the `__mul__` method. Let's see if the above did what we wanted: ```{code-cell} ipython3 c1 = Circle(4) c2 = Circle(2) # Now the important part: c1 eats c2! c1 * c2 ``` Well, we didn't get an error, so that's a good sign. Let's inspect `c1` and see if it's been updated as we expect. Remember: we expect `c1` to have "eaten" `c2`, which means its radius should grow, and its area should be the sum of both previous areas. ```{code-cell} ipython3 print("Radius of c1 after gorging on c2:", c1.radius) print("Area of c1 after gorging on c2:", c1.area()) ``` It worked! The only slightly dissatisfying feature of our implementation is that, after `c1` eats `c2` and expands itself accordingly, `c2` is somehow still around to tell the tale. This probably violates some physical conservation law, but we'll overlook that here. For reasons we won't get into, it's not trivial to delete `c2` from inside `c1`. (There are good reasons for this, and the fact that we can't easily make some of our circles wink out of existence from inside the belly of other circles might lead us to suspect we've architected our code suboptimally. But that's a problem for a different book.) (python_ex15)= #### Exercise Add a `__mul__` method to your implementation of the `Square` class that follows the same principles as the `__mul__` method of the `Circle` class, changing both the `side` attribute of the calling object, as well as the return value of `area` as it swallows the other object. ### Additional resources This chapter provided a high-level look at some of the main features of the Python language—some basic, some more advanced. To develop a stronger working familiarity with the language, you will need to roll up your sleeves and start writing some code. One of the best ways to learn is to pick a small problem that interests or matters to you in some way (e.g., parsing some text data you have lying around), and search the web for help every time you run into problems (there's no shame in consulting the internet! All programmers do it!). If you prefer to have more structure than that, there are hundreds of excellent, and mostly free, resources online to help you on your way. A couple of good ones include: 1. [A Whirlwind Tour of Python](http://www.oreilly.com/programming/free/files/a-whirlwind-tour-of-python.pdf) is an excellent intro to Python by Jake VanderPlas; Jupyter notebooks are available in [the book GitHub repo](https://github.com/jakevdp/WhirlwindTourOfPython). 2. Allen Downey's ["Think Python"](http://greenteapress.com/wp/think-python-2e/) is another excellent introduction to the language.