# Advanced Python Concepts¶

In this tutorial, a few advanced concepts are introduced. This includes

- Python sets
- packing and unpacking
- context manager
- decorator and factories
- Exceptions

## Python sets¶

Python offers a lot of built-in functionality. Next to the most famous containers - tuples, lists, dictionaries - there are also sets (and more).

Sets act like a mathematical set. They are unordered and cannot contain duplicates.

They can be created with the `set()` function or with the `{}` syntax (but using single entries without colon - otherwise we would get a dictionary).

In [1]:

    set1 = {1, 2, 2, "a"}
    list2 = [2, "a", "b", 4, 4]
    set2 = set(list2)
    print(f"set1: {set1}\nset2: {set2}\nlist2: {list2}")

    set1: {1, 2, 'a'}
    set2: {'b', 2, 4, 'a'}
    list2: [2, 'a', 'b', 4, 4]

We see that only unique entries are stored in the set and that the order of the entries is not preserved.

More elements can be added (inplace) to a set with the `add()` method.

In [2]:

    set1.add(3)
    print(f"set1: {set1}")

    set1: {1, 2, 3, 'a'}

In [3]:

    set1.remove(3)
    print(f"set1: {set1}")

    set1: {1, 2, 'a'}

In [4]:

    # multiple elements
    set1.update([4, 5, 6])

In [5]:

    set1

Out[5]:

    {1, 2, 4, 5, 6, 'a'}

### Set operations¶

Sets can be used to perform mathematical set operations. The following operations are supported:

- `union`: returns the union of two sets (alternative syntax `A | B`)
- `intersection`: returns the intersection of two sets (alternative syntax `A & B`)
- `difference`: returns the difference of two sets (alternative syntax `A - B`)
- `symmetric_difference`: returns the symmetric difference (xor) of two sets (alternative syntax `A ^ B`)

All of this methods return a new set, so the original sets are not modified. They all have a `_update` version, which modifies the original set inplace. (Exception: `union_update` is simply `update` as seen above).

Furthermore, many tests can be performed on sets, such as:

- `element in set`: checks if an element is in the set (for example `"a" in set1`)
- `issubset`: checks if a set is a subset of another set (alternative syntax `A <= B`, for proper subset `A < B`)
- `issuperset`: checks if a set is a superset of another set (alternative syntax `A >= B`, for proper superset `A > B`)
- `isdisjoint`: checks if two sets have no common elements (alternative syntax `A /= B`)
- `isempty`: checks if a set is empty
- and more...

In [6]:

    print(set1)
    print(set2)
    set1.intersection(set2)

    {1, 2, 4, 5, 6, 'a'}
    {'b', 2, 4, 'a'}

Out[6]:

    {2, 4, 'a'}

In [7]:

    2 in set1

Out[7]:

    True

### Immutable types only¶

As for dictionary keys, only immutable types can be used as keys in a dictionary (i.e. strings, numbers, tuples, ...). Otherwise, a `TypeError` is raised.

In [8]:

    set3 = {list2, 5, 3}  # Fails with a `TypeError`

    ---------------------------------------------------------------------------
    TypeError                                 Traceback (most recent call last)
    Cell In[8], line 1
    ----> 1 set3 = {list2, 5, 3}  # Fails with a `TypeError`

    TypeError: unhashable type: 'list'

In [9]:

    set4 = {set2, 5, 3}  # Fails with a `TypeError`

    ---------------------------------------------------------------------------
    TypeError                                 Traceback (most recent call last)
    Cell In[9], line 1
    ----> 1 set4 = {set2, 5, 3}  # Fails with a `TypeError`

    TypeError: unhashable type: 'set'

In [10]:

    set5 = {tuple(list2), (4, 3), 3}
    print(f"set5: {set5}")

    set5: {3, (2, 'a', 'b', 4, 4), (4, 3)}

In [11]:

    (4, 3) in set5

Out[11]:

    True

## Packing and unpacking of values¶

In [12]:

    # packing
    tuple_ = 2,3,5,7
    tuple_

Out[12]:

    (2, 3, 5, 7)

In [13]:

    # unpacking
    a, b, c, d = tuple_
    print(c)

    5

Using `*` we can group/ungroup list-like objects. They act as a "removal" of the parenthesis when situated on the right and as an "adder" of parenthesis when situated on the left of the assigmenemt operator (`=`).

Let's play around...

In [14]:

    a, c, *b = [3, 4, 4.5, 5, 6]

In [15]:

    b

Out[15]:

    [4.5, 5, 6]

As can be seen, b catches now all the remaining elements in a list. Interesting to see is also the special case if no element is left.

In [16]:

    d1, d2, *d3, d4 = [1, 2, 3]  # nothing left for d3

In [17]:

    d3

Out[17]:

    []

This is simply an empty list. However, this has the advantage that we know that it is always a list.

Multiple unpackings can be added together (however, the other way around does not work: multiple packings are not possible as it is ill-defined which variable would get how many elements).

In [18]:

    a = [3, 4, 5]

In [19]:

    d, e, f, g, h, i = *a, *b

Now we should be able to understand the `*args` and `**kwargs` for functions. `*args` works like the examples above. `**kwargs` works very similar, but exists only in parameter-lists of functions. There it packs/unpacks keyword-arguments into/from a dictionary. Let's look at it:

In [20]:

    def func(*args, **kwargs):
        print(f'args are {args}')
        print(f"kwargs are {kwargs}")

In [21]:

    mykwargs = {'a': 5, 'b': 3}
    myargs = [1, 3, 4]
    func(*myargs, **mykwargs)

    args are (1, 3, 4)
    kwargs are {'a': 5, 'b': 3}

In [22]:

    func(5, a=4)

    args are (5,)
    kwargs are {'a': 4}

In [23]:

    def func(a, x, **kwargs):
        print(x, a, kwargs)

    kwargs = {"x": 2}
    kwargs['y'] = 3
    func(1,**kwargs)
    func(x=3, a=5)

    2 1 {'y': 3}
    3 5 {}

In [24]:

    # play around with it!

### Aside: do not confuse the different parameter/argument types¶

#### function definition¶

- required parameters: specified without a default value: `def f(p)`
- optional parameters: have a default value: `def f(p=42)`

#### calling a function¶

- positional arguments: are given without a keyword `f(0)`
- key-word arguments are passed using a keyword `f(p=0)`

Both requried and optional parameters can be passed as positional or key-word arguments!

(for the curious: there is a bit more that can be done)

## Context manager¶

A context manager is an object that responds to a `with` statement. It may returns something. The basic idea is that some action is performed when entering a context and again when exiting it.

    with context as var:
        # do something

translates to

    # execute context entering code
    var = result_from_context_entering_code
    # do something
    # execute context leaving code

The great advantage here is that the "leaving code" is automatically executed whenever we step out of the context!

This proved to be incredibly useful when operations have cleanup code that we need to execute yet that is tedious to write manually and can be forgotten.

### Using a class¶

We can have control over the enter and exit methods by creating a class and implementing the two methods `__enter__` and `__exit__`

In [25]:

    class MyContext:

        def __init__(self, x):
            self.x = x
            print('initialized')

        def __enter__(self):
            x = self.x
            print('entered')
            return x**2

        def __exit__(self, type_, value, traceback):  # but let's not go into things in detail here
            self.x = None
            print('exited')

In [26]:

    context_5 = MyContext(5)

    initialized

In [27]:

    with context_5 as value:
        print("inside")
        print(value)

    print("Outside")
    print(value)
    print("context_5.x:", context_5.x)

    entered
    inside
    25
    exited
    Outside
    25
    context_5.x: None

In [28]:

    context_5 = MyContext(5)
    xsquare = context_5.__enter__()
    print("inside", xsquare)
    #context_5.__exit__(...)

    initialized
    entered
    inside 25

#### Where is this useful¶

Basically with stateful objects. This includes anything that can be set and changed (mutable objects).

In [29]:

    with open('python_tricks.ipynb') as notebook:
        print("".join(notebook.readlines()[:5]))

    {
     "cells": [
      {
       "cell_type": "markdown",
       "metadata": {

The implementation roughly looks like this:

In [30]:

    class MyOpen:

        def __init__(self, f, mode):
            self._file = f
            self._mode = mode

        def __enter__(self):
            return self._file.open(self._mode)

        def __exit__(self, type_, value, traceback):  # but let's not go into things in detail here
            self._file.close()

Exercise: create a context manager that temporarily sets a `'value'` key to 42 of a dict and switches it back to the old value on exit

In [31]:

    testdict = {'value': 11, 'name': 'the answer'}

to be invoked like this

    with manager(testdict) as obj:
        # here the value is 42
    # here the value is 11

## Generators¶

Generators are special functions that use `yield`.

What is `yield`?: It's like a return, except that the execution stops at the `yield`, lets other code execute and, at some point, continues again where the yield was. Examples are:

- iterator: a function that yields elements. Everytime it is called, it is supposed to yield an element and then continue from there
- asynchronous programing: it stops and waits until something else is finished
- in the context manager, as we will see

In [32]:

    def squares(start=0, num=100):
        i = start
        while i < start+num:
            yield i**2
            i += 1

In [33]:

    gen = squares()
    print(next(gen))
    print(next(gen))

    0
    1

In [34]:

    for v in squares(5,10):
        print(v)

    25
    36
    49
    64
    81
    100
    121
    144
    169
    196

## Function factories and Decorators¶

Sometimes we can't write a function fully by hand but want to create it programatically. This pattern is called a "factory". To achieve this, instead of having a function that returns an integer (an object), a list (an object), a dict (an object) or an array (an object), we return a function (an object). We see that the concept of Python, "everything is an object", starts being very useful here.

In [35]:

    # goal: create a power function but let the power be specified by the code
    # exponent = 2

    # def power_2(x):  # not 2, but "exponent"
    #     return x ** exponent

In [36]:

    def make_power_func(power):
        def func(x):
            print("Executed")
            return x ** power

        return func

In [37]:

    pow3 = make_power_func(3)

In [38]:

    pow3(2)

    Executed

Out[38]:

    8

### Pitfall lexical lookup¶

The name `power` inside the function `func` is remembered by the name (lexical) and NOT BY REFERENCE. This means that the last object named `power` is the one that is used inside `func`.

In [39]:

    def make_power_func(power):
        def func(x):
            return x ** power

        power = 42
        return func
    power = 1

In [40]:

    pow3 = make_power_func(3)

In [41]:

    pow3(2)

Out[41]:

    4398046511104

Another example is to create a timing wrapper. Exercise: create a timing function that can be used as follows

    timed_pow3 = fime_func(pow3)
    pow3(...)

HINT, scetch of solution

    def time_func(func):
        def new_func(...):
            print('start')
            func(...)
            print('stop')
        return new_func

In [42]:

    import time

    # SOLUTION
    def timed_func(func):
        def wrapped_func(*args, **kwargs):
            start = time.time()
            res = func(*args, **kwargs)
            end = time.time()
            print(f'time needed: {end - start}')
            return res

        return wrapped_func

In [43]:

    def add_notime(x, y):
        return x + y

In [44]:

    add_timed = timed_func(add_notime)

In [45]:

    add_timed(y=4, x=5)

    time needed: 9.5367431640625e-07

Out[45]:

    9

In [46]:

    add_timed(5, y=4)

    time needed: 1.1920928955078125e-06

Out[46]:

    9

### Decorator¶

There is another way, just syntactical sugar, to make this automatic: a decorator. It is invoked as below

In [47]:

    @timed_func
    def add(x, y):
        return x + y

Again, as for the contextmanager, we can also use a class here to give more flexibility and create a decorator that takes arguments.

## ADVANCED ONLY: contect-manager, function decorators and yield all together¶

One "easy" way to create a context manager is to have a function that has a `yield` and decorate it with `@contextlib.contextmanager`

In [48]:

    # ADVANCED ONLY
    import contextlib

    @contextlib.contextmanager
    def printer(x):
        print(f'we just entered the context manager and will yield {x}')
        yield x
        print(f'Finishing the context manager, exiting')

In [49]:

    with printer(5) as number:
        print(f"we're inside, with number={number}")
    print("left manager")

    we just entered the context manager and will yield 5
    we're inside, with number=5
    Finishing the context manager, exiting
    left manager

## Exceptions¶

Exceptions are used to stop the execution at a certain point and surface to higher stacks in the code, e.g. to go up in the call stack. A typical use-case is when an error is encountered, such as the wrong type of object is given. Exceptions can also be caught in a `try ... except ...` block in order to handle the exception.

There are a few built-in exceptions, the most common ones are:

- `TypeError`: object has the wrong type, e.g. string instead of float
- `ValueError`: the value of the object is illegal, e.g. negative but should be positive
- `RuntimeError`: if a function is illegally executed or a status is wrong. E.g. if an object first has to be loaded before it gets parsed. It covers any error that does not fall into an other category.
- `KeyError`, `IndexError`: if a key or index is not available, e.g. in a `dict` or `list`

An Exception can manually be raised by

In [50]:

    raise TypeError("Has to be int, not str")

    ---------------------------------------------------------------------------
    TypeError                                 Traceback (most recent call last)
    Cell In[50], line 1
    ----> 1 raise TypeError("Has to be int, not str")

    TypeError: Has to be int, not str

Note that it is often convenient to create an instance such as in the example above where the first argument is the message (as we see in the raised Exception above), but we can also raise an exception by only using the class itself

In [51]:

    raise TypeError

    ---------------------------------------------------------------------------
    TypeError                                 Traceback (most recent call last)
    Cell In[51], line 1
    ----> 1 raise TypeError

    TypeError:

### Custom Exception¶

In Python, exceptions are simply a class. And as such, we can inherit from it and create our own exception.

Attention: inherit from `Exception` or subclasses of it such as `TypeError`, `ValueError`, but NEVER from `BaseException`.

In [52]:

    class MyError(Exception):
        pass

In [53]:

    raise MyError("Hello world")

    ---------------------------------------------------------------------------
    MyError                                   Traceback (most recent call last)
    Cell In[53], line 1
    ----> 1 raise MyError("Hello world")

    MyError: Hello world

An exception can also be created by inheriting from an already existing exception if it is more specific and provides hints on the nature of the exception.

In [54]:

    class NegativeValueError(ValueError):
        pass

### Catching exceptions¶

An exception can be caught in a `try..except` block. This works as follows:

- if an exception is raised in the `try` block, the next `except` is invoked
- it is tested whether the raised exception is of type subclass of the exception type specified to be caught. For example, `except TypeError` checks if the raised error is of type `TypeError` or a subclass of it.
- if that is not the case, it goes to the next `except` statement (yes, there can be multiple)
- ... more below

In [55]:

    try:
        raise NegativeValueError("Negative value encountered")
    except ValueError as error:
        print(f"Caught {error}")

    Caught Negative value encountered

By using the `as` keyword, the error that is raised is assigned to a variable. We can inspect the error now if we want or, as above, just print it.

If no error is specified, any error is caught (this should NOT be used, except for special cases)

In [56]:

    try:
        raise TypeError
    # Anti-pattern, do NOT use in general!
    except:  # any exception if not specified
        pass

In [57]:

    try:
        raise TypeError("Type was wrong, unfortunately")

    except TypeError as error:  # any exception
        print(f'caught TypeError: {error}')
    except ValueError as error:
        print(f'caugth ValueError: {error}')

    caught TypeError: Type was wrong, unfortunately

To continue from above: after the last `except`, an `else` statement is looked for. The `else` is executed if no exception was raised.

In [58]:

    # comment and uncomment
    try:
    #     print('no error raised')
        raise TypeError("Type was wrong, unfortunately")
    #     raise ValueError("Value was wrong, ")
    except TypeError as error:  # any exception
        print(f'caught Type {error}')
    except ValueError as error:
        print(f'caugth Value: {error}')
    else:
        print("No error was caught")

    print("Executed after block")

    caught Type Type was wrong, unfortunately
    Executed after block

...and finally, after the else, a `finally` block is looked for. This is guaranteed to be executed! Whether an exception is raised, whether it is caught or not, whether there is an `else` or not, the `finally` is always executed.

Therefore it is suitable for any cleanup code such as closing files, removing temporary files and more.

In [59]:

    try:
        #     pass
        #     raise TypeError("Type was wrong, unfortunately")
        raise RuntimeError("Type was wrong, unfortunately")
    except TypeError as error:  # any exception
        print(f'caught Type {error}')
    except ValueError as error:
        print(f'caugth Value: {error}')
    else:
        print("No error was caught")
    finally:  # POWERFUL! Guarantied to be executed
        print('Finally run')
    print("Executed when passed")

    Finally run

    ---------------------------------------------------------------------------
    RuntimeError                              Traceback (most recent call last)
    Cell In[59], line 4
          1 try:
          2     #     pass
          3     #     raise TypeError("Type was wrong, unfortunately")
    ----> 4     raise RuntimeError("Type was wrong, unfortunately")
          5 except TypeError as error:  # any exception
          6     print(f'caught Type {error}')

    RuntimeError: Type was wrong, unfortunately

Note that in the above example, the error was not caught! All the other statements could also be omitted and only a `try...finally` block can be created.

Typical usecase: cleanup (temporary file removal)

In [60]:

    try:
        raise ValueError
    finally:
        print('raised')

    raised

    ---------------------------------------------------------------------------
    ValueError                                Traceback (most recent call last)
    Cell In[60], line 2
          1 try:
    ----> 2     raise ValueError
          3 finally:
          4     print('raised')

    ValueError:

### pitfall "guaranteed execution"¶

As the `finally` is guaranteed to be executed, this can have an odd effect: possible return statements can be ignored before the finally IF the `finally` also has a return statement. The logic says here that the `finally` return must be executed, as it is guaranteed to be executed.

In [61]:

    def func(x):
        try:
            if x == 5:
                raise RuntimeError('called inside func')
        except RuntimeError as error:
            return error
        else:
            print('else before 42')
            return 42
            print('after else 42')
        finally:
            print("cleaned up")
            return 11

In [62]:

    result = func(6)
    print(f"Result: {result}")

    else before 42
    cleaned up
    Result: 11

In [63]:

    try:
        raise ValueError
    except ValueError as error:
        print("raising")
        raise RuntimeError from None
    finally:
        raise TypeError
        print("finally")
    print("Unreachable")

    raising

    ---------------------------------------------------------------------------
    RuntimeError                              Traceback (most recent call last)
    Cell In[63], line 5
          4     print("raising")
    ----> 5     raise RuntimeError from None
          6 finally:

    RuntimeError:

    During handling of the above exception, another exception occurred:

    TypeError                                 Traceback (most recent call last)
    Cell In[63], line 7
          5     raise RuntimeError from None
          6 finally:
    ----> 7     raise TypeError
          8     print("finally")
          9 print("Unreachable")

    TypeError: