Time to take credit

Introduction

You’ve got been programming in Python for some time now, whipping up scripts and solving problems left and right. You’re thinking that you are pretty good, don’t you? Well, hold on to your hats, folks, because you would possibly just be a sophisticated Pythonista without even realizing it!

From closure to context managers, I’ve got an inventory of advanced Python features that can make you say, “I have been using that every one along!”.

Even when these items are recent to you, you’ll need a superb checklist to finish to take your game to the following level.

1. Scope

A critical aspect of advanced Python programming is deep familiarity with the concept of scope.

Scope defines the order through which the Python interpreter looks up names in a program. Python scope follows the rule (local, enclosing, global, and built-in scopes). In keeping with the rule, while you access a reputation (it may possibly be anything, a variable, a function, or a category), the interpreter looks for it in local, enclosing, global, and built-in scopes, so as.

Let’s examine examples to grasp each level higher.

def func():
x = 10
print(x)
func() # 10
print(x) # Raises NameError, x is barely defined throughout the scope of func()

Here, x is barely defined within the scope that’s local to func. That is why it is not accessible anywhere else within the script.

def outer_func():
x = 20
def inner_func():
print(x)
inner_func()
outer_func() # 20

Enclosing scope is the intermediary scope between local and global scopes. In the instance above, x is within the local scope of outer_func. However, x is within the enclosing scope relative to the nested inner_func function. Local scope at all times has read-only access to the enclosing scope.

x = 30
def func():
print(x)
func() # 30

Here, x and func are defined in the worldwide scope, which implies they will be read from anywhere in the present script.

To switch them in smaller levels of scope (local and enclosing), they ought to be accessed with the global keyword:

def func2():
global x
x = 40
print(x)
func2() # 40
print(x) # 40

Built-in scope includes every already-defined library, class, function, and variable that explicit import statements. Some examples of built-in functions and variables in Python include print, len, range, str, int, float, etc.

2. Function closure

A firm grasp of scope opens the doors to a different necessary concept — function closure.

By default, after the function finishes execution, it returns to a blank state. This implies its memory is wiped of all of its past arguments.

def func(x):
return x ** 2
func(3)

9

print(x) # NameError

Above, we assigned the worth of three to x however the function forgot it after execution. What if we don’t need it to forget the worth of x?

That is where function closure comes into play. By defining a variable within the enclosing scope of some inner function, you may store it within the inner function’s memory even after the function returns.

Here is a straightforward example function that counts the variety of times it was executed:

def counter():
count = 0
def inner():
nonlocal count
count += 1
return count
return inner
# Return the inner function
counter = counter()
print(counter()) # 1
print(counter()) # 2
print(counter()) # 3

1
2
3

By all rules of Python, we should always have lost the counter variable after the primary execution. But because it is within the inner function’s closure, it would stay there till you shut the session:

counter.__closure__[0].cell_contents

3

3. Decorators

Function closures have more serious applications than easy counters. One in all them is creating decorators. A decorator is a nested function you may add to other functions to reinforce and even modify their behavior.

For instance, below, we’re making a caching decorator that remembers the state of each positional and keyword argument of a function.

def stateful_function(func):
cache = {}
def inner(*args, **kwargs):
key = str(args) + str(kwargs)
if key not in cache:
cache[key] = func(*args, **kwargs)
return cache[key]
return inner

The stateful_function decorator can now be added to computation-heavy functions that could be reused on the identical arguments. The instance is the next recursive Fibonacci function that returns the nth number within the sequence:

%%time
@stateful_function
def fibonacci(n):
if n <= 0:
return 0
elif n == 1:
return 1
else:
return fibonacci(n-1) + fibonacci(n-2)
fibonacci(1000)

CPU times: user 1.53 ms, sys: 88 µs, total: 1.62 ms
Wall time: 1.62 ms
[OUT]:
43466557686937456435688527675040625802564660517371780402481729089536555417949051890403879840079255169295922593080322634775209689623239873322471161642996440906533187938298969649928516003704476137795166849228875

We found the humongous a thousandth number within the Fibonacci sequence in a fraction of a second. Here is how much the identical process would take without the caching decorator:

%%time
def fibonacci(n):
if n <= 0:
return 0
elif n == 1:
return 1
else:
return fibonacci(n-1) + fibonacci(n-2)
fibonacci(40)

CPU times: user 21 s, sys: 0 ns, total: 21 s
Wall time: 21 s
[OUT]:
102334155

It took 21 seconds to calculate the fortieth number. It could take days to calculate the a thousandth without caching.

You’ll be able to learn the hairy details of the right way to create your individual decorators (including scope and closures) in my separate post:

4. Generators

Generators are powerful constructs in Python that permits processing of huge amounts of information efficiently.

For instance you could have a 10GB log file after the crash of some software. To seek out out what went improper, you could have to efficiently sift through it in Python.

The worst technique to do that is to read the entire file like below:

with open("logs.txt", "r") as f:
contents = f.read()
print(contents)

Because you undergo the logs line by line, you needn’t read all 10GBs, just chunks of it at a time. That is where you should utilize generators:

def read_large_file(filename):
with open(filename) as f:
while True:
chunk = f.read(1024)
if not chunk:
break
yield chunk # Generators are defined with `yield` as a substitute of `return`
for chunk in read_large_file("logs.txt"):
process(chunk)    # Process the chunk

Above, we defined a generator that iterates the lines of the log file only 1024 at a time. Because of this, the for loop at the tip is extremely efficient. In every iteration of the loop, only 1024 lines of the file are in memory. The previous chunks are discarded, while the remainder are only loaded as needed.

One other feature of generators is the flexibility to yield a component at a time, even outside loops, with the next function. Below, we’re defining a blazing-fast function that generates the Fibonacci sequence.

To create the generator, you call the function once and call next on the resulting object:

def fibonacci():
a, b = 0, 1
while True:
yield a
a, b = b, a + b
fib = fibonacci()
type(fib)

generator

print(next(fib)) # 0
print(next(fib)) # 1
print(next(fib)) # 1
print(next(fib)) # 2
print(next(fib)) # 3

You’ll be able to read the next post on generators to learn more.

5. Context managers

You will need to have been using context managers for a very long time now. They permit developers to administer resources efficiently, like files, databases, and network connections. They robotically open and shut resources, leading to wash and error-free code.

But, there may be a giant difference between using context managers and writing your individual. When done right, they mean you can abstract lots of boilerplate code on top of their original functionality.

One popular example of a custom context manager is a timer:

import time
class TimerContextManager:
"""
Measure the time it takes to run
a block of code.
"""
def __enter__(self):
self.start = time.time()
def __exit__(self, type, value, traceback):
end = time.time()
print(f"The code took {end - self.start:.2f} seconds to execute.")

Above, we’re defining a TimerContextManager class that can function our future context manager. Its __enter__ method defines what happens once we enter the context with the with keyword. On this case, we start the timer.

In __exit__, we exit of the context, stop the timer, and report elapsed time.

with TimerContextManager():
# This code is timed
time.sleep(1)

The code took 1.00 seconds to execute.

Here’s a more complex example that permits locking resources in order that they will be utilized by one process at a time.

import threading
lock = threading.Lock()
class LockContextManager:
def __enter__(self):
lock.acquire()
def __exit__(self, type, value, traceback):
lock.release()
with LockContextManager():
# This code is executed with the lock acquired
# Just one process will be inside this block at a time
# The lock is robotically released when the with block ends, even when an error occurs

Should you desire a more gentle introduction to context managers, try my article on the subject.

If you wish to go down the rabbit hole and learn all the things about them, here is one other excellent RealPython article.

Conclusion

There you could have it, folks! How over and over did you say, “I knew that!”? Even when it wasn’t that over and over, you now know the things to learn to change into advanced.

Do not be afraid of the discomfort that comes with learning recent things. Just remember, with great power comes (I won’t say it!) more difficult bugs to repair. But hey, you might be a professional now, what’s just a little debugging to you?

Thanks for reading!

More stories from me…