Home Artificial Intelligence Establishing Python Projects: Part III

Establishing Python Projects: Part III

admin
-
112
Establishing Python Projects: Part III

Photo by Gayatri Malhotra on Unsplash

Whether you’re a seasoned developer or simply getting began with 🐍 Python, it’s essential to know the way to construct robust and maintainable projects. This tutorial will guide you thru the strategy of organising a Python project using a number of the hottest and effective tools within the industry. You’ll learn the way to use GitHub and GitHub Actions for version control and continuous integration, in addition to other tools for testing, documentation, packaging and distribution. The tutorial is inspired by resources reminiscent of Hypermodern Python and Best Practices for a recent Python project. Nevertheless, this shouldn’t be the one method to do things and you would possibly have different preferences or opinions. The tutorial is meant to be beginner-friendly but additionally cover some advanced topics. In each section, you’ll automate some tasks and add badges to your project to point out your progress and achievements.

The repository for this series may be found at github.com/johschmidt42/python-project-johannes

  • OS: Linux, Unix, macOS, Windows (WSL2 with e.g. Ubuntu 20.04 LTS)
  • Tools: python3.10, bash, git, tree
  • Version Control System (VCS) Host: GitHub
  • Continuous Integration (CI) Tool: GitHub Actions

It is predicted that you simply are accustomed to the versioning control system (VCS) git. If not, here’s a refresher for you: Introduction to Git

Commits shall be based on best practices for git commits & Conventional commits. There may be the conventional commit plugin for PyCharm or a VSCode Extension that allow you to to put in writing commits on this format.

Overview

Structure

  • Testing framework (pytest)
  • Pytest configuration (pytest.ini_options)
  • Testing the applying (fastAPI, httpx)
  • Coverage (pytest-coverage)
  • Coverage configuration (coverage.report)
  • CI (test.yml)
  • Badge (Testing)
  • Bonus (Report coverage in README.md)

Testing your code is a crucial a part of software development. It helps you be sure that your code works as expected. You’ll be able to test your code or application manually or use a testing framework to automate the method. Automated tests may be of differing types, reminiscent of unit tests, integration tests, end-to-end tests, penetration tests, etc. On this tutorial, we’ll give attention to writing a straightforward unit test for our single function in our project. This may show that our codebase is well tested and reliable, which is a basic requirement for any proper project.

Python has some testing frameworks to select from, reminiscent of the built-in standard library unittest. Nevertheless, this module has some drawbacks, reminiscent of requiring boilerplate code, class-based tests and specific assert methods. A greater alternative is pytest, which is a preferred and powerful testing framework with many plugins. For those who usually are not accustomed to pytest, you need to read this introductory tutorial before you proceed, because we’ll write a straightforward test without explaining much of the fundamentals.

So let’s start by making a recent branch: feat/unit-tests

In our app src/example_app we only have two files that may be tested: __init__.py and app.py . The __init__ file accommodates just the version and the app.py our fastAPI application and the GET pokemon endpoint. We don’t must test the __init__.py file since it only accommodates the version and it’s going to be executed after we import app.py or some other file from our app.

We are able to create a tests folder within the project’s root and add the test file test_app.py in order that it looks like this:

.
...
├── src
│   └── example_app
│       ├── __init__.py
│       └── app.py
└── tests
└── test_app.py

Before we add a test function with pytest, we’d like to put in the testing framework first and add some configurations to make our lives a little bit easier:

Since the default visual output within the terminal leaves some room for improvement, I prefer to use the plugin pytest-sugar. This is totally optional, but if you happen to just like the visuals, give it a try. We install these dependencies to a recent group that we call test. Again, as explained within the last part (part II), that is to separate app and dev dependencies.

Because pytest won’t know where our tests are positioned, we are able to add this information to the pyproject.toml:

# pyproject.toml
...[tool.pytest.ini_options]
testpaths = ["tests"]
addopts = "-p no:cacheprovider" # deactivating pytest caching.

Where addopts stands for “add options” or “additional options” and the worth -p no:cacheprovider tells pytest to not cache runs. Alternatively, we are able to create a pytest.ini and add these lines there.

Let’s proceed with adding a test to the fastAPI endpoint that we created in app.py. Because we use httpx, we’d like to mock the response from the HTTP call (https://pokeapi.co/api). We could use monkeypatch or unittest.mock to vary the behaviour of some functions or classes in httpx but there already exists a plugin that we are able to use: respx

Mock HTTPX with awesome request patterns and response unintended effects.

Moreover, because fastAPI is an ASGI and never a WSGI, we’d like to put in writing an async test, for which we are able to use the pytest plugin: pytest-asyncio along with trio . Don’t worry if these are recent to you, they are only libraries for async Python and also you don’t need to grasp what they do.

> poetry add --group test respx pytest-asyncio trio

Let’s create our test within the test_app.py:

I won’t go into the small print of the way to create unit-tests with pytest, because this topic could cover a complete series of tutorials! But to summarise, I created an async test called test_get_pokemon through which the response shall be the expected_response because we’re using the respx_mock library. The endpoint of our fastAPI application known as and the result’s in comparison with the expected result. If you would like to find more about the way to test with fastAPI and httpx, try the official documentation: Testing in fastAPI

And if you’ve async functions, and don’t know the way to cope with them, take a have a look at: Testing with async functions in fastAPI

Assuming that you simply installed your application with poetry install we now can run pytest with

> pytest
Running all our tests — Image by creator

and pytest knows through which directory it must search for test files!

To make our linters blissful, we must always also run them on the newly created file. For this, we’d like to switch the command lint-mypy in order that mypy also covers files within the tests directory (previously only src):

# Makefile...lint-mypy:
@mypy ....

Finally, we are able to now run our formatters and linters before committing:

> make format
> make lint
Running formatters and linters — Image by creator

The code coverage in a project is a great indicator of how much of the code is roofed by unit tests. Hence, code coverage is a great metric (not all the time) to examine if a specific codebase is well tested and reliable.

We are able to check our code coverage with the coverage module. It creates a coverage report and provides information in regards to the lines that we missed with our unit-tests. We are able to install it via a pytest plugin pytest-cov:

> poetry add --group test pytest-cov

We are able to run the coverage module through pytest:

> pytest --cov=src --cov-report term-missing --cov-report=html

To only check the coverage for the src directory we add the flag --cov=src . We wish the report back to be displayed within the terminal --cov-report term-missing and stored in a html file with --cov-report html

Coverage report terminal — Image by creator

We see that a coverage HTML report has been created within the directory htmlcov through which we discover an index.html.

.
...
├── index.html
├── keybd_closed.png
├── keybd_open.png
├── status.json
└── style.css

Opening it in a browser allows us to visually see the lines that we covered with our tests:

Coverage report HTML (overview) — Image by creator

Clicking on the link src/example_app/app.py we see an in depth view of what our unit-tests covered within the file and more importantly which lines they missed:

Coverage report HTML (detailed) — Image by creator

We notice that the code under the if __name__ == "important": line is included in our coverage report. We are able to exclude this by setting the right flag when running pytest, or higher, add this configuration in our pyproject.toml:

# pyproject.toml
...[tool.coverage.report]
exclude_lines = [
'if __name__ == "__main__":'
]

The lines after the if __name__==”__main__" are actually excluded*.

*It probably is sensible to incorporate other common lines reminiscent of

  • def __repr__
  • def __str__
  • raise NotImplementedError
  • …

If we run pytest with the coverage module again

> pytest --cov=src --cov-report term-missing --cov-report=html
Coverage report HTML (excluded lines) — Image by creator

the last line shouldn’t be excluded as expected.

We have now covered the fundamentals of the coverage module, but there are more features you can explore. You’ll be able to read the official documentation to learn more in regards to the options.

Let’s add these commands (pytest, coverage) to our Makefile, the identical way we did in Part II, in order that we don’t should remember them. Moreover we add a command that uses the --cov-fail-under=80 flag. This signals pytest to fail if the whole coverage is lower than 80 %. We’ll use this later within the CI a part of this tutorial. Since the coverage report creates some files and directories inside the project, we must always also add a command that removes these for us (clean-up):

# Makefileunit-tests:
@pytestunit-tests-cov:
@pytest --cov=src --cov-report term-missing --cov-report=htmlunit-tests-cov-fail:
@pytest --cov=src --cov-report term-missing --cov-report=html --cov-fail-under=80clean-cov:
@rm -rf .coverage
@rm -rf htmlcov...

And now we are able to invoke these with

> make unit-tests
> make unit-tests-cov

and clean up the created files with

> make clean-cov

Once more, we use the software development practice CI to make certain that nothing is broken each time we commit to our default branch important.

Up until now, we were in a position to run our tests locally. So allow us to create our second workflow that can run on a server from GitHub! We have now the choice of using codecov.io together with the codecov-action OR we are able to create the report within the Pull Request (PR) itself with a pytest-comment motion. I’ll select the second option for simplicity.

We are able to either create a recent workflow that runs parallel to our linter lint.yml (faster) or have one workflow that runs the linters first after which the testing job (more efficient). This can be a design alternative that depends upon the project’s needs. Each options have pros and cons. For this tutorial, I’ll create a separate workflow (test.yml). But before we try this, we’d like to update our command within the Makefile, in order that we create a pytest.xml and a pytest-coverage.txt, that are needed for the pytest-comment motion:

# Makefile...unit-tests-cov-fail:
@pytest --cov=src --cov-report term-missing --cov-report=html --cov-fail-under=80 --junitxml=pytest.xml | tee pytest-coverage.txtclean-cov:
@rm -rf .coverage
@rm -rf htmlcov
   @rm -rf pytest.xml
@rm -rf pytest-coverage.txt...

Now we are able to write our workflow test.yml:

Let’s break it right down to make certain we understand each part. GitHub motion workflows should be created within the .github/workflows directory of the repository within the format of .yaml or .yml files. For those who’re seeing these for the primary time, you possibly can check them out here to raised understand them. Within the upper a part of the file, we give the workflow a reputation name: Testing and define on which signals/events, this workflow needs to be began: on: ... . Here, we wish that it runs when recent commits come right into a PullRequest targeting the important branch or commits go the important branch directly. The job runs in an ubuntu-latest (runs-on) environment and executes the next steps:

  • checkout the repository using the branch name that’s stored within the default environment variable ${{ github.head_ref }} . GitHub motion: checkout@v3
  • install Poetry with pipx since it’s pre-installed on all GitHub runners. If you’ve a self-hosted runner in e.g. Azure, you’d must install it yourself or use an existing GitHub motion that does it for you.
  • Setup the python environment and caching the virtualenv based on the content within the poetry.lock file. GitHub motion: setup-python@v4
  • Install the applying & its requirements along with the test dependencies which can be needed to run the tests with pytest: poetry install --with test
  • Running the tests with the make command: poetry run make unit-tests-cov-vail Please note, that running the tools is simply possible within the virtualenv, which we are able to access through poetry run.
  • We use a GitHub motion that enables us to mechanically create a comment within the PR with the coverage report. GitHub motion: pytest-coverage-comment@important

Once we open a PR targeting the important branch, the CI pipeline will run and we’ll see a comment like this in our PR:

Pytest coverage report in PR comment — Image by creator

It created a small badge with the whole coverage percentage (81%) and has linked the tested files with URLs. With one other commit in the identical feature branch (PR), the identical comment for the coverage report is overwritten by default.

To display the status of our recent CI pipeline on the homepage of our repository, we are able to add a badge to the README.md file.

We are able to retrieve the badge after we click on a workflow run:

Create a standing badge from workflow file on GitHub — Image by creator
Copy the badge markdown — Image by creator

and choose the important branch. The badge markdown may be copied and added to the README.md:

Our landing page of the GitHub now looks like this ❤:

Second badge in README.md: Testing — Image by creator

For those who are interested in how this badge reflects the newest status of the pipeline run within the important branch, you possibly can try the statuses API on GitHub.

112 COMMENTS

  13. Simply wish to say your article is as astonishing.
    The clarity to your put up is just cool and that i could assume you
    are an expert on this subject. Well with your permission let me to
    clutch your feed to stay up to date with approaching post. Thank you one million and please carry on the rewarding work.

  18. hello!,I really like your writing very much!
    proportion we keep up a correspondence extra approximately your article on AOL?
    I require a specialist on this space to solve my problem.
    May be that is you! Having a look forward to look you.

  22. I absolutely love your blog and find a lot of your post’s to be just
    what I’m looking for. Do you offer guest writers to write content
    for yourself? I wouldn’t mind creating a post or elaborating on a few of the subjects you write in relation to here.
    Again, awesome blog!

  27. Have you ever thought about writing an ebook
    or guest autgoring on other blogs? I have a blog centered on the
    same information you discuss annd would love to have you share
    some stories/information. I knoiw my visitors would enjoy
    your work. If you’re even remotely interested, feel frree to shoot me
    an e-mail.

  30. Does your blog have a contact page? I’m having problems locating it but, I’d like to shoot you an email.
    I’ve got some recommendations for your blog you might
    be interested in hearing. Either way, great site and I look
    forward to seeing it develop over time.

  39. I truly love your site.. Pleasant colors & theme. Did you create this amazing site yourself?
    Please reply back as I’m hoping to create my very own website and would like to know where you got this from or exactly what the theme is called.
    Many thanks!

  42. What i do not realize is in reality how you are now
    not actually much more neatly-liked than you may be right now.
    You’re very intelligent. You realize thus significantly relating to
    this matter, produced me in my opinion imagine it
    from a lot of numerous angles. Its like women and men are not involved except it’s something to do with Woman gaga!
    Your individual stuffs nice. Always care for it up!

  43. I simply wanted to thank you a lot more for your amazing website you have developed here. It can be full of useful tips for those who are actually interested in this specific subject, primarily this very post. Your all so sweet in addition to thoughtful of others and reading the blog posts is a great delight in my opinion. And thats a generous present! Dan and I usually have enjoyment making use of your recommendations in what we need to do in the near future. Our checklist is a distance long and tips will certainly be put to excellent use.Here

  45. It is the best time to make some plans for the long run and it is time to be happy. I have learn this publish and if I may just I desire to suggest you few attention-grabbing issues or tips. Maybe you could write subsequent articles relating to this article. I wish to learn even more issues approximately it!|

  46. Hello, Neat post. There’s an issue together with your web site in web explorer, would test this?
    IE nonetheless is the market leader and a large component
    to other folks will pass over your wonderful writing due to this problem.

  50. Hi, I do believe this is a great site. I stumbledupon it 😉 I’m going to come back yet again since
    i have book-marked it. Money and freedom is the greatest way to change,
    may you be rich and continue to help other people.

  51. Hi there! I know this is kinda off topic nevertheless I’d figured I’d ask.
    Would you be interested in trading links or maybe guest authoring a blog post or vice-versa?
    My site goes over a lot of the same subjects as yours and
    I think we could greatly benefit from each other.

    If you’re interested feel free to shoot me an email. I look forward to hearing from you!
    Wonderful blog by the way!

  55. Have you ever thought about adding a little bit more than just your articles?
    I mean, what you say is valuable and all. However think of
    if you added some great visuals or videos to give your posts more,
    “pop”! Your content is excellent but with pics and video clips, this
    blog could definitely be one of the most beneficial in its field.

    Fantastic blog!

  56. I love your blog.. very nice colors & theme. Did you create this
    website yourself or did you hire someone to do it for you?
    Plz reply as I’m looking to create my own blog and would like to know where u got this from.
    thanks a lot

  57. I’m curious to find out what blog platform you are utilizing?
    I’m having some minor security issues with my latest website and I’d like to find something more safeguarded.
    Do you have any suggestions?

  61. Thank you, I have just been looking for information about this subject for a long time and yours is the greatest I have discovered so far. However, what about the bottom line? Are you sure concerning the supply?|

  66. hey there and thank you for your information – I have certainly picked up something new from right here.
    I did however expertise a few technical points using this site, as I experienced to reload
    the web site lots of times previous to I could get it to
    load properly. I had been wondering if your hosting is OK?

    Not that I am complaining, but sluggish loading instances times will sometimes affect your placement in google and can damage your high quality score if ads and marketing with Adwords.
    Anyway I am adding this RSS to my e-mail and can look out for much more of your respective fascinating content.
    Make sure you update this again soon.

  67. Right here is the perfect site for everyone who would like to find out about this
    topic. You realize so much its almost hard to argue
    with you (not that I personally will need to…HaHa).
    You definitely put a new spin on a subject that’s been discussed for many years.
    Excellent stuff, just wonderful!

  71. Just desire to say your article is as astonishing.
    The clearness in your post is simply nice and i can assume you’re an expert on this subject.

    Well with your permission let me to grab your RSS feed to keep up to date with forthcoming post.
    Thanks a million and please carry on the enjoyable work.

  74. Great blog! Do you have any helpful hints for aspiring writers?
    I’m planning to start my own blog soon but I’m a little lost on everything.
    Would you suggest starting with a free platform like WordPress or go for a paid option?
    There are so many options out there that I’m totally overwhelmed
    .. Any recommendations? Thanks!

  75. I’m truly enjoying the design and layout of your blog. It’s a very easy on the eyes which makes
    it much more pleasant for me to come here and visit more often. Did you hire out a designer to
    create your theme? Superb work!

  76. Excellent beat ! I wish to apprentice while you amend
    your website, how can i subscribe for a blog site? The account aided me a acceptable deal.

    I had been tiny bit acquainted of this your broadcast offered bright clear concept

  78. Hi this is somewhat of off topic but I was wondering if blogs use WYSIWYG editors or if you have to manually code with HTML. I’m starting a blog soon but have no coding knowledge so I wanted to get guidance from someone with experience. Any help would be greatly appreciated!kontol kuda

  84. Thanks a lot for sharing this with all people you really understand what
    you are speaking about! Bookmarked. Please additionally discuss with my web site =).
    We will have a hyperlink change contract among us

  88. I know this if off topic but I’m looking into starting my
    own blog and was wondering what all is required to get set
    up? I’m assuming having a blog like yours would cost a pretty
    penny? I’m not very internet smart so I’m not 100% sure.
    Any suggestions or advice would be greatly appreciated.
    Kudos

  89. Thank you a lot for sharing this with all of us you really realize what
    you are speaking approximately! Bookmarked.
    Kindly also visit my website =). We can have a link alternate
    arrangement among us

  92. I was suggested this website by my cousin. I’m not sure whether this post is written by him as nobody else know
    such detailed about my trouble. You are wonderful! Thanks!

  96. Its like you read my mind! You appear to grasp a lot approximately this, such as you
    wrote the ebook in it or something. I think that you
    could do with a few percent to force the message home a bit, but other than that, this is wonderful blog.

    A great read. I’ll certainly be back.

  99. Heya i am for the primary time here. I found this board and I in finding It really useful & it helped me out much. I am hoping to offer one thing back and help others like you helped me.|

  100. I have been exploring for a bit for any high quality articles or blog posts in this kind of space . Exploring in Yahoo I at last stumbled upon this website. Reading this information So i’m happy to convey that I have a very just right uncanny feeling I discovered exactly what I needed. I most no doubt will make certain to do not overlook this website and give it a glance on a continuing basis.|

  103. Generally I do not learn post on blogs, but I would like to say that this write-up
    very compelled me to try and do so! Your writing style has been surprised me.
    Thank you, quite nice article.

  106. Hey! This is my first visit to your blog! We are a team of volunteers and starting a new project in a community in the same niche. Your blog provided us useful information to work on. You have done a outstanding job!|

  108. Hello every one, here every person is sharing these know-how, thus it’s good to read this blog, and I used to pay a visit this website all the time.|

  111. It is the best time to make some plans for the future and it’s time
    to be happy. I’ve read this post and if I could I desire to suggest you few
    interesting things or advice. Perhaps you can write
    next articles referring to this article. I wish to read more things about it!

  112. It’s really very complex in this full of activity life to listen news on TV, thus I only
    use the web for that purpose, and take the most
    up-to-date information.

Leave a Reply to 마약 거래 Cancel reply

Please enter your comment!
Please enter your name here

© Newspaper WordPress Theme by TagDiv