Skip to content

Commit

Permalink
update poetry config
Browse files Browse the repository at this point in the history
  • Loading branch information
@gavinying
gavinying committed Sep 17, 2021
1 parent 5a2d1f0 commit 48af8cd
Show file tree
Hide file tree
Showing 4 changed files with 150 additions and 906 deletions.
306 changes: 33 additions & 273 deletions README.md
View file
Original file line number Diff line number Diff line change
@@ -1,299 +1,59 @@
# Ysdtools - A collection of useful tools
[![pipeline status](https://gitlab.com/helloysd/modpoll/badges/master/pipeline.svg)](https://gitlab.com/helloysd/modpoll/-/commits/master)
[![License](https://img.shields.io/pypi/l/modpoll)](https://gitlab.com/helloysd/modpoll/-/blob/master/LICENSE)

[![Release](https://img.shields.io/pypi/v/ysdtools)](https://pypi.org/project/ysdtools/)
[![build](https://github.com/gavinying/ysdtools-python/actions/workflows/ci.yml/badge.svg)](https://github.com/gavinying/ysdtools-python/actions)
[![codecov](https://codecov.io/gh/gavinying/ysdtools-python/branch/master/graph/badge.svg?token=TDBFIHJDZG)](https://codecov.io/gh/gavinying/ysdtools-python)
[![Read the Docs](https://readthedocs.org/projects/ysdtools-python/badge/?version=latest)](https://ysdtools-python.readthedocs.io/en/latest)
---

**This project is not designed for production, only for personal use!**
# Modpoll

## Table of Contents
A command line tool to communicate with modbus devices.

- [Project Setup](#project-setup)
- [Initialize Project](#initialize-project)
- [Build and Test](#build-and-test)
- [Document Project](#document-project)
- [Setup CI Pipeline](#setup-ci-pipeline)
- [CodeCov](#codecov)
- [Read The Docs](#read-the-docs)
- [Deploy to PyPI](#deploy-to-pypi)
- [Install](#install)
- [Usage](#usage)
- [Maintainers](#maintainers)
- [License](#license)
> Learn more about `modpoll` usage at [documentation](https://helloysd.gitlab.io/modpoll) site.
## Project Setup
This project uses [poetry](https://python-poetry.org/) to setup project. Project setup process is based on https://testdriven.io/blog/python-project-workflow

### Initialize Project
## Motivation

First, let's create a python project with poetry,
The initial idea of creating this tool is to help myself debugging and troubleshooting new modbus devices during site survey. A typical site survey usually has limited time and space, working on-site also pile up some pressures. During that time, a portable swiss-knife toolkit is our best friend.

```bash
$ poetry new --name ysdtools ysdtools-python
This program can be easily deployed to Raspberry Pi or similar embedded devices, continuously polling data from the connected modbus devices, you can choose to save data locally or forward uplink to a MQTT broker for easy debugging, the MQTT broker can be setup on the same Raspberry Pi or on the cloud. On the other hand, a smart phone (Android/Iphone) can be used to visualize collected data and control the devices remotely via the same MQTT broker.

Package name [ysdtools]:
Version [0.1.0]:
Description []:
Author [Your name <[email protected]>, n to skip]:
License []:
Compatible Python versions [>3.7,<4.0]:
However, beside the above recommended setup, you can actually run this program on any PC or server with Python 3 support. One popular use case is to deploy the program onto a server and keep it running as a gateway to bridge between traditional industrial network and modern IoT edge/cloud infrustructure.

Would you like to define your main dependencies interactively? (yes/no) [yes] no
Would you like to define your development dependencies interactively? (yes/no) [yes] no
Do you confirm generation? (yes/no) [yes]
```
> This program is designed as a standalone tool, if you are looing for a python library to communicate with modbus devices, please consider the following two great open source projects, [pymodbus](https://github.com/riptideio/pymodbus) or [minimalmodbus](https://github.com/pyhys/minimalmodbus)
Now you will have a folder `ysdtools-python`, and modify the config file `pyproject.toml` inside according to your project need.
Your project name must be unique since you'll be uploading it to PyPI. So, to avoid name collisions add a unique string to the package name in pyproject.toml.
## Installation

> For more details on Poetry, check out `https://python-poetry.org/docs/basic-usage`
This program is tested on python 3.6+.

Create a new repository on Github named `ysdtools-python`, and initialize with the newly created project.
- Install with pip

Inside the project folder, add a few python dependencies for testing and code quality check,
```bash
$ poetry add --dev pytest pytest-cov black isort flake8 bandit safety
```
Add the new poetry.lock file as well as the updated pyproject.toml file to git,
```bash
$ git add poetry.lock pyproject.toml
```
The package is available in the Python Package Index,

### Build and Test
```bash
pip install modpoll
```

Now the project layout looks like this,
```bash
ysdtools-python
├── poetry.lock
├── pyproject.toml
├── tests
│ ├── __init__.py
│ └── test_ysdtools.py
└── ysdtools
├── __init__.py
└── ysdlog.py
```
Run the script,
```bash
$ poetry run python ysdtools/ysdlog.py
```
Run the test,
```bash
$ poetry run python -m pytest tests
```
Upgrade the tool via pip by the following command,

### Document Project
```bash
pip install -U modpoll
```

We use Google python style for docstrings,
> If you are not familiar with docstrings or documentation, please refer to [Google Python Style Guide](https://google.github.io/styleguide/pyguide.html)
- Install with docker

Assuming you have [Sphinx](https://www.sphinx-doc.org/en/master/usage/quickstart.html) installed, run the following to scaffold out the files and folders for Sphinx in the project root:
```bash
$ sphinx-quickstart docs
(To be added...)

> Separate source and build directories (y/n) [n]: n
> Project name: Ysdtools
> Author name(s): Your Name
> Project release []: 0.1.0
> Project language [en]: en
```
Next, let's update the `docs/conf.py`, set the correct source path,
```bash
import os
import sys
sys.path.insert(0, os.path.abspath('../'))
```
Add autodoc extention,
```bash
extensions = [
'sphinx.ext.autodoc',
]
```

Create a new page `docs/ysdlog.rst` with the following content,
```bash
ysdtools.ysdlog
====================================
.. automodule:: ysdtools.ysdlog
:members:
```
## Examples

Update `docs/index.rst` like so,
```bash
Welcome to ysdtools's documentation!
====================================
Please refer to [documentation](https://helloysd.gitlab.io/modpoll) site for more configures and examples.

.. toctree::
:maxdepth: 2
:caption: Contents:
## Credits

ysdlog
```
### Setup CI Pipeline
Let's setup CI pipeline with Github Actions,
add the following files and folders to the project root:
```bash
.github
└── workflows
└── ci.yml
```
Inside ci.yml,
```bash
name: Push
on: [push, pull_request]
jobs:
test:
strategy:
fail-fast: false
matrix:
python-version: [3.8]
poetry-version: [1.1.5]
os: [ubuntu-latest]
runs-on: ${{ matrix.os }}
steps:
- uses: actions/checkout@v2
- uses: actions/setup-python@v2
with:
python-version: ${{ matrix.python-version }}
- name: Run image
uses: abatilo/[email protected]
with:
poetry-version: ${{ matrix.poetry-version }}
- name: Install dependencies
run: poetry install
- name: Run tests
run: poetry run pytest --cov=./ --cov-report=xml
- name: Upload coverage to Codecov
uses: codecov/codecov-action@v1
with:
token: ${{ secrets.CODECOV_TOKEN }}
code-quality:
strategy:
fail-fast: false
matrix:
python-version: [3.8]
poetry-version: [1.1.5]
os: [ubuntu-latest]
runs-on: ${{ matrix.os }}
steps:
- uses: actions/checkout@v2
- uses: actions/setup-python@v2
with:
python-version: ${{ matrix.python-version }}
- name: Run image
uses: abatilo/[email protected]
with:
poetry-version: ${{ matrix.poetry-version }}
- name: Install dependencies
run: poetry install
- name: Run black
run: poetry run black . --check
- name: Run isort
run: poetry run isort . --check-only
- name: Run flake8
run: poetry run flake8 .
- name: Run bandit
run: poetry run bandit -r . -x ./tests,./docs
- name: Run saftey
run: poetry run safety check
```
There are two jobs defined: test and code-quality. As the names' suggest, the tests run in the test job while our code quality checks run in the code-quality job.
> Now on every push to the GitHub repository, tests and code quality jobs will run.
Before we commit the changes, it is better to pass all local test first,
```bash
$ poetry run black . --check
$ poetry run isort . --check-only
$ poetry run flake8 .
$ poetry run bandit -r . -x ./tests,./docs
$ poetry run safety check
```
Finally, we can commit and push the code,
```bash
$ git commit -m 'CI pipeline ready'
$ git push -u origin master
```
You should see your workflow running on the "Actions" tab on your GitHub repository. Make sure it passes before moving on.
### CodeCov
Next, we'll configure CodeCov to track code coverage. Navigate to http://codecov.io/, and log in with your GitHub account and find your repository.

Run the GitHub Actions workflow again. Once done, you should be able to see the coverage report on CodeCov.

If the github repository is private, you can generate an API token from CodeCov, and put it as an environment variable `CODECOV_TOKEN` in Github `project->settings->secrets`.

### Read The Docs
We'll use Read the Docs to host our documentation. Navigate to https://readthedocs.org, and log in using your GitHub account.
> Read The Docs cloud only support public repository for free.
### Deploy to PyPI
Finally, in order to make the project "pip-installable", we'll publish it to PyPI.

Add the following code in `pyproject.toml`,
```bash
packages = [
{ include = "ysdtools" },
]
```
Add a new file called `cd.yml` to ".github/workflows":
```bash
name: Release
on:
release:
types:
- created
jobs:
publish:
strategy:
fail-fast: false
matrix:
python-version: [3.8]
poetry-version: [1.1.5]
os: [ubuntu-latest]
runs-on: ${{ matrix.os }}
steps:
- uses: actions/checkout@v2
- uses: actions/setup-python@v2
with:
python-version: ${{ matrix.python-version }}
- name: Run image
uses: abatilo/[email protected]
with:
poetry-version: ${{ matrix.poetry-version }}
- name: Publish
env:
PYPI_TOKEN: ${{ secrets.PYPI_TOKEN }}
run: |
poetry config pypi-token.pypi $PYPI_TOKEN
poetry publish --build
```
Next, we'll need to create a PyPI token. Create an account on PyPI if you don't already have one. Then, once signed in, click "Account settings" and add a new API token. Copy the token. You'll now need to add it to your GitHub repository's secrets. Within the repo, click the "Settings" tab, and then click "Secrets". Use PYPI_TOKEN for the secret name and the token value as the secret value.
## Install
Install via pip,
```bash
$ pip install ysdtools
```
## Usage
A basic usage is as,
```bash
from ysdtools import ysdlog
ysdlog.info("Start logging...")
```
## Maintainers
Ying Shaodong <[email protected]>
The implementation of this project is heavily inspired by the following two projects:
- https://github.com/owagner/modbus2mqtt (MIT license)
- https://github.com/mbs38/spicierModbus2mqtt (MIT license)
Thanks to Max Brueggemann and Oliver Wagner for their great work.

## License
[MIT](LICENSE) © Ying Shaodong

MIT © [Ying Shaodong]([email protected])
Loading

0 comments on commit 48af8cd

Please sign in to comment.