Skip to content

Latest commit

 

History

History
1168 lines (791 loc) · 38.3 KB

README.rst

File metadata and controls

1168 lines (791 loc) · 38.3 KB
Raw

openwisp-utils

CI build status Test coverage Requirements Status pypi downloads support chat code style: black

Python and Django functions, classes and settings re-used across different OpenWISP modules, stored here with the aim of avoiding code duplication and ease maintenance.

Don't repeat yourself!

Current features

Table of Contents:

Install stable version from pypi

Install from pypi:

pip install openwisp-utils

# install optional dependencies for REST framework
pip install openwisp-utils[rest]

# install optional dependencies for tests (flake8, black and isort)
pip install openwisp-utils[qa]

# or install everything
pip install openwisp-utils[rest,qa]

Install development version

Install tarball:

pip install https://github.com/openwisp/openwisp-utils/tarball/master

Alternatively you can install via pip using git:

pip install -e git+git://github.com/openwisp/openwisp-utils#egg=openwisp-utils

Using the admin_theme

The admin theme requires Django >= 2.2..

Add openwisp_utils.admin_theme to INSTALLED_APPS in settings.py:

INSTALLED_APPS = [
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',

    'openwisp_utils.admin_theme',    # <----- add this

    'django.contrib.sites',
    # admin
    'django.contrib.admin',
]

Using DependencyLoader and DependencyFinder

Add the list of all packages extended to EXTENDED_APPS in settings.py.

For example, if you've extended django_x509:

EXTENDED_APPS = ['django_x509']

DependencyFinder

This is a static finder which looks for static files in the static directory of the apps listed in settings.EXTENDED_APPS.

Add openwisp_utils.staticfiles.DependencyFinder to STATICFILES_FINDERS in settings.py.

STATICFILES_FINDERS = [
    'django.contrib.staticfiles.finders.FileSystemFinder',
    'django.contrib.staticfiles.finders.AppDirectoriesFinder',
    'openwisp_utils.staticfiles.DependencyFinder',    # <----- add this
]

DependencyLoader

This is a template loader which looks for templates in the templates directory of the apps listed in settings.EXTENDED_APPS.

Add openwisp_utils.loaders.DependencyLoader to template loaders in settings.py as shown below.

TEMPLATES = [
    {
        'BACKEND': 'django.template.backends.django.DjangoTemplates',
        'DIRS': [],
        'OPTIONS': {
            'loaders': [
                # ... other loaders ...
                'openwisp_utils.loaders.DependencyLoader',    # <----- add this
            ],
            'context_processors': [
                # ... omitted ...
            ],
        },
    },
]

Supplying custom CSS and JS for the admin theme

Add openwisp_utils.admin_theme.context_processor.admin_theme_settings to template context_processors in settings.py as shown below. This will allow to set OPENWISP_ADMIN_THEME_LINKS and OPENWISP_ADMIN_THEME_JS settings to provide CSS and JS files to customise admin theme.

TEMPLATES = [
    {
        'BACKEND': 'django.template.backends.django.DjangoTemplates',
        'DIRS': [],
        'OPTIONS': {
            'loaders': [
                # ... omitted ...
            ],
            'context_processors': [
                # ... other context processors ...
                'openwisp_utils.admin_theme.context_processor.admin_theme_settings'    # <----- add this
            ],
        },
    },
]

Note

You will have to deploy these static files on your own.

In order to make django able to find and load these files you may want to use the STATICFILES_DIR setting in settings.py.

You can learn more in the Django documentation.

OpenWISP Dashboard

The admin_theme sub app of this package provides an admin dashboard for OpenWISP which can be manipulated with the functions described in the next sections.

Example 1, monitoring:

https://raw.githubusercontent.com/openwisp/openwisp-utils/master/docs/dashboard1.png

Example 2, controller:

https://raw.githubusercontent.com/openwisp/openwisp-utils/master/docs/dashboard2.png

register_dashboard_template

Allows including a specific django template in the OpenWISP dashboard.

It is designed to allow the inclusion of the geographic map shipped by OpenWISP Monitoring but can be used to include any custom element in the dashboard.

Note: templates are loaded before charts.

Syntax:

register_dashboard_template(position, config)
Parameter Description
position (int) The position of the template.
config (dict) The configuration of the template.

Following properties can be configured for each template config:

Property Description
template (str) Path to pass to the template loader.
css (tuple) List of CSS files to load in the HTML page.
js (tuple) List of Javascript files to load in the HTML page.

Code example:

      from openwisp_utils.admin_theme import register_dashboard_template

register_dashboard_template(
    position=0,
    config={
        'template': 'admin/dashboard/device_map.html',
        'css': (
            'monitoring/css/device-map.css',
            'leaflet/leaflet.css',
            'monitoring/css/leaflet.fullscreen.css',
        ),
        'js': (
            'monitoring/js/device-map.js',
            'leaflet/leaflet.js',
            'leaflet/leaflet.extras.js',
            'monitoring/js/leaflet.fullscreen.min.js'
        )
    }
)

It is recommended to register dashboard templates from the ready method of the AppConfig of the app where the templates are defined.

unregister_dashboard_template

This function can be used to remove a template from the dashboard.

Syntax:

unregister_dashboard_template(template_name)
Parameter Description
template_name (str) The name of the template to remove.

Code example:

from openwisp_utils.admin_theme import unregister_dashboard_template

unregister_dashboard_template('admin/dashboard/device_map.html')

Note: an ImproperlyConfigured exception is raised the specified dashboard template is not registered.

register_dashboard_chart

Adds a chart to the OpenWISP dashboard.

At the moment only pie charts are supported.

The code works by defining the type of query which will be executed, and optionally, how the returned values have to be colored and labeled.

Syntax:

register_dashboard_chart(position, config)
Parameter Description
position (int) Position of the chart.
config (dict) Configuration of chart.

Following properties can be configured for each chart config:

Property Description
query_param

It is a required property in form of dict containing following properties:

Property Description
name (str) Chart title shown in the user interface.
app_label (str) App label of the model that will be used to query the database.
model (str) Name of the model that will be used to query the database.
group_by (str) The property which will be used to group values.
annotate Alternative to group_by, dict used for more complex queries.
aggregate Alternative to group_by, dict used for more complex queries.
colors An optional dict which can be used to define colors for each distinct value shown in the pie charts.
labels An optional dict which can be used to define translatable strings for each distinct value shown in the pie charts. Can be used also to provide fallback human readable values for raw values stored in the database which would be otherwise hard to understand for the user.
filters An optional dict which can be used when using aggregate and annotate in query_params to define the link that will be generated to filter results (pie charts are clickable and clicking on a portion of it will show the filtered results).

Code example:

    from openwisp_utils.admin_theme import register_dashboard_chart

register_dashboard_chart(
    position=1,
    config={
        'query_params': {
            'name': 'Operator Project Distribution',
            'app_label': 'test_project',
            'model': 'operator',
            'group_by': 'project__name',
        },
        'colors': {'Utils': 'red', 'User': 'orange'},
    },
)

For real world examples, look at the code of OpenWISP Controller and OpenWISP Monitoring.

Note: an ImproperlyConfigured exception is raised if a dashboard element is already registered at same position.

It is recommended to register dashboard charts from the ready method of the AppConfig of the app where the models are defined. Checkout app.py of the test_project for reference.

unregister_dashboard_chart

This function can used to remove a chart from the dashboard.

Syntax:

unregister_dashboard_chart(chart_name)
Parameter Description
chart_name (str) The name of the chart to remove.

Code example:

from openwisp_utils.admin_theme import unregister_dashboard_chart

unregister_dashboard_chart('Operator Project Distribution')

Note: an ImproperlyConfigured exception is raised the specified dashboard chart is not registered.

Main navigation menu

Add openwisp_utils.admin_theme.context_processor.menu_items to template context_processors in settings.py as shown below.

TEMPLATES = [
    {
        'BACKEND': 'django.template.backends.django.DjangoTemplates',
        'DIRS': [],
        'OPTIONS': {
            'loaders': [
                # ... omitted ...
            ],
            'context_processors': [
                # ... other context processors ...
                'openwisp_utils.admin_theme.context_processor.menu_items'    # <----- add this
            ],
        },
    },
]

If you need to define custom menu items, see: OPENWISP_ADMIN_MENU_ITEMS.

Users will only be able to see menu items for objects they have permission to either view or edit.

Model utilities

openwisp_utils.base.UUIDModel

Model class which provides a UUID4 primary key.

openwisp_utils.base.TimeStampedEditableModel

Model class inheriting UUIDModel which provides two additional fields:

  • created
  • modified

Which use respectively AutoCreatedField, AutoLastModifiedField from model_utils.fields (self-updating fields providing the creation date-time and the last modified date-time).

openwisp_utils.base.KeyField

A model field whic provides a random key or token, widely used across openwisp modules.

Admin utilities

openwisp_utils.admin.TimeReadonlyAdminMixin

Admin mixin which adds two readonly fields created and modified.

This is an admin mixin for models inheriting TimeStampedEditableModel which adds the fields created and modified to the database.

openwisp_utils.admin.ReadOnlyAdmin

A read-only ModelAdmin base class.

Will include the id field by default, which can be excluded by supplying the exclude attribute, eg:

from openwisp_utils.admin import ReadOnlyAdmin

class PostAuthReadOnlyAdmin(ReadOnlyAdmin):
    exclude = ['id']

openwisp_utils.admin.AlwaysHasChangedMixin

A mixin designed for inline items and model forms, ensures the item is created even if the default values are unchanged.

Without this, when creating new objects, inline items won't be saved unless users change the default values.

openwisp_utils.admin.UUIDAdmin

An admin class that provides the UUID of the object as a read-only input field (to make it easy and quick to copy/paste).

openwisp_utils.admin.ReceiveUrlAdmin

An admin class that provides an URL as a read-only input field (to make it easy and quick to copy/paste).

Code utilities

openwisp_utils.utils.get_random_key

Generates an random string of 32 characters.

openwisp_utils.utils.register_menu_items

Takes input of a list of models name and settings_menu name and adds them to the side navigation bar in the admin dashboard interface.

Usage:

from openwisp_utils.utils import register_menu_items
from openwisp_utils.api.apps import ApiAppConfig

class YourAwesomeAppConfig(ApiAppConfig):
    def ready(self, *args, **kwargs):
        super().ready(*args, **kwargs)
        items = [{'model': 'your_project.your_model_name'}]
        # register_menu_items(items[, name_menu=YOUR_SETTINGS_MENU_NAME])
        register_menu_items(items, name_menu='OPENWISP_DEFAULT_ADMIN_MENU_ITEMS')

openwisp_utils.utils.deep_merge_dicts

Returns a new dict which is the result of the merge of the two dictionaries, all elements are deep-copied to avoid modifying the original data structures.

Usage:

from openwisp_utils.utils import deep_merge_dicts

mergd_dict = deep_merge_dicts(dict1, dict2)

openwisp_utils.utils.default_or_test

If the program is being executed during automated tests the value supplied in the test argument will be returned, otherwise the one supplied in the value argument is returned.

from openwisp_utils.utils import default_or_test

THROTTLE_RATE = getattr(
    settings,
    'THROTTLE_RATE',
    default_or_test(value='20/day', test=None),
)

openwisp_utils.utils.print_color

default colors: ['white_bold', 'green_bold', 'yellow_bold', 'red_bold']

If you want to print a string in Red Bold, you can do it as below.

from openwisp_utils.utils import print_color

print_color('This is the printed in Red Bold', color_name='red_bold')

You may also provide the end arguement similar to built-in print method.

openwisp_utils.utils.SorrtedOrderedDict

Extends collections.SortedDict and implements logic to sort inserted items based on key value. Sorting is done at insert operation which incurs memory space overhead.

Storage utilities

openwisp_utils.storage.CompressStaticFilesStorage

A static storage backend for compression inheriting from django-compress-staticfiles's CompressStaticFilesStorage class.

Adds support for excluding file types using OPENWISP_STATICFILES_VERSIONED_EXCLUDE setting.

To use point STATICFILES_STORAGE to openwisp_utils.storage.CompressStaticFilesStorage in settings.py.

STATICFILES_STORAGE = 'openwisp_utils.storage.CompressStaticFilesStorage'

REST API utilities

openwisp_utils.api.serializers.ValidatedModelSerializer

A model serializer which calls the model instance full_clean().

openwisp_utils.api.apps.ApiAppConfig

If you're creating an OpenWISP module which provides a REST API built with Django REST Framework, chances is that you may need to define some default settings to control its throttling or other aspects.

Here's how to easily do it:

from django.conf import settings
from django.utils.translation import ugettext_lazy as _
from openwisp_utils.api.apps import ApiAppConfig


class MyModuleConfig(ApiAppConfig):
    name = 'my_openwisp_module'
    label = 'my_module'
    verbose_name = _('My OpenWISP Module')

    # assumes API is enabled by default
    API_ENABLED = getattr(settings, 'MY_OPENWISP_MODULE_API_ENABLED', True)
    # set throttling rates for your module here
    REST_FRAMEWORK_SETTINGS = {
        'DEFAULT_THROTTLE_RATES': {'my_module': '400/hour'},
    }

Every openwisp module which has an API should use this class to configure its own default settings, which will be merged with the settings of the other modules.

Test utilities

openwisp_utils.tests.catch_signal

This method can be used to mock a signal call inorder to easily verify that the signal has been called.

Usage example as a context-manager:

from openwisp_utils.tests import catch_signal

with catch_signal(openwisp_signal) as handler:
    model_instance.trigger_signal()
    handler.assert_called_once_with(
        arg1='value1',
        arg2='value2',
        sender=ModelName,
        signal=openwisp_signal,
    )

openwisp_utils.tests.TimeLoggingTestRunner

https://raw.githubusercontent.com/openwisp/openwisp-utils/master/docs/TimeLoggingTestRunner.png

This class extends the default test runner provided by Django and logs the time spent by each test, making it easier to spot slow tests by highlighting time taken by it in yellow (time shall be highlighted in red if it crosses the second threshold).

By default tests are considered slow if they take more than 0.3 seconds but you can control this with OPENWISP_SLOW_TEST_THRESHOLD.

In order to switch to this test runner you have set the following in your settings.py:

TEST_RUNNER = 'openwisp_utils.tests.TimeLoggingTestRunner'

openwisp_utils.tests.capture_stdout

This decorator can be used to capture standard output produced by tests, either to silence it or to write assertions.

Example usage:

from openwisp_utils.tests import capture_stdout

@capture_stdout()
def test_something(self):
    function_generating_output() # pseudo code

@capture_stdout()
def test_something_again(self, captured_ouput):
    # pseudo code
    function_generating_output()
    # now you can create assertions on the captured output
    self.assertIn('expected stdout', captured_ouput.getvalue())
    # if there are more than one assertions, clear the captured output first
    captured_error.truncate(0)
    captured_error.seek(0)
    # you can create new assertion now
    self.assertIn('another output', captured_ouput.getvalue())

Notes:

  • If assertions need to be made on the captured output, an additional argument (in the example above is named captured_output) can be passed as an argument to the decorated test method, alternatively it can be omitted.
  • A StingIO instance is used for capturing output by default but if needed it's possible to pass a custom StringIO instance to the decorator function.

openwisp_utils.tests.capture_stderr

Equivalent to capture_stdout, but for standard error.

Example usage:

from openwisp_utils.tests import capture_stderr

@capture_stderr()
def test_error(self):
    function_generating_error() # pseudo code

@capture_stderr()
def test_error_again(self, captured_error):
    # pseudo code
    function_generating_error()
    # now you can create assertions on captured error
    self.assertIn('expected error', captured_error.getvalue())
    # if there are more than one assertions, clear the captured error first
    captured_error.truncate(0)
    captured_error.seek(0)
    # you can create new assertion now
    self.assertIn('another expected error', captured_error.getvalue())

openwisp_utils.tests.capture_any_output

Equivalent to capture_stdout and capture_stderr, but captures both types of output (standard output and standard error).

Example usage:

from openwisp_utils.tests import capture_any_output

@capture_any_output()
def test_something_out(self):
    function_generating_output() # pseudo code

@capture_any_output()
def test_out_again(self, captured_output, captured_error):
    # pseudo code
    function_generating_output_and_errors()
    # now you can create assertions on captured error
    self.assertIn('expected stdout', captured_output.getvalue())
    self.assertIn('expected stderr', captured_error.getvalue())

Quality Assurance Checks

This package contains some common QA checks that are used in the automated builds of different OpenWISP modules.

openwisp-qa-format

Shell script to automatically format Python code. It runs isort and black.

openwisp-qa-check

Shell script to run the following quality assurance checks:

If a check requires a flag, it can be passed forward in the same way.

Usage example:

openwisp-qa-check --migration-path <path> --message <commit-message>

Any unneeded checks can be skipped by passing --skip-<check-name>

Usage example:

openwisp-qa-check --skip-isort

You can do multiple checkmigrations by passing the arguments with space-delimited string.

For example, this multiple checkmigrations:

checkmigrations --migrations-to-ignore 3 \
        --migration-path ./openwisp_users/migrations/ || exit 1

checkmigrations --migrations-to-ignore 2 \
        --migration-path ./tests/testapp/migrations/ || exit 1

Can be changed with:

openwisp-qa-check --migrations-to-ignore "3 2" \
        --migration-path "./openwisp_users/migrations/ ./tests/testapp/migrations/"

checkmigrations

Ensures the latest migrations created have a human readable name.

We want to avoid having many migrations named like 0003_auto_20150410_3242.py.

This way we can reconstruct the evolution of our database schemas faster, with less efforts and hence less costs.

Usage example:

checkmigrations --migration-path ./django_freeradius/migrations/

checkcommit

Ensures the last commit message follows our commit message style guidelines.

We want to keep the commit log readable, consistent and easy to scan in order to make it easy to analyze the history of our modules, which is also a very important activity when performing maintenance.

Usage example:

checkcommit --message "$(git log --format=%B -n 1)"

If, for some reason, you wish to skip this QA check for a specific commit message you can add #noqa to the end of your commit message.

Usage example:

[qa] Improved #20

Simulation of a special unplanned case
#noqa

checkendline

Ensures that a blank line is kept at the end of each file.

checkpendingmigrations

Ensures there django migrations are up to date and no new migrations need to be created.

It accepts an optional --migration-module flag indicating the django app name that should be passed to ./manage.py makemigrations, eg: ./manage.py makemigrations $MIGRATION_MODULE.

checkrst

Checks the syntax of all ReStructuredText files to ensure they can be published on pypi or using python-sphinx.

Settings

OPENWISP_ADMIN_SITE_CLASS

default: openwisp_utils.admin_theme.admin.OpenwispAdminSite

If you need to use a customized admin site class, you can use this setting.

OPENWISP_ADMIN_SITE_TITLE

default: OpenWISP Admin

Title value used in the <title> HTML tag of the admin site.

OPENWISP_ADMIN_SITE_HEADER

default: OpenWISP

Heading text used in the main <h1> HTML tag (the logo) of the admin site.

OPENWISP_ADMIN_INDEX_TITLE

default: Network administration

Title shown to users in the index page of the admin site.

OPENWISP_ADMIN_DASHBOARD_ENABLED

default: False

When True, enables the OpenWISP Dashboard. Upon login, the user will be greeted with the dashboard instead of the default Django admin index page.

OPENWISP_ADMIN_MENU_ITEMS

default: []

Allows to pass a custom list of menu items to display in the admin menu.

If passed, overrides the default menu which is built by different openwisp modules.

The list should not include "home", "change password" and "log out", because those are automatically added and cannot be removed.

Example usage:

OPENWISP_ADMIN_MENU_ITEMS = [
    {'model': 'config.Device'},
    {'model': 'config.Template'},
    {'model': 'openwisp_users.User'},
    {
        'model': 'openwisp_radius.Accounting',
        'label': 'Radius sessions'  # custom label
    }
]

OPENWISP_ADMIN_THEME_LINKS

default: []

Note: this setting requires the admin_theme_settings context processor in order to work.

Allows to override the default CSS and favicon, as well as add extra <link> HTML elements if needed.

This setting overrides the default theme, you can reuse the default CSS or replace it entirely.

The following example shows how to keep using the default CSS, supply an additional CSS and replace the favicon.

Example usage:

OPENWISP_ADMIN_THEME_LINKS = [
    {'type': 'text/css', 'href': '/static/admin/css/openwisp.css', 'rel': 'stylesheet', 'media': 'all'},
    {'type': 'text/css', 'href': '/static/admin/css/custom-theme.css', 'rel': 'stylesheet', 'media': 'all'},
    {'type': 'image/x-icon', 'href': '/static/favicon.png', 'rel': 'icon'}
]

OPENWISP_ADMIN_THEME_JS

default: []

Allows to pass a list of strings representing URLs of custom JS files to load.

Example usage:

OPENWISP_ADMIN_THEME_JS = [
    '/static/custom-admin-theme.js',
]

OPENWISP_ADMIN_SHOW_USERLINKS_BLOCK

default: False

When True, enables Django user links on the admin site.

i.e. (USER NAME/ VIEW SITE / CHANGE PASSWORD / LOG OUT).

These links are already shown in the main navigation menu and for this reason are hidden by default.

OPENWISP_API_DOCS

default: True

Whether the OpenAPI documentation is enabled.

When enabled, you can view the available documentation using the Swagger endpoint at /api/v1/docs/.

You also need to add the following url to your project urls.py:

urlpatterns += [
    url(r'^api/v1/', include('openwisp_utils.api.urls')),
]

OPENWISP_API_INFO

default:

{
    'title': 'OpenWISP API',
    'default_version': 'v1',
    'description': 'OpenWISP REST API',
}

Define OpenAPI general information. NOTE: This setting requires OPENWISP_API_DOCS = True to take effect.

For more information about optional parameters check the drf-yasg documentation.

OPENWISP_SLOW_TEST_THRESHOLD

default: [0.3, 1] (seconds)

It can be used to change the thresholds used by TimeLoggingTestRunner to detect slow tests (0.3s by default) and highlight the slowest ones (1s by default) amongst them.

OPENWISP_STATICFILES_VERSIONED_EXCLUDE

default: ['leaflet/*/*.png']

Allows to pass a list of Unix shell-style wildcards for files to be excluded by CompressStaticFilesStorage.

By default Leaflet PNGs have been excluded to avoid bugs like openwisp/ansible-openwisp2#232.

Example usage:

OPENWISP_STATICFILES_VERSIONED_EXCLUDE = [
    '*png',
]

Installing for development

Install sqlite:

sudo apt-get install sqlite3 libsqlite3-dev

Install your forked repo:

git clone git://github.com/<your_fork>/openwisp-utils
cd openwisp-utils/
pip install -e .[qa,rest]

Install test requirements:

pip install -r requirements-test.txt

Set up the pre-push hook to run tests and QA checks automatically right before the git push action, so that if anything fails the push operation will be aborted:

openwisp-pre-push-hook --install

Create database:

cd tests/
./manage.py migrate
./manage.py createsuperuser

Run development server:

cd tests/
./manage.py runserver

You can access the admin interface of the test project at http://127.0.0.1:8000/admin/.

Run tests with:

./runtests.py --parallel

Contributing

Please refer to the OpenWISP contributing guidelines.

Support

See OpenWISP Support Channels.

Changelog

See CHANGES.

License

See LICENSE.