Python and Django functions, classes and settings re-used across different OpenWISP modules, stored here with the aim of avoiding code duplication and ease maintenance.
Don't repeat yourself!
- Configurable admin theme
- Configurable navigation menu
- OpenAPI / Swagger documentation
- Model utilities
- Storage utilities
- Admin utilities
- Code utilities
- REST API utilities
- Test utilities
- Quality assurance checks
Table of Contents:
- Current features
- Install stable version from pypi
- Install development version
- Using the
admin_theme - OpenWISP Dashboard
- Main navigation menu
- Model utilities
- Admin utilities
- Code utilities
- Storage utilities
- REST API utilities
- Test utilities
- Quality Assurance Checks
- Settings
OPENWISP_ADMIN_SITE_CLASSOPENWISP_ADMIN_SITE_TITLEOPENWISP_ADMIN_SITE_HEADEROPENWISP_ADMIN_INDEX_TITLEOPENWISP_ADMIN_DASHBOARD_ENABLEDOPENWISP_ADMIN_MENU_ITEMSOPENWISP_ADMIN_THEME_LINKSOPENWISP_ADMIN_THEME_JSOPENWISP_ADMIN_SHOW_USERLINKS_BLOCKOPENWISP_API_DOCSOPENWISP_API_INFOOPENWISP_SLOW_TEST_THRESHOLDOPENWISP_STATICFILES_VERSIONED_EXCLUDE
- Installing for development
- Contributing
- Support
- Changelog
- License
Install from pypi:
pip install openwisp-utils
# install optional dependencies for REST framework
pip install openwisp-utils[rest]
# install optional dependencies for tests (flake8, black and isort)
pip install openwisp-utils[qa]
# or install everything
pip install openwisp-utils[rest,qa]Install tarball:
pip install https://github.com/openwisp/openwisp-utils/tarball/masterAlternatively you can install via pip using git:
pip install -e git+git://github.com/openwisp/openwisp-utils#egg=openwisp-utilsThe admin theme requires Django >= 2.2..
Add openwisp_utils.admin_theme to INSTALLED_APPS in settings.py:
INSTALLED_APPS = [
'django.contrib.auth',
'django.contrib.contenttypes',
'django.contrib.sessions',
'django.contrib.messages',
'django.contrib.staticfiles',
'openwisp_utils.admin_theme', # <----- add this
'django.contrib.sites',
# admin
'django.contrib.admin',
]Add the list of all packages extended to EXTENDED_APPS in settings.py.
For example, if you've extended django_x509:
EXTENDED_APPS = ['django_x509']This is a static finder which looks for static files in the static
directory of the apps listed in settings.EXTENDED_APPS.
Add openwisp_utils.staticfiles.DependencyFinder to STATICFILES_FINDERS
in settings.py.
STATICFILES_FINDERS = [
'django.contrib.staticfiles.finders.FileSystemFinder',
'django.contrib.staticfiles.finders.AppDirectoriesFinder',
'openwisp_utils.staticfiles.DependencyFinder', # <----- add this
]This is a template loader which looks for templates in the templates
directory of the apps listed in settings.EXTENDED_APPS.
Add openwisp_utils.loaders.DependencyLoader to
template loaders in settings.py as shown below.
TEMPLATES = [
{
'BACKEND': 'django.template.backends.django.DjangoTemplates',
'DIRS': [],
'OPTIONS': {
'loaders': [
# ... other loaders ...
'openwisp_utils.loaders.DependencyLoader', # <----- add this
],
'context_processors': [
# ... omitted ...
],
},
},
]Add openwisp_utils.admin_theme.context_processor.admin_theme_settings to
template context_processors in settings.py as shown below.
This will allow to set OPENWISP_ADMIN_THEME_LINKS
and OPENWISP_ADMIN_THEME_JS settings
to provide CSS and JS files to customise admin theme.
TEMPLATES = [
{
'BACKEND': 'django.template.backends.django.DjangoTemplates',
'DIRS': [],
'OPTIONS': {
'loaders': [
# ... omitted ...
],
'context_processors': [
# ... other context processors ...
'openwisp_utils.admin_theme.context_processor.admin_theme_settings' # <----- add this
],
},
},
]Note
You will have to deploy these static files on your own.
In order to make django able to find and load these files
you may want to use the STATICFILES_DIR setting in settings.py.
You can learn more in the Django documentation.
The admin_theme sub app of this package provides an admin dashboard
for OpenWISP which can be manipulated with the functions described in
the next sections.
Example 1, monitoring:
Example 2, controller:
Allows including a specific django template in the OpenWISP dashboard.
It is designed to allow the inclusion of the geographic map shipped by OpenWISP Monitoring but can be used to include any custom element in the dashboard.
Note: templates are loaded before charts.
Syntax:
register_dashboard_template(position, config)| Parameter | Description |
position |
(int) The position of the template. |
config |
(dict) The configuration of the template. |
Following properties can be configured for each template config:
| Property | Description |
template |
(str) Path to pass to the template loader. |
css |
(tuple) List of CSS files to load in the HTML page. |
js |
(tuple) List of Javascript files to load in the HTML page. |
Code example:
from openwisp_utils.admin_theme import register_dashboard_template
register_dashboard_template(
position=0,
config={
'template': 'admin/dashboard/device_map.html',
'css': (
'monitoring/css/device-map.css',
'leaflet/leaflet.css',
'monitoring/css/leaflet.fullscreen.css',
),
'js': (
'monitoring/js/device-map.js',
'leaflet/leaflet.js',
'leaflet/leaflet.extras.js',
'monitoring/js/leaflet.fullscreen.min.js'
)
}
)It is recommended to register dashboard templates from the ready
method of the AppConfig of the app where the templates are defined.
This function can be used to remove a template from the dashboard.
Syntax:
unregister_dashboard_template(template_name)| Parameter | Description |
template_name |
(str) The name of the template to remove. |
Code example:
from openwisp_utils.admin_theme import unregister_dashboard_template
unregister_dashboard_template('admin/dashboard/device_map.html')Note: an ImproperlyConfigured exception is raised the
specified dashboard template is not registered.
Adds a chart to the OpenWISP dashboard.
At the moment only pie charts are supported.
The code works by defining the type of query which will be executed, and optionally, how the returned values have to be colored and labeled.
Syntax:
register_dashboard_chart(position, config)| Parameter | Description |
position |
(int) Position of the chart. |
config |
(dict) Configuration of chart. |
Following properties can be configured for each chart config:
| Property | Description | ||||||||||||||
query_param |
It is a required property in form of
|
||||||||||||||
colors |
An optional dict which can be used to define colors for each distinct
value shown in the pie charts. |
||||||||||||||
labels |
An optional dict which can be used to define translatable strings for each distinct
value shown in the pie charts. Can be used also to provide fallback human readable values for
raw values stored in the database which would be otherwise hard to understand for the user. |
||||||||||||||
filters |
An optional dict which can be used when using aggregate and annotate in
query_params to define the link that will be generated to filter results (pie charts are
clickable and clicking on a portion of it will show the filtered results). |
Code example:
from openwisp_utils.admin_theme import register_dashboard_chart
register_dashboard_chart(
position=1,
config={
'query_params': {
'name': 'Operator Project Distribution',
'app_label': 'test_project',
'model': 'operator',
'group_by': 'project__name',
},
'colors': {'Utils': 'red', 'User': 'orange'},
},
)For real world examples, look at the code of OpenWISP Controller and OpenWISP Monitoring.
Note: an ImproperlyConfigured exception is raised if a
dashboard element is already registered at same position.
It is recommended to register dashboard charts from the ready method
of the AppConfig of the app where the models are defined.
Checkout app.py of the test_project
for reference.
This function can used to remove a chart from the dashboard.
Syntax:
unregister_dashboard_chart(chart_name)| Parameter | Description |
chart_name |
(str) The name of the chart to remove. |
Code example:
from openwisp_utils.admin_theme import unregister_dashboard_chart
unregister_dashboard_chart('Operator Project Distribution')Note: an ImproperlyConfigured exception is raised the
specified dashboard chart is not registered.
Add openwisp_utils.admin_theme.context_processor.menu_items to
template context_processors in settings.py as shown below.
TEMPLATES = [
{
'BACKEND': 'django.template.backends.django.DjangoTemplates',
'DIRS': [],
'OPTIONS': {
'loaders': [
# ... omitted ...
],
'context_processors': [
# ... other context processors ...
'openwisp_utils.admin_theme.context_processor.menu_items' # <----- add this
],
},
},
]If you need to define custom menu items, see: OPENWISP_ADMIN_MENU_ITEMS.
Users will only be able to see menu items for objects they have permission to either view or edit.
Model class which provides a UUID4 primary key.
Model class inheriting UUIDModel which provides two additional fields:
createdmodified
Which use respectively AutoCreatedField, AutoLastModifiedField from model_utils.fields
(self-updating fields providing the creation date-time and the last modified date-time).
A model field whic provides a random key or token, widely used across openwisp modules.
Admin mixin which adds two readonly fields created and modified.
This is an admin mixin for models inheriting TimeStampedEditableModel
which adds the fields created and modified to the database.
A read-only ModelAdmin base class.
Will include the id field by default, which can be excluded by supplying
the exclude attribute, eg:
from openwisp_utils.admin import ReadOnlyAdmin
class PostAuthReadOnlyAdmin(ReadOnlyAdmin):
exclude = ['id']A mixin designed for inline items and model forms, ensures the item is created even if the default values are unchanged.
Without this, when creating new objects, inline items won't be saved unless users change the default values.
An admin class that provides the UUID of the object as a read-only input field (to make it easy and quick to copy/paste).
An admin class that provides an URL as a read-only input field (to make it easy and quick to copy/paste).
Generates an random string of 32 characters.
Takes input of a list of models name and settings_menu name and adds them to the side navigation bar in the admin dashboard interface.
Usage:
from openwisp_utils.utils import register_menu_items
from openwisp_utils.api.apps import ApiAppConfig
class YourAwesomeAppConfig(ApiAppConfig):
def ready(self, *args, **kwargs):
super().ready(*args, **kwargs)
items = [{'model': 'your_project.your_model_name'}]
# register_menu_items(items[, name_menu=YOUR_SETTINGS_MENU_NAME])
register_menu_items(items, name_menu='OPENWISP_DEFAULT_ADMIN_MENU_ITEMS')Returns a new dict which is the result of the merge of the two dictionaries,
all elements are deep-copied to avoid modifying the original data structures.
Usage:
from openwisp_utils.utils import deep_merge_dicts
mergd_dict = deep_merge_dicts(dict1, dict2)If the program is being executed during automated tests the value supplied in
the test argument will be returned, otherwise the one supplied in the
value argument is returned.
from openwisp_utils.utils import default_or_test
THROTTLE_RATE = getattr(
settings,
'THROTTLE_RATE',
default_or_test(value='20/day', test=None),
)default colors: ['white_bold', 'green_bold', 'yellow_bold', 'red_bold']
If you want to print a string in Red Bold, you can do it as below.
from openwisp_utils.utils import print_color
print_color('This is the printed in Red Bold', color_name='red_bold')You may also provide the end arguement similar to built-in print method.
Extends collections.SortedDict and implements logic to sort inserted
items based on key value. Sorting is done at insert operation which
incurs memory space overhead.
A static storage backend for compression inheriting from django-compress-staticfiles's CompressStaticFilesStorage class.
Adds support for excluding file types using OPENWISP_STATICFILES_VERSIONED_EXCLUDE setting.
To use point STATICFILES_STORAGE to openwisp_utils.storage.CompressStaticFilesStorage in settings.py.
STATICFILES_STORAGE = 'openwisp_utils.storage.CompressStaticFilesStorage'A model serializer which calls the model instance full_clean().
If you're creating an OpenWISP module which provides a REST API built with Django REST Framework, chances is that you may need to define some default settings to control its throttling or other aspects.
Here's how to easily do it:
from django.conf import settings
from django.utils.translation import ugettext_lazy as _
from openwisp_utils.api.apps import ApiAppConfig
class MyModuleConfig(ApiAppConfig):
name = 'my_openwisp_module'
label = 'my_module'
verbose_name = _('My OpenWISP Module')
# assumes API is enabled by default
API_ENABLED = getattr(settings, 'MY_OPENWISP_MODULE_API_ENABLED', True)
# set throttling rates for your module here
REST_FRAMEWORK_SETTINGS = {
'DEFAULT_THROTTLE_RATES': {'my_module': '400/hour'},
}Every openwisp module which has an API should use this class to configure its own default settings, which will be merged with the settings of the other modules.
This method can be used to mock a signal call inorder to easily verify that the signal has been called.
Usage example as a context-manager:
from openwisp_utils.tests import catch_signal
with catch_signal(openwisp_signal) as handler:
model_instance.trigger_signal()
handler.assert_called_once_with(
arg1='value1',
arg2='value2',
sender=ModelName,
signal=openwisp_signal,
)
This class extends the default test runner provided by Django and logs the time spent by each test, making it easier to spot slow tests by highlighting time taken by it in yellow (time shall be highlighted in red if it crosses the second threshold).
By default tests are considered slow if they take more than 0.3 seconds but you can control this with OPENWISP_SLOW_TEST_THRESHOLD.
In order to switch to this test runner you have set the following in your settings.py:
TEST_RUNNER = 'openwisp_utils.tests.TimeLoggingTestRunner'This decorator can be used to capture standard output produced by tests, either to silence it or to write assertions.
Example usage:
from openwisp_utils.tests import capture_stdout
@capture_stdout()
def test_something(self):
function_generating_output() # pseudo code
@capture_stdout()
def test_something_again(self, captured_ouput):
# pseudo code
function_generating_output()
# now you can create assertions on the captured output
self.assertIn('expected stdout', captured_ouput.getvalue())
# if there are more than one assertions, clear the captured output first
captured_error.truncate(0)
captured_error.seek(0)
# you can create new assertion now
self.assertIn('another output', captured_ouput.getvalue())Notes:
- If assertions need to be made on the captured output, an additional argument
(in the example above is named
captured_output) can be passed as an argument to the decorated test method, alternatively it can be omitted. - A
StingIOinstance is used for capturing output by default but if needed it's possible to pass a customStringIOinstance to the decorator function.
Equivalent to capture_stdout, but for standard error.
Example usage:
from openwisp_utils.tests import capture_stderr
@capture_stderr()
def test_error(self):
function_generating_error() # pseudo code
@capture_stderr()
def test_error_again(self, captured_error):
# pseudo code
function_generating_error()
# now you can create assertions on captured error
self.assertIn('expected error', captured_error.getvalue())
# if there are more than one assertions, clear the captured error first
captured_error.truncate(0)
captured_error.seek(0)
# you can create new assertion now
self.assertIn('another expected error', captured_error.getvalue())Equivalent to capture_stdout and capture_stderr, but captures both types of
output (standard output and standard error).
Example usage:
from openwisp_utils.tests import capture_any_output
@capture_any_output()
def test_something_out(self):
function_generating_output() # pseudo code
@capture_any_output()
def test_out_again(self, captured_output, captured_error):
# pseudo code
function_generating_output_and_errors()
# now you can create assertions on captured error
self.assertIn('expected stdout', captured_output.getvalue())
self.assertIn('expected stderr', captured_error.getvalue())This package contains some common QA checks that are used in the automated builds of different OpenWISP modules.
Shell script to automatically format Python code. It runs isort and black.
Shell script to run the following quality assurance checks:
- checkmigrations
- checkcommit
- checkendline
- checkpendingmigrations
- checkrst
flake8- Python code linterisort- Sorts python imports alphabetically, and seperated into sectionsblack- Formats python code using a common standard
If a check requires a flag, it can be passed forward in the same way.
Usage example:
openwisp-qa-check --migration-path <path> --message <commit-message>
Any unneeded checks can be skipped by passing --skip-<check-name>
Usage example:
openwisp-qa-check --skip-isort
You can do multiple checkmigrations by passing the arguments with space-delimited string.
For example, this multiple checkmigrations:
checkmigrations --migrations-to-ignore 3 \
--migration-path ./openwisp_users/migrations/ || exit 1
checkmigrations --migrations-to-ignore 2 \
--migration-path ./tests/testapp/migrations/ || exit 1
Can be changed with:
openwisp-qa-check --migrations-to-ignore "3 2" \
--migration-path "./openwisp_users/migrations/ ./tests/testapp/migrations/"
Ensures the latest migrations created have a human readable name.
We want to avoid having many migrations named like 0003_auto_20150410_3242.py.
This way we can reconstruct the evolution of our database schemas faster, with less efforts and hence less costs.
Usage example:
checkmigrations --migration-path ./django_freeradius/migrations/
Ensures the last commit message follows our commit message style guidelines.
We want to keep the commit log readable, consistent and easy to scan in order to make it easy to analyze the history of our modules, which is also a very important activity when performing maintenance.
Usage example:
checkcommit --message "$(git log --format=%B -n 1)"
If, for some reason, you wish to skip this QA check for a specific commit message
you can add #noqa to the end of your commit message.
Usage example:
[qa] Improved #20 Simulation of a special unplanned case #noqa
Ensures that a blank line is kept at the end of each file.
Ensures there django migrations are up to date and no new migrations need to be created.
It accepts an optional --migration-module flag indicating the django app
name that should be passed to ./manage.py makemigrations, eg:
./manage.py makemigrations $MIGRATION_MODULE.
Checks the syntax of all ReStructuredText files to ensure they can be published on pypi or using python-sphinx.
default: openwisp_utils.admin_theme.admin.OpenwispAdminSite
If you need to use a customized admin site class, you can use this setting.
default: OpenWISP Admin
Title value used in the <title> HTML tag of the admin site.
default: OpenWISP
Heading text used in the main <h1> HTML tag (the logo) of the admin site.
default: Network administration
Title shown to users in the index page of the admin site.
default: False
When True, enables the OpenWISP Dashboard.
Upon login, the user will be greeted with the dashboard instead of the default
Django admin index page.
default: []
Allows to pass a custom list of menu items to display in the admin menu.
If passed, overrides the default menu which is built by different openwisp modules.
The list should not include "home", "change password" and "log out", because those are automatically added and cannot be removed.
Example usage:
OPENWISP_ADMIN_MENU_ITEMS = [
{'model': 'config.Device'},
{'model': 'config.Template'},
{'model': 'openwisp_users.User'},
{
'model': 'openwisp_radius.Accounting',
'label': 'Radius sessions' # custom label
}
]default: []
Note: this setting requires the admin_theme_settings context processor in order to work.
Allows to override the default CSS and favicon, as well as add extra <link> HTML elements if needed.
This setting overrides the default theme, you can reuse the default CSS or replace it entirely.
The following example shows how to keep using the default CSS, supply an additional CSS and replace the favicon.
Example usage:
OPENWISP_ADMIN_THEME_LINKS = [
{'type': 'text/css', 'href': '/static/admin/css/openwisp.css', 'rel': 'stylesheet', 'media': 'all'},
{'type': 'text/css', 'href': '/static/admin/css/custom-theme.css', 'rel': 'stylesheet', 'media': 'all'},
{'type': 'image/x-icon', 'href': '/static/favicon.png', 'rel': 'icon'}
]default: []
Allows to pass a list of strings representing URLs of custom JS files to load.
Example usage:
OPENWISP_ADMIN_THEME_JS = [
'/static/custom-admin-theme.js',
]default: False
When True, enables Django user links on the admin site.
i.e. (USER NAME/ VIEW SITE / CHANGE PASSWORD / LOG OUT).
These links are already shown in the main navigation menu and for this reason are hidden by default.
default: True
Whether the OpenAPI documentation is enabled.
When enabled, you can view the available documentation using the
Swagger endpoint at /api/v1/docs/.
You also need to add the following url to your project urls.py:
urlpatterns += [
url(r'^api/v1/', include('openwisp_utils.api.urls')),
]default:
{
'title': 'OpenWISP API',
'default_version': 'v1',
'description': 'OpenWISP REST API',
}Define OpenAPI general information.
NOTE: This setting requires OPENWISP_API_DOCS = True to take effect.
For more information about optional parameters check the drf-yasg documentation.
default: [0.3, 1] (seconds)
It can be used to change the thresholds used by TimeLoggingTestRunner to detect slow tests (0.3s by default) and highlight the slowest ones (1s by default) amongst them.
default: ['leaflet/*/*.png']
Allows to pass a list of Unix shell-style wildcards for files to be excluded by CompressStaticFilesStorage.
By default Leaflet PNGs have been excluded to avoid bugs like openwisp/ansible-openwisp2#232.
Example usage:
OPENWISP_STATICFILES_VERSIONED_EXCLUDE = [
'*png',
]Install sqlite:
sudo apt-get install sqlite3 libsqlite3-devInstall your forked repo:
git clone git://github.com/<your_fork>/openwisp-utils
cd openwisp-utils/
pip install -e .[qa,rest]Install test requirements:
pip install -r requirements-test.txtSet up the pre-push hook to run tests and QA checks automatically right before the git push action, so that if anything fails the push operation will be aborted:
openwisp-pre-push-hook --installCreate database:
cd tests/
./manage.py migrate
./manage.py createsuperuserRun development server:
cd tests/
./manage.py runserverYou can access the admin interface of the test project at http://127.0.0.1:8000/admin/.
Run tests with:
./runtests.py --parallelPlease refer to the OpenWISP contributing guidelines.
See OpenWISP Support Channels.
See CHANGES.
See LICENSE.