Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cache Mongo-DB calls (in-memory and/or Redis) #926

Draft
wants to merge 29 commits into
base: master
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
29 commits
Select commit Hold shift + click to select a range
4e6f9f1
Cache Mongo-DB calls.
jason-fox Oct 26, 2020
7872370
Linting - move functions.
jason-fox Oct 26, 2020
c2a7ea0
Merge branch 'master' into feature/cache
jason-fox Nov 2, 2020
9471f42
Make Caching configurable.
jason-fox Nov 2, 2020
7a8edc8
Add documentation
jason-fox Nov 2, 2020
d0fe4f8
Amend documentation.
jason-fox Nov 2, 2020
5d08c0e
Tidying up code and tests.
jason-fox Nov 2, 2020
6b6dba7
Merge branch 'master' into feature/cache
jason-fox Nov 16, 2020
3c1190c
Merge commit 'master' into feature/cache
jason-fox Nov 18, 2020
60d7dca
Complete linting using ESLint.
jason-fox Nov 18, 2020
56c8023
Merge branch 'master' into feature/cache
jason-fox Nov 27, 2020
90d1318
Merge branch 'master' into feature/cache
jason-fox Dec 4, 2020
acad317
Fixing merge
jason-fox Dec 4, 2020
4b7a08a
Adding Redis Cache
jason-fox Dec 7, 2020
b0daf0a
Adding Start-up test for Redis and Memcache
jason-fox Dec 7, 2020
5eaef79
Updating Docs
jason-fox Dec 7, 2020
94bc29e
Merge branch 'master' into feature/cache
jason-fox Dec 10, 2020
76dcd15
Remove debug
jason-fox Dec 10, 2020
7d88f72
Merge branch 'master' into feature/cache
jason-fox Dec 15, 2020
bb0c277
Merge branch 'master' into feature/cache
jason-fox Jan 21, 2021
55e017b
Merge branch 'master' into feature/cache
jason-fox Feb 11, 2021
a0b1dd8
Merge branch 'master' into feature/cache
jason-fox Feb 12, 2021
880f164
Merge branch 'master' into feature/cache
jason-fox Mar 2, 2021
cfdd819
Merge branch 'master' into feature/cache
jason-fox Mar 16, 2021
d39a90b
Merge branch 'master' into feature/cache
jason-fox Apr 13, 2021
e98c02e
Set cache rather than don't cache
jason-fox Apr 13, 2021
7753fec
Delete file.
jason-fox Jun 24, 2021
f3d6622
Merge branch 'master' into feature/cache
jason-fox Jun 24, 2021
12f9bcb
Merge branch 'master' into feature/cache
jason-fox Sep 17, 2021
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,11 @@ jobs:
image: mongo:4.2
ports:
- 27017:27017
redis:
image: redis:6
ports:
- 6379:6379

strategy:
matrix:
node-version:
Expand Down Expand Up @@ -76,6 +81,10 @@ jobs:
image: mongo:4.2
ports:
- 27017:27017
redis:
image: redis:6
ports:
- 6379:6379
steps:
- name: Git checkout
uses: actions/checkout@v2
Expand Down
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -364,8 +364,8 @@ related with the fact that different people assign different interpretations on
used in these licenses. Due to this, some people believe that there is a risk in just _using_ software under GPL or AGPL
licenses (even without _modifying_ it).

For the avoidance of doubt, the owners of this software licensed under an AGPL-3.0 license
wish to make a clarifying public statement as follows:
For the avoidance of doubt, the owners of this software licensed under an AGPL-3.0 license wish to make a clarifying
public statement as follows:

> Please note that software derived as a result of modifying the source code of this software in order to fix a bug or
> incorporate enhancements is considered a derivative work of the product. Software that merely uses or aggregates (i.e.
Expand Down
94 changes: 48 additions & 46 deletions doc/advanced-topics.md
Original file line number Diff line number Diff line change
@@ -1,37 +1,36 @@
## Advanced Topics

* [Secured access to the Context Broker](#secured-access-to-the-context-broker)
* [GeoJSON support NGSI-LD only](#geojson-support-ngsi-ld-only)
* [Metadata support](#metadata-support)
* [NGSI LD data and metadata considerations](#ngsi-ld-data-and-metadata-considerations)
* [NGSI-LD Linked Data support](#ngsi-ld-linked-data-support)
* [Autoprovision configuration (autoprovision)](#autoprovision-configuration-autoprovision)
* [Explicitly defined attributes (explicitAttrs)](#explicitly-defined-attributes-explicitattrs)
* [Configuring operation to persist the data in Context Broker (appendMode)](#configuring-operation-to-persist-the-data-in-context-broker-appendmode)
* [Data mapping plugins](#data-mapping-plugins)
* [Development](#development)
* [Provided plugins](#provided-plugins)
* [Timestamp Compression plugin (compressTimestamp)](#timestamp-compression-plugin-compresstimestamp)
* [Attribute Alias plugin (attributeAlias)](#attribute-alias-plugin-attributealias)
* [Event plugin (addEvents)](#event-plugin-addevents)
* [Timestamp Processing Plugin (timestampProcess)](#timestamp-processing-plugin-timestampprocess)
* [Expression Translation plugin (expressionTransformation)](#expression-translation-plugin-expressiontransformation)
* [Multientity plugin (multiEntity)](#multientity-plugin-multientity)
* [Bidirectionality plugin (bidirectional)](#bidirectionality-plugin-bidirectional)
* [Autoprovision configuration (autoprovision)](#autoprovision-configuration-autoprovision)
* [Explicitly defined attributes (explicitAttrs)](#explicitly-defined-attributes-explicitattrs)
* [Configuring operation to persist the data in Context Broker (appendMode)](#configuring-operation-to-persist-the-data-in-context-broker-appendmode)
* [Old IoTAgent data migration](#old-iotagent-data-migration)

- [Secured access to the Context Broker](#secured-access-to-the-context-broker)
- [GeoJSON support NGSI-LD only](#geojson-support-ngsi-ld-only)
- [Metadata support](#metadata-support)
- [NGSI LD data and metadata considerations](#ngsi-ld-data-and-metadata-considerations)
- [NGSI-LD Linked Data support](#ngsi-ld-linked-data-support)
- [Autoprovision configuration (autoprovision)](#autoprovision-configuration-autoprovision)
- [Explicitly defined attributes (explicitAttrs)](#explicitly-defined-attributes-explicitattrs)
- [Configuring operation to persist the data in Context Broker (appendMode)](#configuring-operation-to-persist-the-data-in-context-broker-appendmode)
- [Data mapping plugins](#data-mapping-plugins)
- [Development](#development)
- [Provided plugins](#provided-plugins)
- [Timestamp Compression plugin (compressTimestamp)](#timestamp-compression-plugin-compresstimestamp)
- [Attribute Alias plugin (attributeAlias)](#attribute-alias-plugin-attributealias)
- [Event plugin (addEvents)](#event-plugin-addevents)
- [Timestamp Processing Plugin (timestampProcess)](#timestamp-processing-plugin-timestampprocess)
- [Expression Translation plugin (expressionTransformation)](#expression-translation-plugin-expressiontransformation)
- [Multientity plugin (multiEntity)](#multientity-plugin-multientity)
- [Bidirectionality plugin (bidirectional)](#bidirectionality-plugin-bidirectional)
- [Autoprovision configuration (autoprovision)](#autoprovision-configuration-autoprovision)
- [Explicitly defined attributes (explicitAttrs)](#explicitly-defined-attributes-explicitattrs)
- [Configuring operation to persist the data in Context Broker (appendMode)](#configuring-operation-to-persist-the-data-in-context-broker-appendmode)
- [Old IoTAgent data migration](#old-iotagent-data-migration)

### Secured access to the Context Broker

For access to instances of the Context Broker secured with a
[PEP Proxy](https://github.com/telefonicaid/fiware-orion-pep), an authentication mechanism based in Keystone Trust
tokens is provided. A trust token is a way of Keystone to allow an user delegates a role to another user for a subservice.
It is a long-term token that can be issued by any user to give another user permissions
to impersonate him with a given role in a given project (subservice). Such impersonation itself is in turn based on a short-term
access token.
tokens is provided. A trust token is a way of Keystone to allow an user delegates a role to another user for a
subservice. It is a long-term token that can be issued by any user to give another user permissions to impersonate him
with a given role in a given project (subservice). Such impersonation itself is in turn based on a short-term access
token.

For the authentication mechanisms to work, the `authentication` attribute in the configuration has to be fully
configured, and the `authentication.enabled` subattribute should have the value `true`.
Expand Down Expand Up @@ -73,8 +72,8 @@ Agent.

Complete info on Keystone trust tokens could be found at:

- [Trusts concept](https://docs.openstack.org/keystone/stein/user/trusts)
- [Trusts API](https://docs.openstack.org/keystone/stein/api_curl_examples.html#post-v3-os-trust-trusts)
- [Trusts concept](https://docs.openstack.org/keystone/stein/user/trusts)
- [Trusts API](https://docs.openstack.org/keystone/stein/api_curl_examples.html#post-v3-os-trust-trusts)

### GeoJSON support NGSI-LD only

Expand Down Expand Up @@ -269,26 +268,28 @@ updated as shown:

### Autoprovision configuration (autoprovision)

By default, when a measure arrives to the IoTAgent, if the `device_id` does not match with an existing one, then, the IoTA
creates a new device and a new entity according to the group config. Defining the field `autoprovision` to `false`
when provisioning the device group, the IoTA to reject the measure at the southbound, allowing only to persist the
data to devices that are already provisioned. It makes no sense to use this field in device provisioning since it is
intended to avoid provisioning devices (and for it to be effective, it would have to be provisional).
By default, when a measure arrives to the IoTAgent, if the `device_id` does not match with an existing one, then, the
IoTA creates a new device and a new entity according to the group config. Defining the field `autoprovision` to `false`
when provisioning the device group, the IoTA to reject the measure at the southbound, allowing only to persist the data
to devices that are already provisioned. It makes no sense to use this field in device provisioning since it is intended
to avoid provisioning devices (and for it to be effective, it would have to be provisional).

### Explicitly defined attributes (explicitAttrs)

If a given measure element (object_id) is not defined in the mappings of the device or group provision, the measure is stored
in the Context Broker by adding a new attribute to the entity with the same name of the undefined measure element. By adding the
field `explicitAttrs` with `true` value to device or group provision, the IoTAgent rejects the measure elements that are not defined
in the mappings of device or group provision, persisting only the one defined in the mappings of the provision. If `explicitAttrs`
is provided both at device and group level, the device level takes precedence.
If a given measure element (object_id) is not defined in the mappings of the device or group provision, the measure is
stored in the Context Broker by adding a new attribute to the entity with the same name of the undefined measure
element. By adding the field `explicitAttrs` with `true` value to device or group provision, the IoTAgent rejects the
measure elements that are not defined in the mappings of device or group provision, persisting only the one defined in
the mappings of the provision. If `explicitAttrs` is provided both at device and group level, the device level takes
precedence.

### Configuring operation to persist the data in Context Broker (appendMode)

This is a flag that can be enabled by activating the parameter `appendMode` in the configuration file or by using the `IOTA_APPEND_MODE`
environment variable (more info [here](https://github.com/telefonicaid/iotagent-node-lib/blob/master/doc/installationguide.md)).
If this flag is activated, the update requests to the Context Broker will be performed always with APPEND type, instead of the
default UPDATE. This have implications in the use of attributes with Context Providers, so this flag should be used with care.
This is a flag that can be enabled by activating the parameter `appendMode` in the configuration file or by using the
`IOTA_APPEND_MODE` environment variable (more info
[here](https://github.com/telefonicaid/iotagent-node-lib/blob/master/doc/installationguide.md)). If this flag is
activated, the update requests to the Context Broker will be performed always with APPEND type, instead of the default
UPDATE. This have implications in the use of attributes with Context Providers, so this flag should be used with care.

### Data mapping plugins

Expand Down Expand Up @@ -373,8 +374,8 @@ events in the IoT Agent with the configured type name will be marked as events.

##### Timestamp Processing Plugin (timestampProcess)

This plugin processes the entity attributes looking for a `TimeInstant` attribute. If one is found, for NGSIv2,
the plugin adds a `TimeInstant` attribute as metadata for every other attribute in the same request. With NGSI-LD, the
This plugin processes the entity attributes looking for a `TimeInstant` attribute. If one is found, for NGSIv2, the
plugin adds a `TimeInstant` attribute as metadata for every other attribute in the same request. With NGSI-LD, the
Standard `observedAt` property-of-a-property is used instead.

##### Expression Translation plugin (expressionTransformation)
Expand Down Expand Up @@ -438,7 +439,8 @@ When a device is provisioned with bidirectional attributes, the IoTAgent subscri
change notification for that attribute arrives to the IoTA, it applies the transformation defined in the device
provisioning payload to the notification, and calls the underlying notification handler with the transformed entity.

The following `attributes` section shows an example of the plugin configuration (using IOTA_AUTOCAST=false to avoid translation from geo:point to geo:json)
The following `attributes` section shows an example of the plugin configuration (using IOTA_AUTOCAST=false to avoid
translation from geo:point to geo:json)

```json
"attributes": [
Expand Down
Loading