[Cloud Synchronization] Method to send/receive data with cloud infrastructure

[suresh-lc]

Is your feature request related to a problem? Please describe.
Currently all the device generated data is stored using EdgeX container in the coredata db. With time the amount of data stored would be increasing exponentially. And this would in turn make the device slower for service execution due to lower memory.

Describe the solution you'd like
In typical small devices(embedded system) the data would be stored at cloud infrastructure for future use. Hence it is always better to synchronize data to cloud. Method to save data to cloud infra is recommended.

Describe alternatives you've considered
Alternative method to save the data could be other high memory devices in the same network. But considering a Home ecosystem where all the devices are memory and CPU constrained.

Additional context
We need to define and consider clear method for the synchronization. Also the cloud system like AWS, Azure needs to be considered.

What/Where data resides :
Within Home Edge we have the device meta data - like configuration, network and service info.
With EdgeX we have the device generated data being stored.

Typical Use case:
Using Meta data - Understand the device configuration and services offered by these
Using Device data - Analytics on the device usage/performance, automation

Phase - I:
Synchronize the data residing in coredata to cloud system extending the functionality provided by EdgeX for cloud synchronization

Phase - II:
Method - I - Provide method to synchronize the data from coredata to cloud independent of EdgeX.
Method - II - Provide system to store the data generated by devices within Home Edge and consecutively method synchronize to cloud

[MoonkiHong]

Good suggestion! 💯 @suresh-lc

As a phase I, do you think of which candidate would be feasible for us to start with?

[tiokim]

Thanks for the great suggestion!
As for Method - I in Phase - II, we don't have to think about it. coredata is tightly coupled with edgex so it can use edgex cloud sync mechanism.
When it comes to Method - II, it would be good to provide some interfaces not only the data generated by home edge but also any service applications which want to publish data to cloud.

[suresh-lc]

Thanks for the great suggestion! As for Method - I in Phase - II, we don't have to think about it. coredata is tightly coupled with edgex so it can use edgex cloud sync mechanism. When it comes to Method - II, it would be good to provide some interfaces not only the data generated by home edge but also any service applications which want to publish data to cloud.

Thanks for the suggestions. When coming to method II, the synch of data to cloud can be done via REST or MQTT method. Supporting one method should be good to start of with, what is your opinion.

[tiokim]

Thanks for the great suggestion! As for Method - I in Phase - II, we don't have to think about it. coredata is tightly coupled with edgex so it can use edgex cloud sync mechanism. When it comes to Method - II, it would be good to provide some interfaces not only the data generated by home edge but also any service applications which want to publish data to cloud.

Thanks for the suggestions. When coming to method II, the synch of data to cloud can be done via REST or MQTT method. Supporting one method should be good to start of with, what is your opinion.

@suresh-lc If we choose MQTT, the cloud should connect to the MQTT broker.
What should the cloud do if we choose REST? Do we have to force some interfaces to the cloud?

[suresh-lc]

Thanks for the great suggestion! As for Method - I in Phase - II, we don't have to think about it. coredata is tightly coupled with edgex so it can use edgex cloud sync mechanism. When it comes to Method - II, it would be good to provide some interfaces not only the data generated by home edge but also any service applications which want to publish data to cloud.

Thanks for the suggestions. When coming to method II, the synch of data to cloud can be done via REST or MQTT method. Supporting one method should be good to start of with, what is your opinion.

@suresh-lc If we choose MQTT, the cloud should connect to the MQTT broker. What should the cloud do if we choose REST? Do we have to force some interfaces to the cloud?

In case of MQTT, the MQTT broker can run in the cloud and all the edge devices can connect to this mqtt broker. The service application will have to specify the cloud end point where the broker is running.

Incase of REST, we will have to expose interfaces to post the data to cloud. There needs to be active cloud endpoints to consume the data.

Was going through of how AWS IoT core works: https://aws.amazon.com/iot-core/features/?pg=ln&sec=hs.
Also to make the data generic across different cloud vendors, there is cloudevents : https://github.com/cloudevents/spec
Need to dig in deep of these although.

[tiokim]

@suresh-lc @MoonkiHong How about starting with MQTT?

[MoonkiHong]

Seems good. @suresh-lc Thought?

[suresh-lc]

@MoonkiHong @t25kim 👍 Was having the same thoughts of supporting with MQTT as first step.
The overall architecture was thinking as

1. Expose API to Service Application to send data
2. Have a MQTT client with Home Edge
3. MQTT broker would be configured at cloud by the service application
4. The MQTT broker configuration details would be given to Home Edge by service app
5. Home Edge would send the data to broker by publishing on event topic

[tiokim]

@MoonkiHong @t25kim 👍 Was having the same thoughts of supporting with MQTT as first step. The overall architecture was thinking as

1. Expose API to Service Application to send data

2. Have a MQTT client with Home Edge

3. MQTT broker would be configured at cloud by the service application

4. The MQTT broker configuration details would be given to Home Edge by service app

5. Home Edge would send the data to broker by publishing on event topic

@suresh-lc Thanks for sharing your idea.

Is the service application running on the device and part of the home edge module?
Does the service application launch the cloud MQTT broker?

[suresh-lc]

Is the service application running on the device and part of the home edge module?
Had referred to "When it comes to Method - II, it would be good to provide some interfaces not only the data generated by home edge but also any service applications which want to publish data to cloud." for the reference to Service Application. So to my understanding service application uses Home Edge API's. Service application is not part of Home Edge module.

Does the service application launch the cloud MQTT broker?
Yes service application configures and launches MQTT broker in cloud. This gives the developers to launch the broker in the cloud of their choice. Also developers can use the data stored in the cloud for further data analysis or machine learning purpose. Thus the Home Edge does not manage the cloud components and the cost involved.

[tiokim]

Is the service application running on the device and part of the home edge module?
Had referred to "When it comes to Method - II, it would be good to provide some interfaces not only the data generated by home edge but also any service applications which want to publish data to cloud." for the reference to Service Application. So to my understanding service application uses Home Edge API's. Service application is not part of Home Edge module.

Does the service application launch the cloud MQTT broker?
Yes service application configures and launches MQTT broker in cloud. This gives the developers to launch the broker in the cloud of their choice. Also developers can use the data stored in the cloud for further data analysis or machine learning purpose. Thus the Home Edge does not manage the cloud components and the cost involved.

@suresh-lc Seems good! Looking forward to seeing PR soon!

[suresh-lc]

Thinking of an architecture as below

AWS used as sample cloud where mqtt broker can run.

[suresh-lc]

Some thoughts on setting broker URL by the service application. Each service application can have its own broker running in different cloud. And hence when every time send api is being called by the service application data to be sent and URL of the broker could be the parameters. Connect from the mqtt client (Home Edge) to the broker would be made and then the data would be published to the broker.
After the base code has been added for mqtt client, further enhancement of making the api more generic would be thought of.

[suresh-lc]

Following would be the sub tasks:

• MQTT Client Connect and Publish data to AWS
• Expose Orchestration API. The API would internally use Publish to send data
• Add security : SSL based authentication
• Maintain separate service application ids and connection url (instance)
• Subscribe Data by MQTT Client of Home Edge from AWS
• Send Data from Home Edge to Service Application

[nitu-s-gupta]

Issue can be closed as the task mentioned in it are completed.

[suresh-lc]

Basic feature has been added. Design changes are in progress in another PR.