Skip to content

Latest commit

 

History

History
948 lines (745 loc) · 42.7 KB

api.md

File metadata and controls

948 lines (745 loc) · 42.7 KB
Raw

Devo API (Client)

Overview

This library performs queries to the Client API (Search rest api) of Devo.

Endpoints

API

To perform a request with API, first choose the required endpoint depending on the region your are accessing from:

Region Endpoint
USA https://apiv2-us.devo.com/search/query
Canada https://apiv2-ca.devo.com/search/query
Europe https://apiv2-eu.devo.com/search/query
Spain https://apiv2-es.devo.com/search/query
APAC https://api-apac.devo.com/search/query

You have more information in the official documentation of Devo, REST API.

Differences in use from version 2 to 3

You have a special README to quickly show the important changes suffered from version 2 to 3

You can go read it here

Differences in use from version 3 to 4

You have a special README to quickly show the important changes suffered from version 3 to 4

You can go read it here

USAGE

Constructor

  • address: endpoint
  • auth: object with auth params (key and secret, token or jwt)
    • key: The key of the domain
    • secret: The secret of the domain
    • token: Auth token
    • jwt: JWT token
  • retries: number of retries for connection errors in a query (0, no retry by default)
  • timeout: timeout of socket (300 seconds by default)
  • retry_delay: retry delay value in seconds (5 seconds by default). This is the base delay for the Exponential backoff algorithm with rate reduction of 2
  • config: dict or ClientConfig object
  • verify: whether enable or disable the TLS authentication of endpoint (True by default). DEVO will always provide a secure endpoint for ALL its services. Disable at your own risk.
from devo.api import Client

api = Client(auth= {"key":"myapikey", "secret":"myapisecret"}, 
             address="https://apiv2-eu.devo.com/search/query")


api = Client(auth={"token":"myauthtoken"},
             address="https://apiv2-eu.devo.com/search/query")

api = Client(auth={"jwt":"myauthtoken"},
             address="https://apiv2-eu.devo.com/search/query")

ClientConfig

  • processor: processor for response, default is None
  • response: format of response
  • destination: Destination options, see Documentation for more info
  • stream: Stream queries or not
  • pragmas: pragmas por query: {"user": "Username", "app_name": "app name", "comment": "This query is for x"}
    • All pragmas are optional
  • keepAliveToken: KeepAlive token for long responses queries, see specific Section in this document for more info
from devo.api import Client, ClientConfig

config = ClientConfig(response="json", pragmas={"user": "jose_ramon_id153"})

api = Client(auth= {"key":"myapikey", "secret":"myapisecret"}, 
             address="https://apiv2-eu.devo.com/search/query",
             config=config)


config = ClientConfig(response="json/compact", stream=False,
                      pragmas={"user": "jose_ramon_id153", "app_name": "devo-sdk-tests-master"})

api = Client(auth= {"key":"myapikey", "secret":"myapisecret"}, 
             address="https://apiv2-eu.devo.com/search/query",
             config=config)

query() params

def query(self, **kwargs)

  • query: Query to perform
  • query_id: Query ID to perform the query
  • dates: Dict with "from" and "to" keys for date rangue, and "timeZone" for define timezone, if you want
  • limit: Max number of rows
  • offset: start of needle for query
  • comment: comment for query pragmas
  • ip_as_string: Flag to return IP as string
  • Result of the query (dict) or Iterator object

Result returned

Client support streaming of responses

  • When the stream mode is enabled, the response is asynchronously available for the code using the Client, as the server is returning it. Therefore, there is no need to wait for the whole response to start processing it. It is also quite useful for query running online with no ending or ending expecting in the future. As the data is available we can be working with it.
  • When the stream mode is not enabled, we have to wait for the query response to be completed and returned by server.

Depending on the stream mode enabled in ClientConfig:

Response mode Stream mode enabled Stream mode not enabled
json Not supported str
json/compact Not supported str
json/simple Iterator[str] str
json/simple/compact Iterator[str] str
msgpack Not supported bytes
csv Iterator[str] str
tsv Iterator[str] str
xls Not supported bytes

Non stream call

  • Result of the query in str/bytes when query work
  • JSON Object when query has errors
  • You can use all the response formats in non-stream mode.

Stream call

  • Generator with result of the query, str, when query work
  • JSON Object when query has errors
  • Stream available response formats:
    • json/simple
    • json/simple/compact
    • csv (comma separated values)
    • tsv (tabulator separated Values)

Normal/Non stream response:

from devo.api import Client, ClientConfig

api = Client(auth={"key":"myapikey", "secret": "myapisecret"},
             address="https://apiv2-eu.devo.com/search/query", 
             config=ClientConfig(response="json", stream=False))
             
response = api.query(query="from my.app.web.activityAll select * limit 10",
                     dates= {'from': "2018-02-06 12:42:00"})
                     
print(response)

api.config.response = "json/compact"

response = api.query(query="from my.app.web.activityAll select * limit 10",
                     dates= {'from': "2018-02-06 12:42:00", 'timeZone': "GMT+2"})
                     
print(response)

Real time/stream query:

from devo.api import Client, ClientConfig

api = Client(auth={"key":"myapikey", "secret": "myapisecret"},
             address="https://apiv2-eu.devo.com/search/query", 
             config=ClientConfig(response="json/simple/compact", 
                                 pragmas={"name": "jose_ramon_id123",
                                          "app_name": "sdk-tests-master"}))
             
response = api.query(query="from my.app.web.activityAll select * limit 10",
                     dates= {'from': "2018-02-06 12:42:00"})

try:
    for item in response:
        print(item)
except Exception as error:
    print(error)

Query by id has the same parameters as query (), changing the field "query" to "query_id", which is the ID of the query in Devo.

Not verify certificates

If server has https certificates with problems, autogenerated or not verified, you can deactivate the secure calls (Verifying the certificates https when making the call) disabling it with:

from devo.api import Client, ClientConfig


api = Client(auth= {"key":"myapikey", "secret":"myapisecret"}, 
             address="https://apiv2-eu.devo.com/search/query")
api.verify_certificates(False)

You can revert it with:

api.verify_certificates(True)

Use custom CA to verify

For customs servers, and custom certificates, you can use custom CA for verity that certificates. You can put CA cert path instead of "False" or "True"

from devo.api import Client, ClientConfig


api = Client(auth= {"key":"myapikey", "secret":"myapisecret"}, 
             address="https://apiv2-eu.devo.com/search/query")
api.verify_certificates("/path/to/cafile.ca")

You can revert it with:

api.verify_certificates(True)

Processors flags

By default, you receive response in str/bytes (Depends of your python version) direct from Socket, and you need manipulate the data. But you can specify one default processor for data, soo you receive in diferents format:

from devo.api import Client, ClientConfig, JSON_SIMPLE

config = ClientConfig(response="json/simple", processor=JSON_SIMPLE, stream=True)

api = Client(auth={"key":"myapikey", "secret":"myapisecret"},
             address="https://apiv2-eu.devo.com/search/query",
             config=config)
             
response = api.query(query="from my.app.web.activityAll select * limit 10",
                     dates= {'from': "2018-02-06 12:42:00", 'timeZone': "GMT-2"})

try:
    for item in response:
        print(item)
except Exception as error:
    print(error)

Flag list

  • DEFAULT: It is the default processor, it returns str or bytes, depending on the Python version
  • TO_STR: Return str, decoding data if receive bytes
  • TO_BYTES: Return bytes, encoding data if receive str
  • JSON: Use it if you want json objects, when ask for json responses, instead of str/bytes. Ignored when response=csv
  • JSON_SIMPLE: Use it if you want json objects, when ask for json/simple response, instead of str/bytes. Ignored when response=csv
  • COMPACT_TO_ARRAY: Use it if you want arrays, when ask for json/compact responses, instead of str/bytes. Ignored when response=csv
  • SIMPLECOMPACT_TO_OBJ: Use it if you want json objects, when ask for json/simple/compact responses, instead of str/bytes. Ignored when response=csv
  • SIMPLECOMPACT_TO_ARRAY: Use it if you want json objects, when ask for json/simple/compact responses, instead of str/bytes. Ignored when response=csv

JSON_SIMPLE, COMPACT_TO_ARRAY, SIMPLECOMPACT_TO_OBJ and SIMPLECOMPACT_TO_ARRAY only work with the api in mode stream=True

You can change the processor in any moment with one default processor or custom with the function:

from devo.api import Client, ClientConfig, JSON_SIMPLE

config = ClientConfig(response="json/simple", processor=JSON_SIMPLE, stream=True)

api = Client(auth={"key":"myapikey", "secret":"myapisecret"},
             address="https://apiv2-eu.devo.com/search/query",
             config=config)
api.config.set_processor(processor)

DEFAULT example in Python 3, response csv

b'18/Jan/2019:09:58:51 +0000,/category.screen?category_id=BEDROOM&JSESSIONID=SD10SL6FF10ADFF7,404,http://www.bing.com/,Googlebot/2.1 ( http://www.googlebot.com/bot.html),gaqfse5dpcm690jdh5ho1f00o2:-'

DEFAULT example in Python 2.7, response csv

'18/Jan/2019:09:58:51 +0000,/category.screen?category_id=BEDROOM&JSESSIONID=SD10SL6FF10ADFF7,404,http://www.bing.com/,Googlebot/2.1 ( http://www.googlebot.com/bot.html),gaqfse5dpcm690jdh5ho1f00o2:-'

TO_STR example in Python 3, response csv

'18/Jan/2019:09:58:51 +0000,/category.screen?category_id=BEDROOM&JSESSIONID=SD10SL6FF10ADFF7,404,http://www.bing.com/,Googlebot/2.1 ( http://www.googlebot.com/bot.html),gaqfse5dpcm690jdh5ho1f00o2:-'

TO_BYTES example in Python 2.7, response csv

b'18/Jan/2019:09:58:51 +0000,/category.screen?category_id=BEDROOM&JSESSIONID=SD10SL6FF10ADFF7,404,http://www.bing.com/,Googlebot/2.1 ( http://www.googlebot.com/bot.html),gaqfse5dpcm690jdh5ho1f00o2:-'

SIMPLECOMPACT_TO_ARRAY example in Python 3, response json/simple/compact

['18/Jan/2019:09:58:51 +0000', '/category.screen?category_id=BEDROOM&JSESSIONID=SD10SL6FF10ADFF7', 404, 'http://www.bing.com/', 'Googlebot/2.1 ( http://www.googlebot.com/bot.html)', 'gaqfse5dpcm690jdh5ho1f00o2:-']

SIMPLECOMPACT_TO_OBJ example in Python 3, response json/simple/compact

{'statusCode': 404, 'uri': '/category.screen?category_id=BEDROOM&JSESSIONID=SD10SL6FF10ADFF7', 'referralUri': 'http://www.bing.com/', 'userAgent': 'Googlebot/2.1 ( http://www.googlebot.com/bot.html)', 'cookie': 'gaqfse5dpcm690jdh5ho1f00o2:-', 'timestamp': '18/Jan/2019:09:58:51 +0000'}

The "from" and "to", formats and other stuff

Here we define the start and end of the query (query eventdate filter are secondary), those are the limits of the query.

From                                                                          To
|-----------------|OOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOO|------------------------|
                today()  <=    eventdate      <=    now()

Date Formats

  • Fixed format: As described on Official Python Docs. Accepted formats are:

    • '%Y-%m-%d %H:%M:%S'
    • '%Y-%m-%d', the time will be truncated to 00:00:00

  • Timestamp: From epoch in seconds

  • Dynamic expression: Using the LinQ sintax we can use several functions. timeZone can be wrong with this syntax

    • Relative functions:

      • now(): Current date and time
      • today(): Current date and time fixed to 00:00:00
      • yesterday(): Current date minus one day and time fixed to 00:00:00

    • Amount functions:

      • second(): Return 1
      • minute(): Return 60
      • hour(): Return 60 * 60
      • day(): Return 24 60 60
      • week(): Return 7 24 60 * 60
      • month(): Return 30 24 60 * 60

      With this sintax you can use expressions like: from="now()-2*day() equivalent to now minus two days. Or from=today()-6*month(), etc.

  • New REST API dinamyc dates:

    For all the examples that don't use a timestamp to specify a date, we assume that the moment of execution is 08-10-2018, 14:33:12 UTC. This is a copy of official Devo docs you can see HERE

____Dates

Operator Example Description
today Get the current day at 00:00:00. Note that the timeZone parameter affects the date settings.
"from": "today" This sets the starting date to 08-10-2018, 00:00:00 UTC
"to": "today" This sets the ending date to 08-10-2018, 00:00:00 UTC
"from": "today", "timeZone": "GMT+2" This sets the starting date to 08-10-2018, 00:00:00 GMT+2 (07-10-2018, 22:00:00 UTC)
"to": "today", "timeZone": "GMT+2" This sets the ending date to 08-10-2018, 00:00:00 GMT+2 (07-10-2018, 22:00:00 UTC)
now Get the current day and time
"from": "now" This sets the starting date to 08-10-2018, 14:33:12 UTC
"to": "now" This sets the ending date to 08-10-2018, 14:33:12 UTC
endday If you use this in the from field you will get the current day and the last second of the day. If you use it in the to field you will get the from date and the last second of that day. Note that the timeZone parameter affects the date settings.
"from": "endday" This sets the starting date to 08-10-2018, 23:59:59 UTC
"from": 1515500531, "to": "endday" (this timestamp corresponds to 01/09/2018 12:22:11 UTC) This sets the ending date to 01-09-2018, 23:59:59 UTC.
"from": "endday", "timeZone": "GMT+2" This sets the ending date to 08-10-2018, 23:59:59 GMT+2 (08-10-2018, 21:59:59 UTC)
"from": 1515493331, "to": "endday", "timeZone": "GMT+2" (this timestamp corresponds to 01/09/2018, 12:22:11 GMT+2) This sets the ending date to 01-09-2018 23:59:59 GMT+2 (01-09-2018, 21:59:59 UTC)
"from": 1515452400, "to": "endday", "timeZone": "GMT+2" (this timestamp corresponds to 01/09/2018, 01:00:00 GMT+2) This sets the ending date to 01-09-2018 23:59:59 GMT+2 (01-09-2018, 21:59:59 UTC)
endmonth If you use this in the from field you will get the last day of the current month and the last second of that day. If you use it in the to field, you will get last day of the month indicated in the date field and the last second of that day. Note that the timeZone parameter affects the date settings.
"from": "endmonth" This sets the starting date to 31-10-2018, 23:59:59 UTC
"to": "endmonth" This sets the ending date to 30-09-2018, 23:59:59 UTC.
"from": 1536150131, "to": "endmonth" (this timestamp corresponds to 05/09/2018, 12:22:11 UTC) This sets the ending date to 30-09-2018, 23:59:59 UTC
"from": 1536142931, "to": "endmonth", "timeZone": "GMT+2" (this timestamp corresponds to 05/09/2018, 12:22:11 GMT+2) This sets the ending date to 30-09-2018 23:59:59 GMT+2 (30-09-2018, 21:59:59 UTC)

____Days

Operator Example Description
d Enter a number followed by d in the from parameter to substract N days from the current date. If you use it in the to field you will get the from date plus the indicated number of days.
"from": "2d" This sets the starting date to 06-10-2018, 14:33:12 UTC
"from": 1536150131, "to": "2d" This sets the ending date to 07-09-2018, 12:22:11 UTC
"from": "5d", "to": "2d" This sets the starting date to 03-10-2018, 14:33:12 UTC and the ending date to 05-10-2018, 14:33:12 UTC
ad Enter a number followed by ad in the from parameter to subtract N days from the current date and set time to 00:00:00. If you use it in the to field you will get the from date plus the indicated number of days and set time to 00:00:00. Note that the timeZone parameter affects the date settings.
"from": "2ad" This sets the starting date to 06-10-2018, 00:00:00 UTC
"from": 1536150131, "to": "2ad" (this timestamp corresponds to 05-09-2018, 12:22:11 UTC) This sets the ending date to 07-09-2018, 00:00:00 UTC
"from":"5ad", "to": "2ad" This sets the starting date to 03-10-2018, 00:00:00 UTC and the ending date to 05-10-2018, 00:00:00 UTC
"from": "5ad", "to": "2ad" This sets the starting date to 03-10-2018, 00:00:00 UTC and the ending date to 05-10-2018, 00:00:00 UTC
"from": 1536142931, "to": "2ad", "timeZone": "GMT+2" (this timestamp corresponds to 05-09-2018, 12:22:11 UTC) This sets the ending date to 07-09-2018, 00:00:00 GMT+2 (06-09-2018, 22:00:00 UTC)
"from": "5ad", "to": "2ad", "timeZone": "GMT+2" This sets the starting date to 03-10-2018, 00:00:00 GMT+2 (02-10-2018, 22:00:00 UTC), and the ending date to 05-10-2018, 00:00:00 GMT+2 (04-10-2018, 22:00:00 UTC)

____Hours

Operator Example Description
h Enter a number followed by h in the from parameter to subtract N hours from the current time. If you use it in the to field you will get the from time plus the indicated number of hours
"from": "2h" This sets the starting date to 08-10-2018, 12:33:12 UTC
"from": "16h" This sets the starting date to 07-10-2018, 22:33:12 UTC
"from": 1536150131, "to": "2h" (this timestamp corresponds to 05/09/2018, 12:22:11 UTC) This sets the ending date to 05-09-2018, 14:22:11 UTC
"from": "5h", "to": "2h" This sets the starting date to 08-10-2018, 09:33:12 UTC and the ending date to 08-10-2018, 11:33:12 UTC
ah Enter a number followed by ah in the from parameter to subtract N hours from the current date at 00:00:00. If you use it in the to field you will add the indicated number of hours to the from date at 00:00:00. Note that the timeZone parameter affects the date settings.
"from": "2ah" This sets the starting date to 07-10-2018, 22:00:00 UTC
"from": "2ah", "timeZone": "GMT+2" This sets the starting date to 07-10-2018, 22:00:00 GMT+2 (07-10-2018, 20:00:00 UTC)
"from": 1536114131, "to": "12ah" (this timestamp corresponds to 05-09-2018, 02:22:11 UTC) This sets the starting date to 07-10-2018, 22:00:00 GMT+2 (07-10-2018, 20:00:00 UTC)
"from": 1536106931, "to": "12aH", "timeZone": "GMT+2" (this timestamp corresponds to 05-09-2018, 12:22:11 GMT+2) This sets the ending date to 05-09-2018, 12:00:00 GMT+2 (05-09-2018, 10:00:00 UTC)
"from": "5ah", "to": "21ah" This sets the starting date to 07-10-2018, 19:00:00 UTC and the ending date to 07-10-2018, 21:00:00 UTC

Retries

The Client supports a retry mechanism for connectivity issues only. The retry mechanism is disabled by default (the retries parameter is set to 0by default in the Client constructor).

In order to enable it, you must set up the retries parameter with a value bigger or equals than 1. The regular action/command is not considered a retry, only the additional attempts.

There is a delay within retries. The default base delay is 5 seconds, it is updated by Exponential backoff algorithm in every retry with a rate reduction of 2. The base delay is multiplied by 2 in every retry. The base delay can be configured by parameter retry_delay

Keep Alive mechanism support

Some queries may require a big time to start returning data, because of the calculations required, the load of the platform or just because the data belongs to the future (because the data hasn't been ingested yet).

In such a cases, as the client is a common HTTP client, there is a timeout for the server for start returning data. When this deadline is over the client cancels the request, and it returns a timeout error.

In order to avoid this timeout errors, the server returns tokens every little time to reset the timeout control in the client. Client supports the processing of these tokens to not spoil the data and returning the right data.

Client detects and removes tokens from response transparently, so developer has not to deal with them.

Client support several modes for supporting this mechanism. The mode is set up in parameter keepAliveToken in ClientConfig. Its default value is devo.api.client.DEFAULT_KEEPALIVE_TOKEN

  • devo.api.client.NO_KEEPALIVE_TOKEN: No keep alive token mechanism will be used
    • json, json/compact, json/simple and json/simple/compact modes are forced to use always DEFAULT_KEEPALIVE_TOKEN mode, so it cannot be disabled for them. The token for these modes is always b' ' (four utf-8 spaces chars)
    • For csv, tsv and xls the keep alive mechanism is disabled and no token is sent by server
    • msgpack does not support the keep alive mechanism in any case and this is the only valid choice for this response mode
  • devo.api.client.DEFAULT_KEEPALIVE_TOKEN: The default token used
    • json, json/compact, json/simple and json/simple/compact token is always b' ' (four utf-8 spaces chars)
    • For csv, tsv and xls token is always \n (new line char) in this mode
    • msgpack does not support the keep alive mechanism, NO_KEEPALIVE_TOKEN is forced
  • devo.api.client.EMPTY_EVENT_KEEPALIVE_TOKEN: A new empty event or line is created by the server. The format of the event will depend on response mode, csvor tsv
    • json, json/compact, json/simple and json/simple/compact token is always b' ' (four utf-8 spaces chars)
    • For csv and tsv token contains as many separator chars (, or \t) as columns/fields the response has minus 1, followed by \n (new line char)
    • msgpack and xls do not support this mode
  • CUSTOM keep alive token: Any str may be a valid token. This str will be used by the sever as keep alive token
    • json, json/compact, json/simple and json/simple/compact token is always b' ' (four utf-8 spaces chars)
    • For csv and tsv token is the custom str set as parameter
    • msgpack and xls do not support this mode
  • Queries using destination functionality does not support keep alive. NO_KEEP_ALIVE is forced
Response mode default mode NO_KEEPALIVE_TOKEN DEFAULT_KEEPALIVE_TOKEN EMPTY_EVENT_KEEPALIVE_TOKEN Custom keep alive token
json DEFAULT_KEEPALIVE_TOKEN b' ' b' ' b' ' b' '
json/compact DEFAULT_KEEPALIVE_TOKEN b' ' b' ' b' ' b' '
json/simple DEFAULT_KEEPALIVE_TOKEN b' ' b' ' b' ' b' '
json/simple/compact DEFAULT_KEEPALIVE_TOKEN b' ' b' ' b' ' b' '
msgpack NO_KEEPALIVE_TOKEN Not supported Not supported Not supported Not supported
csv DEFAULT_KEEPALIVE_TOKEN Mechanism not used \n ,+\n 1 str token
tsv DEFAULT_KEEPALIVE_TOKEN Mechanism not used \n \t+\n 1 strtoken
xls NO_KEEPALIVE_TOKEN Not supported Not supported Not supported 2 Not supported 2

1 the csv and tsv token contains as many separator chars (, or \t) as columns/fields the response has minus 1, followed by \n (new line char)

2 xls does support EMPTY_EVENT_KEEPALIVE_TOKEN by inserting empty lines in Excel file. As the Client can not clean these lines, the mode was not supported in this implementation.

CLI USAGE

Usage: devo-api query [OPTIONS]

Perform query by query string

Options:
  -c, --config PATH       Optional JSON/Yaml File with configuration info.
  -e, --env TEXT          Use env vars for configuration
  -d, --default TEXT      Use default file for configuration
  -a, --address TEXT      Endpoint for the api.
  --user TEXT             User for the api.
  --app_name TEXT         Application name for the api.
  --comment TEXT          Comment for the queries.
  --key TEXT              Key for the api.
  --secret TEXT           Secret for the api.
  --token TEXT            Secret for the api.
  --jwt TEXT              JWT auth for the api.
  -q, --query TEXT        Query.
  --ip_as_string          Flag to return ip fields as string. If not present, 
                          the default is false and will return an integer.
  --stream / --no-stream  Flag for make streaming query or full query with
                          start and end. Default is true

  --output TEXT           File path to store query response if not want stdout
  -r, --response TEXT     The output format. Default is json/simple/compact
  --from TEXT             From date. For valid formats see API README. Default
                          if now - 1 hour

  --to TEXT               To date. For valid formats see API README
  --timeZone TEXT         Timezone info. For valid formats see API README
  --verify BOOLEAN        Verify certificates
  --debug / --no-debug    For testing purposes
  --help                  Show this message and exit.

A configuration file does not have to have all the necessary keys, you can have the common values: url, port, certificates. And then send with the call the tag, file to upload, etc.

Both things are combined at runtime, prevailing the values that are sent as arguments of the call over the configuration file

Config file key: The CLI uses the API key to query the data. You can see one examples in tests folders

json example 1:

  {
    "api": {
      "key": "MyAPIkeytoaccessdevo",
      "secret": "MyAPIsecrettoaccessdevo",
      "address": "https://apiv2-us.devo.com/search/query"
    }
  }

json example 2:

  {
    "api": {
      "auth": {
        "key": "MyAPIkeytoaccessdevo",
        "secret": "MyAPIsecrettoaccessdevo",
      },
      "address": "https://apiv2-us.devo.com/search/query"
    }
  }

yaml example 1:

  api:
    key: "MyAPIkeytoaccessdevo"
    secret: "MyAPIsecrettoaccessdevo"
    address: "https://apiv2-us.devo.com/search/query"

yaml example 2:

  api:
    auth:
      key: "MyAPIkeytoaccessdevo"
      secret: "MyAPIsecrettoaccessdevo"
    address: "https://apiv2-us.devo.com/search/query"

You can use environment variables or a global configuration file for the KEY, SECRET, URL, USER, APP_NAME and COMMENT values

How do the CLI resolve the set up of this parameters? This is the evalutaion order:

  1. -c configuration file option: use it to set API key, secret and url, or token and url in one file.
  2. The parameters in CLI call can complete values not in configuration file, but not override it
  3. Environment vars: if you set key, secrey or token in config file or params cli, this option should not be used.
  4. ~/.devo.json or ~/.devo.yaml: if you set key, secrey or token in other way, this option should not be used.

Environment vars are:

  • DEVO_API_ADDRESS
  • DEVO_API_KEY
  • DEVO_API_SECRET
  • DEVO_API_USER
  • DEVO_API_TOKEN
  • DEVO_API_JWT

Choosing Format

The default response format (response) is json, to change the default value, it can be established directly:

api.response = 'json/compact'

To change the response format (format) of a query, just change the value of the attribute response of the query call

api.config.response = config.get('response')

response = api.query(config.get('query'), 
                     dates={"from": config.get('from'), "to": config.get('to')})

Format allow the following values:

  • json
  • json/compact
  • json/simple
  • json/simple/compact
  • msgpack
  • csv (comma separated values)
  • tsv (Tab separated Values)

Response type JSON

When response is set to json, response is a Json Object with the following structure:

Field Name Type Description
success boolean True → ok
False → error
msg String Message Description in case of error
status Integer Numeric value that especify the error code.
0 - OK
1 - Invalid request
object Json Object The Query Result. The format of the content, depends on the Query data.

Example

{
 "success": True,
 "status": 0,
 "msg": "valid request",
 "object": [
   {
     "eventdate": "2016-10-24 06:35:00.000",
     "host": "aws-apiodata-euw1-52-49-216-97",
     "memory_heap_used": "3.049341704E9",
     "memory_non_heap_used": "1.21090632E8"
   },
   {
     "eventdate": "2016-10-24 06:36:00.000",
     "host": "aws-apiodata-euw1-52-49-216-97",
     "memory_heap_used": "3.04991028E9",
     "memory_non_heap_used": "1.21090632E8"
   },
   {
     "eventdate": "2016-10-24 06:37:00.000",
     "host": "aws-apiodata-euw1-52-49-216-97",
     "memory_heap_used": "3.050472496E9",
     "memory_non_heap_used": "1.21090632E8"
   }
 ]
}

Response type json/compact

When response is set to json/compact, response is a Json Object with the following structure:

Field Name Type Description
success boolean True → ok
False → error
msg String Message Description in case of error
status Integer Numeric value that especify the error code.
0 - OK
1 - Invalid request
object Json Object
object.m Json Object Json Object with Metadata information, the key is the name of the field, and the value is an Object with the following information:.
  • type: type of the value returned:
    • timestamp: epoch value in seconds
    • str: string
    • int8: 8 byte integer
    • int4: 4 byte integer
    • bool: boolean
    • float8: 8 byte floating point.
  • index: integer value, that points to where in the array of values is the value of this field
object.d Json Object Array of Arrays with the values of the response of the query.

Example

{
    "msg": "",
    "status": 0,
    "object": {
        "m": {
            "eventdate": {
                "type": "timestamp",
                "index": 0
            },
            "domain": {
                "type": "str",
                "index": 1
            },
            "userEmail": {
                "type": "str",
                "index": 2
            },
            "country": {
                "type": "str",
                "index": 3
            },
            "count": {
                "type": "int8",
                "index": 4
            }
        },
"d": [
        [
            1506442210000,
            "self",
            "[email protected]",
            null,
            2
        ],
        [
            1506442220000,
            "self",
            "[email protected]",
            null,
            2
        ],
    .....
    ]
}

Response type json/simple

When response is set to json/simple
The response is a stream of Json Objects with the following structure of the values that the Query generates, separated each registry CRLF.

When the query does not generate more information, the connection is closed by the server.

In case, no date to is requested, the connections is keeped alive.

Example

{"eventdate":1488369600000,"domain":"none","userEmail":"","country":null,"count":3}
{"eventdate":1488369600000,"domain":"[email protected]","userEmail":"0:0:0:0:0:0:0:1","country":null,"count":18}
{"eventdate":1488369600000,"domain":"none","userEmail":"[email protected]","country":null,"count":7}
{"eventdate":1488373200000,"domain":"[email protected]","userEmail":"127.0.0.1","country":null,"count":10}
{"eventdate":1488373200000,"domain":"[email protected]","userEmail":"0:0:0:0:0:0:0:1","country":null,"count":28}
{"eventdate":1488373200000,"domain":"self","userEmail":"[email protected]","country":null,"count":15}
{"eventdate":1488373200000,"domain":"self","userEmail":"[email protected]","country":null,"count":49}
{"eventdate":1488376800000,"domain":"[email protected]","userEmail":"127.0.0.1","country":null,"count":10}
{"eventdate":1488376800000,"domain":"self","userEmail":"[email protected]","country":null,"count":16}
{"eventdate":1488376800000,"domain":"[email protected]","userEmail":"0:0:0:0:0:0:0:1","country":null,"count":5}
{"eventdate":1488376800000,"domain":"self","userEmail":"[email protected]","country":null,"count":35}
{"eventdate":1488376800000,"domain":"self","userEmail":"[email protected]","country":null,"count":128}
{"eventdate":1488376800000,"domain":"self","userEmail":"[email protected]","country":null,"count":7}
{"eventdate":1488376800000,"domain":"self","userEmail":"[email protected]","country":null,"count":3}
{"eventdate":1488380400000,"domain":"[email protected]","userEmail":"127.0.0.1","country":null,"count":21}
{"eventdate":1488380400000,"domain":"self","userEmail":"[email protected]","country":null,"count":71}
{"eventdate":1488380400000,"domain":"[email protected]","userEmail":"0:0:0:0:0:0:0:1","country":null,"count":38}
{"eventdate":1488380400000,"domain":"self","userEmail":"[email protected]","country":null,"count":1}
...

Response type json/simple/compact

When response is set to json/simple/compact
The response is a stream of Json Objects with the following structure each line is separated by CRLF:

The First Line is a JSON object map with the Metadata information, the key is the name of the field, and the value is a Object with the following information:

{
  "m": {
    "eventdate": {
      "type": "timestamp",
      "index": 0
    },
    "domain": {
      "type": "str",
      "index": 1
    },
    "userEmail": {
      "type": "str",
      "index": 2
    },
    "country": {
      "type": "str",
      "index": 3
    },
    "count": {
      "type": "int8",
      "index": 4
    }
  }
}
  • type: type of the value returned:
    • timestamp: epoch value in seconds
    • str: string
    • int8: 8 byte integer
    • int4: 4 byt integer
    • bool: boolean
    • float8: 8 byte floating point.
  • index: integer value, that points to where in the array of values is the value of this field

The rest of the lines are data lines:

{
  "d": [
    1506439800000,
    "self",
    "[email protected]",
    null,
    1
  ]
}

a field with name "d", gives access to the array of values with the information.

When the query does not generate more information, the connection is closed by the server.
In case, no date to is requested, the connections is keeped alive.

Example

{"m":{"eventdate":{"type":"timestamp","index":0},"domain":{"type":"str","index":1},"userEmail":{"type":"str","index":2},"country":{"type":"str","index":3},"count":{"type":"int8","index":4}}}
{"d":[1506439800000,"self","[email protected]",null,1]}
{"d":[1506439890000,"self","[email protected]",null,1]}
{"d":[1506439940000,"self","[email protected]",null,1]}
{"d":[1506439950000,"self","[email protected]",null,1]}
{"d":[1506440130000,"self","[email protected]",null,1]}
{"d":[1506440130000,"self","[email protected]",null,2]}
{"d":[1506440170000,"self","[email protected]",null,3]}
{"d":[1506440200000,"self","[email protected]",null,3]}
{"d":[1506440220000,"self","[email protected]",null,1]}
{"d":[1506440230000,"self","[email protected]",null,1]}
{"d":[1506440260000,"self","[email protected]",null,3]}
{"d":[1506440270000,"self","[email protected]",null,1]}
{"d":[1506440280000,"self","[email protected]",null,1]}
{"d":[1506440280000,"self","[email protected]",null,1]}
{"d":[1506440290000,"self","[email protected]",null,3]}
{"d":[1506440350000,"self","[email protected]",null,1]}

Response type msgpack

When response is set to msgpack
The response format is the same that the Json format, but encoded using MsgPack an efficient binary serialization format (http://msgpack.org/)

Response type csv

When response is set to csv
The system returns the information in CSV(Comma Separated Values) format, as follows

Example:

    eventdate,domain,userEmail,country,count
    2017-03-01 12:00:00.000,none,,,3
    2017-03-01 12:00:00.000,[email protected],0:0:0:0:0:0:0:1,,18
    2017-03-01 12:00:00.000,none,[email protected],,7
    2017-03-01 13:00:00.000,[email protected],127.0.0.1,,10
    2017-03-01 13:00:00.000,[email protected],0:0:0:0:0:0:0:1,,28
    2017-03-01 13:00:00.000,self,[email protected],,15
    2017-03-01 13:00:00.000,nombre,[email protected],,49
    2017-03-01 14:00:00.000,[email protected],127.0.0.1,,10
    2017-03-01 14:00:00.000,self,[email protected],,16
    2017-03-01 14:00:00.000,[email protected],0:0:0:0:0:0:0:1,,5
    2017-03-01 14:00:00.000,self,[email protected],,35
    2017-03-01 14:00:00.000,nombre,[email protected],,128
    2017-03-01 14:00:00.000,self,[email protected],,7
    2017-03-01 14:00:00.000,self,[email protected],,3
    2017-03-01 15:00:00.000,[email protected],127.0.0.1,,21
    2017-03-01 15:00:00.000,self,[email protected],,71
    2017-03-01 15:00:00.000,[email protected],0:0:0:0:0:0:0:1,,38
    2017-03-01 15:00:00.000,self,[email protected],,1
    2017-03-01 15:00:00.000,nombre,[email protected],,80

Response type tsv

When response is set to tsv
The system returns the information in TSV (Tab Separated Values) format as follows

Example

    eventdate   domain  userEmail   country count
    2017-03-01 12:00:00.000 none            3
    2017-03-01 12:00:00.000 [email protected]    0:0:0:0:0:0:0:1     18
    2017-03-01 12:00:00.000 none    [email protected]        7
    2017-03-01 13:00:00.000 [email protected]    127.0.0.1       10
    2017-03-01 13:00:00.000 [email protected]    0:0:0:0:0:0:0:1     28
    2017-03-01 13:00:00.000 self    [email protected]     15
    2017-03-01 13:00:00.000 nombre [email protected]       49
    2017-03-01 14:00:00.000 [email protected]    127.0.0.1       10
    2017-03-01 14:00:00.000 self    [email protected]        16
    2017-03-01 14:00:00.000 [email protected]    0:0:0:0:0:0:0:1     5
    2017-03-01 14:00:00.000 self    [email protected]     35
    2017-03-01 14:00:00.000 nombre [email protected]       128
    2017-03-01 14:00:00.000 self    [email protected]       7
    2017-03-01 14:00:00.000 self    [email protected]     3
    2017-03-01 15:00:00.000 [email protected]    127.0.0.1       21
    2017-03-01 15:00:00.000 self    [email protected]        71
    2017-03-01 15:00:00.000 [email protected]    0:0:0:0:0:0:0:1     38

Exceptions and Errors

Every issue or error found during the request of queries of data is reported through DevoClientException exception. But since version 5.1.0 there are additional exception classes for a more detailed feedback of errors:

  • devo.api.client.DevoClientException: Common legacy Exception class that is thrown for every error during the querying of data. It inherits from Exception class
  • devo.api.client.DevoClientRequestException: Specific exception class that is thrown whenever a query is requested to the server endpoint and it fails due to internal errors or bad requests. It contains the whole requests.models.Response from server for reference. It inherits from DevoClientException
  • devo.api.client.DevoClientDataResponseException: Specific exception class that is thrown whenever a query (of type stream) is run correctly but an error is thrown when processing one specific event of the response. It is only thrown for streamed queries. It inherits from DevoClientException