This library allows you to send lookups to the Devo platform.
You have two types of uses, in script (This doc) or send using shell
You need to know how work Sender and Data to use Lookup functions in script, but not for shell
Just like the send events case, to create a new lookup or send data to existent lookup table we need to initialize the collector configuration.
In case to initialize the collector configuration from a json/yaml file, you must include a new object into the lookup variable with the new parameters or add in the CLI flags:
Example:
{
sender": {
...
},
"lookup": {
"name": "Test_Lookup_of_180306_02",
"file": "test_lookup.csv",
"lkey": "KEY",
"escape_quotes": true
}
}
You can see more examples below
After initializing the lookup, you can send data to the lookup. There are two ways to do this.
The first option is to generate a string with the headers structure and then send a control instruction to indicate the start or the end of operation over the lookup. Between those control instructions must be the operations over every row of the lookup.
The header structure is an object list with values and data types of the lookup data.
Example:
[{"KEY":{"type":"str","key":true}},{"HEX":{"type":"str"}},{"COLOR":{"type":"str"}}]
To facilitate the creation of this string we can call list_to_headers method of Lookup class.
Params
- headers (list required): column names list of the lookup
- type_of_key (string optional): specify a concrete type for the key (Default string)
- key (string optional): name of key
- key_index (string optional): index of key. You can use this or key
- types (list optional'): list of type: with types of columns in order
Accepted types:
- String -> "str"
- Integer -> "int" (default int type: int8)
- Integer4 -> "int4"
- Integer8 -> "int8"
- Float -> "float"
- Boolean -> "bool"
- IP -> "ip4"
Example:
from devo.sender import Lookup
pHeaders = Lookup.list_to_headers(
headers=list, #List with all headers names
type_of_key="str",
key=None, #Name of the Key
key_index=None, #Or index of key in headers list, you can use whatever you want
types=list
)With this string we can call send_control method of Sender class to send the control instruction.
Params
- event (string required 'START'|'END'): header type
- START: start of header
- END: end of header
- headers (string required): header structure
- action (string required 'FULL'|'INC'): action type
- FULL: delete previous lookup data and then add the new
- INC: add new row to lookup table
Example:
lookup.send_control(event='START', headers=["id", "name", "email"], action='INC')The other option is basically the same operations but with less instructions: You can use send_headers method of LtLookup class we can unify list_to_headers + send_control.
send_headers Params
- headers (list default: [] ): column name list of lookup
- key (string default 'KEY'): column name of key
- key_index (string default 'KEY'): column name of key
- event (string default: 'START'): header event
- START: start of header
- END: end of header
- headers (string required): header structure
- action (string required 'FULL'|'INC'): action type
- FULL: delete previous lookup data and then add the new
- INC: add new row to lookup table
Example:
lookup.send_headers(headers=['KEY', 'HEX', 'COLOR'], key='KEY', event='START', action='FULL')Finally, to send a new row we can use send_data_line method from LtLooup class.
Params
- key_index (string optional default: None): or key index in list fields
- fields (list default: []): values list: can be str, int, float and bool
- delete (boolean default: False): row must be deleted
Example:
lookup.send_data_line(key_index=0, fields=["11", "HEX11", "COLOR11" ])A complete example to send a lookup row is:
from devo.common import Configuration
from devo.sender import Sender, Lookup
conf = Configuration(path="./config.json.example")
con = Sender(config=conf.get("sender"))
lookup = Lookup(name=conf.get('lookup').get('name', "default"), historic_tag=None, con=con)
pHeaders = Lookup.list_to_headers(headers=['KEY','HEX', 'COLOR'], key='KEY')
lookup.send_control(event='START', headers=pHeaders, action='INC')
lookup.send_data_line(key_index=0, fields=["11", "HEX11", "COLOR11" ])
lookup.send_data_line(key_index=0, fields=[22, "HEX22", "COLOR22"])
lookup.send_control(event='END', headers=pHeaders, action='INC')
con.socket.shutdown(0)A simplify complete example to send a row of lookup is:
from devo.common import Configuration
from devo.sender import Sender, Lookup
conf = Configuration()
conf.load_config("./config.json.example", 'sender')
conf.load_config("./config.json.example", 'lookup')
con = Sender(config=conf.get("sender"))
lookup = Lookup(name=conf.get('name', "default"), historic_tag=None, con=con)
lookup.send_headers(headers=['KEY', 'HEX', 'COLOR'], key='KEY', event='START', action="INC")
lookup.send_data_line(key_index=0, fields=["11", "HEX12", "COLOR12"], delete=True)
lookup.send_headers(headers=['KEY', 'HEX', 'COLOR'], key='KEY', event='END', action="INC")
con.socket.shutdown(0)NOTE:
- The start and end control instructions should have the list of the names of the columns in the same order in which the lookup was created.
- Keep in mind that the sockets must be closed at the end
- The lookups are sent to Devo and a process running in the background loads and distributes them. It can take a few minutes to have the lookup available.
After initializing the lookup, you can upload a CSV file with the lookup data by send_csv method from LtLookup class.
Params
- path (string required): CSV file path
- has_header (boolean default: True): CSV has header
- delimiter (string default: ','): CSV delimiter
- quotechar (string default: '"'): CSV quote char
- headers (list default: []): header array
- key (string default: 'KEY'): lookup key
- historic_tag (string default: None): tag
- action (string default: None): FULL (Delete old if exist) or INC (Update if exist), action for the lookup
- action_field (string default: None): field name (Name in header) with the field of action for the row (add or delete)
- types (list default: None): List with type of fields
- detect_types (string default: False): Detect types of fields reading first line
Example
lookup.send_csv(config['file'], headers=['KEY', 'COLOR', 'HEX'], key=config['lkey'])Complete example
from devo.common import Configuration
from devo.sender import Sender, Lookup
conf = Configuration("./config.json.example")
con = Sender(config=conf.get("sender"))
lookup = Lookup(name=conf.get('lookup').get('name', "default"), historic_tag=None, con=con)
lookup.send_csv(path=conf.get('lookup').get('file', "example.csv"),
has_header=True, key=conf.get('lkey', "ID"))
con.socket.shutdown(0)You can use config file to send types list:
{
'lookup' : {
'types': ["int", "str", "int4", "bool", "int8"]
....
}
}
Any double quotes inside the field value can cause trouble when sending the lookup if this is not escaped. An example of this case can be seen below:
lookup.send_data_line(key_index=0, fields=["11", 'double quotes must escaped"'])That lookup creation will fail because that double quote will be interpreted as a field termination and the number of fields for that row will unmatch the corresponding number of columns. To avoid this, add "escape_quotes": true to the lookup configuration file and escape_quotes=True to the Lookup constructor. Below, an example for the constructor is shown:
lookup = Lookup(name=self.lookup_name, historic_tag=None, con=con, escape_quotes=True)To see an example for the lookup configuration please refer to the one shown at the start of the document.
This will escape ALL double quotes in the field value by adding a second double quote to it.
The default behavior is to NOT escape any double quotes.