Arcaflow OpenSearch

A plugin for loading data into an OpenSearch-compatible instance.

Development

During the development of this plugin it is useful to start a local Elasticsearch via:

docker compose -f docker-compose-dev.yaml up -d

and stop it again via:

docker compose -f docker-compose-dev.yaml down -v

Testing

The tests of this plugin are split up into unit and integration tests located in

• ./tests/integration/
• ./tests/unit/

Unit Tests

Run all unit tests via:

# Run all unit tests
python -m unittest tests.unit.test_opensearch_plugin

Integration Tests

Running all integration tests can be run either

• using a running a local Elasticsearch as described in Development and then execute the tests via

# Run all integration tests
python -m unittest tests.integration.test_opensearch_plugin

• using the docker-compose.yaml and run

# the --abort-on-container-exit ensures a `docker compose down` after the tests have run
docker compose -f docker-compose.yaml up --abort-on-container-exit

Note: Make sure to docker compose down and remove the volume after one run as there is currently no cleanup done.

Autogenerated Input/Output Documentation by Arcaflow-Docsgen Below

OpenSearch (opensearch)

Load data into opensearch compatible instance

Input

Type:	scope
Root object:	DocumentRequest
Properties	bulk_upload_list (list[reference[BulkUploadObject]]) Name: bulk upload list Description: The list of objects for the bulk upload operation. Required: Yes Type: list[reference[BulkUploadObject]] List items Type: reference[BulkUploadObject] Referenced object: BulkUploadObject default_index (string) Name: index Description: Name of the default index that will receive the data. Required: Yes Type: string Minimum length: 1 metadata (map[string,any]) Name: metadata Description: Optional global metadata object that will be added to every document. Required: No Type: map[string,any] Key type Type: string Value type Type: any password (string) Name: password Description: The password for the given user. Required: No Type: string tls_verify (bool) Name: TLS verify Description: For development and testing purposes, this can be set to False to disable TLS verification for connections to Opensearch-compatible services. Required: No Default (JSON encoded): true Type: bool url (string) Name: url Description: The URL for the Opensearch-compatible instance. Required: Yes Type: string username (string) Name: username Description: A username for an authorized user for the given Opensearch-compatible instance. Required: No Type: string Minimum length: 1
Objects BulkUploadObject (object) Type: object Properties data (map[string,any]) Name: data Description: The JSON data document to upload to your index. Required: Yes Type: map[string,any] Key type Type: string Value type Type: any operation (map[string,reference[OperationMeta]]) Name: operation Description: The operation type and associated operation metadata. Required: Yes Type: map[string,reference[OperationMeta]] Key type Type: string Value type Type: reference[OperationMeta] Referenced object: OperationMeta DocumentRequest (object) Type: object Properties bulk_upload_list (list[reference[BulkUploadObject]]) Name: bulk upload list Description: The list of objects for the bulk upload operation. Required: Yes Type: list[reference[BulkUploadObject]] List items Type: reference[BulkUploadObject] Referenced object: BulkUploadObject default_index (string) Name: index Description: Name of the default index that will receive the data. Required: Yes Type: string Minimum length: 1 metadata (map[string,any]) Name: metadata Description: Optional global metadata object that will be added to every document. Required: No Type: map[string,any] Key type Type: string Value type Type: any password (string) Name: password Description: The password for the given user. Required: No Type: string tls_verify (bool) Name: TLS verify Description: For development and testing purposes, this can be set to False to disable TLS verification for connections to Opensearch-compatible services. Required: No Default (JSON encoded): true Type: bool url (string) Name: url Description: The URL for the Opensearch-compatible instance. Required: Yes Type: string username (string) Name: username Description: A username for an authorized user for the given Opensearch-compatible instance. Required: No Type: string Minimum length: 1 OperationMeta (object) Type: object Properties _id (string) Name: ID Description: Optional ID for the data document. Required: No Type: string _index (string) Name: index Description: Optional name of the index that will receive the data. If none is provided, the default index will be used. Required: No Type: string

Outputs

error

Type:	scope
Root object:	ErrorOutput
Properties	error (string) Required: Yes Type: string
Objects ErrorOutput (object) Type: object Properties error (string) Required: Yes Type: string

success

Type:	scope
Root object:	SuccessOutput
Properties	document_ids (list[string]) Required: Yes Type: list[string] List items Type: string message (string) Required: Yes Type: string
Objects SuccessOutput (object) Type: object Properties document_ids (list[string]) Required: Yes Type: list[string] List items Type: string message (string) Required: Yes Type: string

Process List (process_list)

Process list input into a bulk_upload_list

Input

Type:	scope
Root object:	DataList
Properties	data_list (list[any]) Name: data list Description: List of data object to process into the bulk_upload_list. Required: Yes Type: list[any] List items Type: any operation (map[string,reference[OperationMeta]]) Name: operation Description: The operation type and associated operation metadata. Required: Yes Type: map[string,reference[OperationMeta]] Key type Type: string Value type Type: reference[OperationMeta] Referenced object: OperationMeta
Objects DataList (object) Type: object Properties data_list (list[any]) Name: data list Description: List of data object to process into the bulk_upload_list. Required: Yes Type: list[any] List items Type: any operation (map[string,reference[OperationMeta]]) Name: operation Description: The operation type and associated operation metadata. Required: Yes Type: map[string,reference[OperationMeta]] Key type Type: string Value type Type: reference[OperationMeta] Referenced object: OperationMeta OperationMeta (object) Type: object Properties _id (string) Name: ID Description: Optional ID for the data document. Required: No Type: string _index (string) Name: index Description: Optional name of the index that will receive the data. If none is provided, the default index will be used. Required: No Type: string

Outputs

error

Type:	scope
Root object:	ErrorOutput
Properties	error (string) Required: Yes Type: string
Objects ErrorOutput (object) Type: object Properties error (string) Required: Yes Type: string

success

Type:	scope
Root object:	BulkUploadList
Properties	bulk_upload_list (list[reference[BulkUploadObject]]) Name: bulk upload list Description: The list of objects for the bulk upload operation. Required: Yes Type: list[reference[BulkUploadObject]] List items Type: reference[BulkUploadObject] Referenced object: BulkUploadObject
Objects BulkUploadList (object) Type: object Properties bulk_upload_list (list[reference[BulkUploadObject]]) Name: bulk upload list Description: The list of objects for the bulk upload operation. Required: Yes Type: list[reference[BulkUploadObject]] List items Type: reference[BulkUploadObject] Referenced object: BulkUploadObject BulkUploadObject (object) Type: object Properties data (map[string,any]) Name: data Description: The JSON data document to upload to your index. Required: Yes Type: map[string,any] Key type Type: string Value type Type: any operation (map[string,reference[OperationMeta]]) Name: operation Description: The operation type and associated operation metadata. Required: Yes Type: map[string,reference[OperationMeta]] Key type Type: string Value type Type: reference[OperationMeta] Referenced object: OperationMeta OperationMeta (object) Type: object Properties _id (string) Name: ID Description: Optional ID for the data document. Required: No Type: string _index (string) Name: index Description: Optional name of the index that will receive the data. If none is provided, the default index will be used. Required: No Type: string