Datarator

Stateless data generator with HTTP based JSON API.

Running

Run dockerized

• Using official immage:

    docker run -p 9292:9292 --rm --name datarator datarator/datarator:edge
  

• or building docker image locally:

    docker build -t datarator:edge . && \
    docker run -p 9292:9292 --rm --name datarator datarator:edge
  

Build & Run

1. make sure to have ruby + bundler installed for Ubuntu run:

    sudo apt-get install ruby ruby-dev
    gem install bundler --user-install
   

2. build from source:

    git clone https://github.com/datarator/datarator.git && \
    pushd datarator && \
    bundle install && \
    gem build datarator.gemspec && \
    gem install datarator-0.0.1.gem && \
    popd
   

3. and run:

    datarator
   

4. console should contain:

    Datarator starting ...
   

Use

curl -H "content-type: application/json" -X POST -d '{"template":"csv","document":"foo_document","count":"1","columns":[{"name":"greeting","type":"const", "options":{"value":"hello"}},{"name":"from","type":"const","options":{"value":"from"}},{"name":"server","type":"const","options":{"value":"datarator"}}],"options":{"header":"false", "separator": " "}}' http://127.0.0.1:9292/api/schemas

JSON API

##JSON syntax:

{
   "template": "<template_name>",
   "document": "<document_name>",
   "count": "<count>",
   "locale": "<locale>",
   "columns": [ < column > , < column > , ...],
   "options": < options >
}

Legend:

• <template_name> - name of the output template (see Output templates)
• <document_name> - name of the document
• <count> - generated rows count
• <locale> - locale (ignored currently)
• <column> - column to generate (see Column)
• <options> - options for generation (see Options)

##Output templates

Following output templates are available:

• csv
• sql
• liquibase xml
• liquibase yaml
• liquibase json

###csv

Enabled via: "template":"csv".

Optional options available:

• "header":"true" / "header":"false" - whether names of the colums should included (as the 1.st row) or not. By default is false.
• "empty_value":"<empty value>"- empty value. By default is empty string.
• "separator":"<separator>" - the separator string to be used for joining values.

For example, input JSON:

{
	"template": "csv",
	"document": "foo_document",
	"count": "3",
	"columns": [{
		"name": "name1",
		"type": "const",
		"options": {
			"value": "value1"
		}
	}, {
		"name": "name2",
		"type": "const",
		"options": {
			"value": "value2"
		}
	}, {
		"name": "name3",
		"type": "const",
		"emptyPercent": "100",
		"options": {
			"value": "value2"
		}
	}],
	"options": {
		"header": "true",
		"empty_value": "NULL"
	}
}

results in:

	name1,name2,name3
	value1,value2,NULL
	value1,value2,NULL
	value1,value2,NULL

###sql

Enabled via: "template":"sql".

For example, input JSON:

{
	"template": "sql",
	"document": "foo_document",
	"count": "3",
	"columns": [{
		"name": "name1",
		"type": "const",
		"options": {
			"value": "value1"
		}
	}, {
		"name": "name2",
		"type": "const",
		"options": {
			"value": "value2"
		}
	}]
}

results in:

    	INSERT INTO (name1,name2) values ('value1','value2');
    	INSERT INTO (name1,name2) values ('value1','value2');
    	INSERT INTO (name1,name2) values ('value1','value2');

###liquibase xml

Enabled via: "template":"liquibase.xml".

Optional options available:

• "changeset":"true" / "changeset":"false" - whether changeset section should be included or not. By default is false.

For example, input JSON:

{
	"template": "liquibase.xml",
	"document": "foo_document",
	"count": "3",
	"columns": [{
		"name": "name1",
		"type": "const",
		"options": {
			"value": "value1"
		}
	}, {
		"name": "name2",
		"type": "const",
		"options": {
			"value": "value2"
		}
	}],
	"options": {
		"changeset": "true"
	}
}

results in:

	<databaseChangeLog
	xmlns="http://www.liquibase.org/xml/ns/dbchangelog"
	  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	  xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-3.0.xsd">

	  <changeset author="datarator.io" id="foo_document-1418423861">
	    <insert tableName="foo_document">
	      <column name="name1" value="value1"/>
	      <column name="name2" value="value2"/>
	    </insert>
	    <insert tableName="foo_document">
	      <column name="name1" value="value1"/>
	      <column name="name2" value="value2"/>
	    </insert>
	    <insert tableName="foo_document">
	      <column name="name1" value="value1"/>
	      <column name="name2" value="value2"/>
	    </insert>
	  </changeset>

###liquibase yaml

Enabled via: "template":"liquibase.yaml".

Optional options available:

• "changeset":"true" / "changeset":"false" - whether changeset section should be included or not. By default is false.

For example, input JSON:

{
	"template": "liquibase.yaml",
	"document": "foo_document",
	"count": "3",
	"columns": [{
		"name": "name1",
		"type": "const",
		"options": {
			"value": "value1"
		}
	}, {
		"name": "name2",
		"type": "const",
		"options": {
			"value": "value2"
		}
	}],
	"options": {
		"changeset": "true"
	}
}

results in:

	databaseChangeLog:
	  - changeset
	      author: datarator.io
	      id: foo_document-1418426125
	      changes:
		- insert:
		    tableName: foo_document
		    columns:
		      - column:
			  name: name1
			  value: value1
		      - column:
			  name: name2
			  value: value2
		- insert:
		    tableName: foo_document
		    columns:
		      - column:
			  name: name1
			  value: value1
		      - column:
			  name: name2
			  value: value2
		- insert:
		    tableName: foo_document
		    columns:
		      - column:
			  name: name1
			  value: value1
		      - column:
			  name: name2
			  value: value2

####liquibase json

Enabled via: "template":"liquibase.json".

Optional options available:

• "changeset":"true" / "changeset":"false" - whether changeset section should be included or not. By default is false.

For example, input JSON:

{
	"template": "liquibase.json",
	"document": "foo_document",
	"count": "3",
	"columns": [{
		"name": "name1",
		"type": "const",
		"options": {
			"value": "value1"
		}
	}, {
		"name": "name2",
		"type": "const",
		"options": {
			"value": "value2"
		}
	}],
	"options": {
		"changeset": "true"
	}
}

results in:

    databaseChangeLog: [
        {
           "changeset"
                "author": "datarator.io"
                "id": "foo_document-1418426254"
                "changes": [
                    {
		    	"insert": {
                            "tableName": "foo_document",
                            "columns" [
                                {
                                    "column": {
                                        "name": "name1",
                                        "value": "value1"
                                        },
                                    "column": {
                                        "name": "name2",
                                        "value": "value2"
                                        }
                                    },
                            ]
                        }
                        "insert": {
                            "tableName": "foo_document",
                            "columns" [
                                {
                                    "column": {
                                        "name": "name1",
                                        "value": "value1"
                                        },
                                    "column": {
                                        "name": "name2",
                                        "value": "value2"
                                        }
                                    },
                            ]
                        }
                        "insert": {
                            "tableName": "foo_document",
                            "columns" [
                                {
                                    "column": {
                                        "name": "name1",
                                        "value": "value1"
                                        },
                                    "column": {
                                        "name": "name2",
                                        "value": "value2"
                                        }
                                    },
                            ]
                        }
                    }
                ]
            }
        ]

##Options

Holds template (if present in root node) or column specific options (if present in column node).

Options syntax:

	{"<name>":"<value>"}}

Legend:

• <name> - name of the option
• <value> - value of the option

##Column

Column syntax:

{
	"name": "<name>",
	"type": "<type>",
	"emptyPercent": "<emptyPercent>",
	"options": < options >
}

Legend:

• <name> - name of the option
• <value> - value of the option
• <emptyPercent> - (optional) empty values percent
• <options> - (optional) column options (see Options)

Following column types are available:

• without column nesting supported:
  • const
  • row_index
  • copy
  • name:
    • name.name
    • name.first_name
    • name.prefix
    • name.suffix
    • name.title
  • bitcoin.address
  • book:
    • book.name
    • book.publisher
    • book.genre
  • boolean
  • code:
    • code.ean
    • code.isbn
  • color:
    • color.hex
    • color.name
  • credit card:
    • credit_card.number
    • credit_card.type
  • number:
    • number.binary
    • number.decimal
    • number.hexadecimal
    • number.octal
  • regexp
• with column nesting supported:
  • list.seq
  • list.rand
  • join

###const

Generates constant value provided in options.

Mandatory options available:

• "value":"<value>" - the constant value to generate.

For example, input JSON:

"columns": [{
	"name": "name1",
	"type": "const",
	"options": {
		"value": "value1"
	}
}]

results in value:

	value1

###row_index

Generates row index of the currently generated row.

For example, input JSON:

"columns": [{
	"name": "name1",
	"type": "row_index"
}]

results in value:

	0
	1
	2
	3
	...

###copy

Generates the same value as the column referred.

Mandatory options available:

• "from":"<column_name>" - the column name whose value is to be copied.

For example, input JSON:

"columns": [{
	"name": "name1",
	"type": "const",
	"options": {
		"value": "value1"
	}
}, {
	"name": "name2",
	"type": "copy",
	"options": {
		"from": "name1"
	}
}]

results (for columns: name1 as well as name2) in value:

	value1

###name.name

Generates the random name value.

For example, input JSON:

"columns": [{
	"name": "name1",
	"type": "name.name"
}]

could result in value:

	Christophe Bartell

###name.first_name

Generates the random first name value.

For example, input JSON:

"columns": [{
	"name": "name1",
	"type": "name.first_name"
}]

could result in value:

	Christophe

###name.last_name

Generates the random last name value.

For example, input JSON:

"columns": [{
	"name": "name1",
	"type": "name.last_name"
}]

could result in value:

	Bartell

###name.prefix

Generates the random name "prefix" value.

For example, input JSON:

"columns": [{
	"name": "name1",
	"type": "name.prefix"
}]

could result in value:

	Mr.

###name.suffix

Generates the random name "suffix" value.

For example, input JSON:

"columns": [{
	"name": "name1",
	"type": "name.suffix"
}]

could result in value:

	IV

###name.title

Generates the random name "title" value.

For example, input JSON:

"columns": [{
	"name": "name1",
	"type": "name.title"
}]

could result in value:

	Legacy Creative Director

###bitcoin.address

Generates the random bitcoin address value.

For example, input JSON:

"columns": [{
	"name": "foo",
	"type": "bitcoin.address"
}]

could result in value:

	1HUoGjmgChmnxxYhz87YytV4gVjfPaExmh

###book.name

Generates the random book name value.

For example, input JSON:

"columns": [{
	"name": "foo",
	"type": "book.name"
}]

could result in value:

	The Odd Sister

###book.publisher

Generates the random book publisher value.

For example, input JSON:

"columns": [{
	"name": "foo",
	"type": "book.publisher"
}]

could result in value:

	Opus Reader

###book.genre

Generates the random book genre value.

For example, input JSON:

"columns": [{
	"name": "foo",
	"type": "book.genre"
}]

could result in value:

	Mystery

###boolean

Generates the random boolean.

Optional options available:

• "true_ratio":"<true_ratio>" - the true ratio (in range: <0,1>)

For example, input JSON:

"columns": [{
	"name": "foo",
	"type": "boolean",
	"options": {
		"true_ratio": 0.5
	}
}]

could result in value:

	true

###code.ean

Generates the random ean code value.

For example, input JSON:

"columns": [{
	"name": "foo",
	"type": "code.ean"
}]

could result in value:

	4600051000057

###code.isbn

Generates the random isbn code value.

For example, input JSON:

"columns": [{
	"name": "foo",
	"type": "code.isbn"
}]

could result in value:

	759021701-8

###color.hex

Generates the random hex color value.

For example, input JSON:

"columns": [{
	"name": "foo",
	"type": "color.hex"
}]

could result in value:

	#31a785

###color.name

Generates the random color name value.

For example, input JSON:

"columns": [{
	"name": "foo",
	"type": "color.name"
}]

could result in value:

	red

###credit_card.number

Generates the random credit card number value.

For example, input JSON:

"columns": [{
	"name": "foo",
	"type": "credit_card.number"
}]

could result in value:

	1212-1221-1121-1234

###credit_card.type

Generates the random credit card type value.

For example, input JSON:

"columns": [{
	"name": "foo",
	"type": "credit_card.type"
}]

could result in value:

	visa

###number.binary

Generates the random binary number.

Optional options available:

• "max":"<max>" - the maxiumum value (by default is 10).
• "min":"<min>" - the minimum value (by default is 0).

For example, input JSON:

"columns": [{
	"name": "foo",
	"type": "number.binary",
	"options": {
		"min": 1,
		"max": 10
	}
}]

could result in value:

	10

###number.decimal

Generates the random decimal number.

Optional options available:

• "max":"<max>" - the maxiumum value (by default is 10).
• "min":"<min>" - the minimum value (by default is 0).

For example, input JSON:

"columns": [{
	"name": "foo",
	"type": "number.decimal",
	"options": {
		"min": 1,
		"max": 10
	}
}]

could result in value:

	2

###number.hexadecimal

Generates the random hexadecimal number.

Optional options available:

• "max":"<max>" - the maxiumum value (by default is 10).
• "min":"<min>" - the minimum value (by default is 0).

For example, input JSON:

"columns": [{
	"name": "foo",
	"type": "number.hexadecimal",
	"options": {
		"min": 1,
		"max": 10
	}
}]

could result in value:

	A

###number.octal

Generates the random octal number.

Optional options available:

• "max":"<max>" - the maxiumum value (by default is 10).
• "min":"<min>" - the minimum value (by default is 0).

For example, input JSON:

"columns": [{
	"name": "foo",
	"type": "number.octal",
	"options": {
		"min": 1,
		"max": 10
	}
}]

could result in value:

	2

###regexp

Generates the random string matching specified regular expression.

Mandatory options available:

• "pattern":"<pattern>" - the pattern to match.

Optional options available:

• "escape": true|false - whether to escape (quote) result or not (used for specfific output formats only).

For example, input JSON:

"columns": [{
	"name": "foo",
	"type": "regexp",
	"options": {
		"pattern": "[0-9]{1,4}"
	}
}]

could result in value:

	720

###list.seq

Picks next value in a sequence from the provided nested column values.

For example, input JSON:

"columns": [{
	"name": "name1",
	"type": "list.seq",
	"columns": [{
		"name": "name1",
		"type": "const",
		"options": {
			"value": "value1"
		}
	}, {
		"name": "name2",
		"type": "const",
		"options": {
			"value": "value2"
		}
	}]
}]

results in values:

	value1
	value2
	value1
	value2
	...

###list.rand

Picks random value from the provided nested column values.

For example, input JSON:

"columns": [{
	"name": "name1",
	"type": "list.seq",
	"columns": [{
		"name": "name1",
		"type": "const",
		"options": {
			"value": "value1"
		}
	}, {
		"name": "name2",
		"type": "const",
		"options": {
			"value": "value2"
		}
	}]
}]

could result in values:

	value2
	value1
	value1
	value2
	...

###join

Joins nested column values with the separator (optionaly) provided.

Optional options available:

• "separator":"<separator>" - the separator string to be used for joining values.

For example (without separator), input JSON:

"columns": [{
	"name": "name1",
	"type": "join",
	"columns": [{
		"name": "name1",
		"type": "const",
		"options": {
			"value": "value1"
		}
	}, {
		"name": "name2",
		"type": "const",
		"options": {
			"value": "value2"
		}
	}]
}]

would result in value:

	value1value2

For example (with separator), input JSON:

"columns": [{
	"name": "name1",
	"type": "join",
	"columns": [{
		"name": "name1",
		"type": "const",
		"options": {
			"value": "value1"
		}
	}, {
		"name": "name2",
		"type": "const",
		"options": {
			"value": "value2"
		}
	}]
}, "options": {
	"separator": ", "
}]

would result in value:

	value1, value2

Contributing

1. Fork it ( https://github.com/datarator/datarator/fork )
2. Create your feature branch (git checkout -b my-new-feature)
3. Commit your changes (git commit -am 'Add some feature')
4. Push to the branch (git push origin my-new-feature)
5. Create a new Pull Request