This tool generates products tables data file that can be re-imported wih very fast MySQL native tools.
You need a PIM installation with all structural entities already configured (attributes, families, categories, attribute options, etc...), but without product.
usage: <akeneo-product-generator-full-jar-file>
-u <DATABASE-URL> Database JDBC URL. Required.
-c <PRODUCTS-COUNT> Number of products to generate. Required.
-o <OUTPUT-DIR> Output directory for generated files. Required.
-m <arg> Elasticsearch Product and Product Model Index name.
Defaults to 'akeneo_pim_product_and_product_model'.
-p <arg> Elasticsearch Product Index name. Defaults to
'akeneo_pim_product'.
- example:
$ java -jar akeneo-product-generator-full-0.3.0.jar -u="jdbc:mysql://localhost/pim_ce_20?user=akeneo_pim&password=akeneo_pim" -o=/var/tmp -c=1000000
It will generate 1 million products based on the structure from the local MySQL on the pim_ce_20
database.
mysql> LOAD DATA LOCAL INFILE '/var/tmp/products.tsv' INTO TABLE pim_catalog_product;
mysql> LOAD DATA LOCAL INFILE '/var/tmp/products-categories.tsv' INTO TABLE pim_catalog_category_product;
mysql> LOAD DATA LOCAL INFILE '/var/tmp/products-unique-data.tsv' INTO TABLE pim_catalog_product_unique_data;
for ES_FILE in /var/tmp/es/*.gzip; do
echo "Loading $ES_FILE..."
gunzip -c $ES_FILE | curl -s -H "Content-Type: application/x-ndjson" -XPOST localhost:9200/_bulk --data-binary @- | cut -b 1-50
done
ERROR 1206 (HY000): The total number of locks exceeds the lock table size
If you meet this error during the LOAD DATA
commands (especially category associations and unique data),
you need to increase the innodb_buffer_pool_size
for you MySQL server.
From the root directory:
$ gradle build
Some metrics for 1 million products from the icecat_demo_dev structure:
- Generating data files: 64s
- Importing into the pim_catalog_product table: 2 min 30s
- Importing into the pim_catalog_category_product table: 1 min 25s
- Importing into pim_catalog_product_unique_data: 33s
=> full process from 0 product to 1 million products in DB: 5 min 30s