The process is too slow

[harsh9t]

Hi,

I am using dbpedia 2015-04 version of dbpedia to generate the graph.db file for neo4j. There are a total of 71 .nt files to be loaded. So, far only 1 file has been loaded which 1.2 GB (name: category_labels_en.nt) in size and it took almost one whole day to do so. I am just wondering how much time it will take to load and process all 71 files.

Any insights?

[tiepologian]

Hi!

The time it takes depends on your hardware.
One of the most important factors is disk speed. Do you have an SSD disk?
Are you running a virtual machine? Or a VPS server? If you are, it will be very slow.

On our hardware the entire process, for all files, takes less than an hour.
Cheers

[harsh9t]

The machine I am using has a RAID storage of 6TB, there is also a 512 GB SSD but I am not using it as it is shared between users. The machine is a 12 core Xeon 2.8 GHz processor, 128 GB RAM device.

So far it has just managed to crunch 55 files out of 71 in 3 days :D I am just wondering and waiting it to finish. Any further comments?

Also one more question, How can I add multiple files in one single python command? I tried python $ dbpediaNeo4j.py /home/batman/git/dbpediaNeo4j/db_nt/ ; but the scripts terminates loading 0 relationships and no error. It is not able to take multiple .nt files.. should i mention each file separately?

*apologies for lame questions in advance :) *