Speeding up inserts

[jonasob]

At present, it seems the load script only deals with ~1,5M works per day, which means that with a hashing rate of at least 3M per day, the load script will never catch up and just get more and more out of sync.

What can we do to speed up loading massively? @artfwo has suggested loading in parallel. @petli would appreciate your thoughts on this if you have some bandwidth soon?

[petli]

The datamodel is not at all suited to what we're doing now, and it starts to seem like a mistake to not have modified it earlier. However, one rather small change we could do that I think would speed up things is this:

Instead of storing Media as separate objects and just linking to them from the Work, they could be stored as subdocuments directly in the Work.media array. This will not allow the fancy forking etc we planned originally, but that's not needed as long as we only work with pre-existing collections. Later, we could revert to supporting both embedded and linked Media if necessary.

The createWork function in the core module should also allow all annotations and media objects to be included directly. I.e. the load script can then build up a complete Work object with all annotations and media subobjects in it, and then just make one call to push this into mongo rather than as now adding each annotation and media one by one.

[petli]

Fixing #85 and #86 should also shave a little time off.

[petli]

Better idea: let the load script modify the database directly, since we can't yet use the events generated by the core module anyway:

14:43 <goodoldshep> In fact, changing the load script to just write directly to mongodb, but not changing the data model, would probably be sufficient and fast enough.
14:44 <goodoldshep> It should still use the schemas and models in modules/core/lib/db.js to ensure correct data model, but just ignore the API on top of that.  We can trust the load script, so we don't need all the checks in the core module intended for the REST API.
14:45 <artfwo> indeed
14:45 <goodoldshep> Create all the Media objects first, add them to Work.media together with the annotations, then create the Work.
14:46 <goodoldshep> It'll be a couple of database writes only, instead as now when each annotation and media requires first reading the work, updating it and then saving the results.
14:47 <goodoldshep> s/add them/add their IDs/
14:48 <artfwo> that still implies linked media, right?
14:49 <goodoldshep> Yes.  I don't think we need to change that if we take the shortcut in the load script, so then we can just let the core module and the REST API be untouched.
14:50 <artfwo> sounds good

[artfwo]

<goodoldshep> The load script should also make sure that https://github.com/commonsmachinery/catalog/issues/85 is taken into consideration.  But I think it effectively would just by not creating the fields in the first place.