Synchronization process

Synchronize

Download

• Download all updates up until today for both CHIEF and TARIC.
• We only download updates since 6th of June 2012 for TARIC and 30th of June 2012 for CHIEF (as initial seed files cover measures before these dates).
• We notify tariff alerts list if there is no CHIEF or TARIC update for 3rd time in a row (missing_updates email).
• We notify tariff alerts list if update cannot be written to file system due to permission errors (file_write_error email).
• We mark update as failed if we retry downloading it for pre-set max retry count. If there are any failed updates, updates will not applied later. We send retry_count_exceeded email to tariff alerts list.
• If we receive update with blank content we mark it as failed and send email (blank_update) to tariff alerts list.
• We do not redownload failed updates. You have to manually remove this entry from file system so that it would get redownloaded.
• We mark update as missing if server successfully returns 404 Not Found response without errors. We mark updates as missing only on the next day, considering the fact that may appear in the server at any time throughout the day. Missing updates do not need redownloading.
• Successfully downloaded update is marked as Pending (as pending for application).
• Applied update is the one that has been parsed and its records were transfered to the database.

Apply

• Pre-condition: application process does not start if there are updates marked as failed. We notify tariff alerts list with failures_reminder email.
• We apply updates in ascending (by issue date) order starting with TARIC as CHIEF can depend on changes in TARIC.
• We invoke Chief Transformer (see next section for details) after each CHIEF update is applied.
• Each update application (and transformation) is wrapped into transaction, so in case of any errors we undo the changes.
• We send error email (applied) to tariff alerts list in case there is an exception during application/transformation.

Transform

• Traverse unprocessed operation in chief_mfcm, chief_tame tables an process them.
• Send success email if transformation succeeded (successful_transformation_notice) to tariff alerts list.
• Send email with exception details if transformation fails (failed_transformation_notice) to tariff alerts list.
• Undo transaction, reverting all changes (see above in apply section).

Notes

• There is a reason behind sequence of each_day.loop { TARIC update application; CHIEF update application; CHIEF update transformation }. If we would apply all TARIC updates where certain GoodsNomenclature entry would be introduced and later removed (say in a long term), then applying CHIEF updates after all TARIC's updates may yield errors as certain measures couldn't be created since that GoodsNomenclature entry won't exist.