Fix variable value changes not invalidating cache issue and etc.

[ZXCroon]

1. Fix caching: changing variables does not invalidate cache #104. The idea here is doing the templating (variable substitution) for the result of task.get_ds() to get the correct fingerprint of the task. To do the templating we need some runtime information, which we are not able to obtain in Task object. The hacky way I use is using gc module to obtain the WorkerProcess obj that references Task obj.

2. Fix Failed ansible tasks are written to cache #226. Let me explain why this issue happens: when a task failed, it is very likely (always when I tried it) is_failed here still returns False. It's a little bit complicated to explain why this happens behind the scene. Briefly speaking, the send_callback function does a clean_copy for task result that is gonna be sent to the hooks. And the clean_copy will wipe off the failed key in task result. The solution is to use status specific hooks instead of v2_on_any.

I tested my changes locally and two tests failed: integration/test_api.py::test_caching and integration/test_api.py::test_caching_mechanism but I think they failed for some reasons.

For test_caching, I am not sure I understand why it asserts that the task should not be cached (https://github.com/ansible-community/ansible-bender/blob/master/tests/integration/test_api.py#L43)

assert not b2.layers[4].cached

The corresponding task is a file action, but the source file does not change in between two builds. It should be cached.

- name: create a file
    copy:
      src: '{{ playbook_dir }}/a_bag_of_fun'
      dest: /fun

And this test passes before I made the changes due to a bug. Briefly speaking, in https://github.com/ansible-community/ansible-bender/blob/master/ansible_bender/callback_plugins/snapshoter.py, the task content is calculated twice (get_task_content called twice), before the task execution and after the task execution. For file-related task the result of two calculation will differ, causing wrong caching.

For test_caching_mechanism, it asserts this task will be cached

debug:
    msg: "{{ lookup('env','ANSIBLE_CONFIG', 'AB_BUILDER_NAME', 'AB_TARGET_IMAGE_NAME') }}"

If we do go down the variable substitution path, it makes more sense that this task does not get cached, because the ANSIBLE_CONFIG variable changes every run.

So we'll need to discuss about how we modify the tests, if I am not getting it wrong.

This is the first time I contribute to an open source project. Let me know if I am missing anything. Thanks!

[TomasTomecek]

Thank for this PR! Awesome changes and very detailed explanation, this is how PRs should be done! :)

[TomasTomecek]

could you please move this function out of the class (e.g. to the top of this file) and please specify in the docstring it's meant to be used in this method

[TomasTomecek]

thank you for all the details, this is super-helpful!

[TomasTomecek]

make this an empty dict please since you treat as a dict below

[TomasTomecek]

we should probably assert first that those keys are present in the dict so we wouldn't get KeyErrors in future if ansible-core ever changes its internals

[TomasTomecek]

I wonder if we should fall back if get_templated_ds runs into problems

[TomasTomecek]

It seems the test cases are wrong actually after reading your explanation. Please update them to what you feel like it's the right thing to do :)

[TomasTomecek]

Also sorry for replying so late, I was super-busy past few weeks, will try to provide replies in a timely manner so we can merge this next week.