feat(kfp): add train stage

[tumido]

Competition PR to #14

This bring Training stage to the pipeline

This is how it looks when run with faked SDG (I didn't want to wait for the PyTorchJob so I made it timeout on the wait step):

What works:

• Scheduling PyTorchJob
• Waiting for it
• Consuming SDG output and passing it to PVC
• Downloading base model from HuggingFace and storing it in Artifact
• Populating a PVC with base model
• Creating a PVC for output collection
• Collecting outputs from PVC into artifacts (Model + other outputs)
• Mounting all PVC to PyTorchJob

TODO:

• Fix passing data to PytorchJob, PVC requirement may not be the best option here
• Fix kubectl wait to allow timeout/namespace params, I was initially not successful here due to some KFP templating issues...
• See if we can use something better for PyTorchJob metadata.name until dsl.PIPELINE_* is available ([sdk] Cannot get real job_id by using PIPELINE_JOB_ID_PLACEHOLDER in component kubeflow/pipelines#10453)
• See if we can do better than the storage class enforced by KFP: https://github.com/kubeflow/pipelines/blob/1cded35cf5e93d8c8d32fefbddceb2eed8de9a0a/backend/src/v2/driver/driver.go#L1428-L1436
• Add params/args everywhere for more customization

[MichaelClifford]

There is an [rank1]: PermissionError: [Errno 13] Permission denied: '/.cache' error during training, as torch attempts to write to the .cache dir. We can fix it by setting the following env variable.

Suggested change

	XDG_CACHE_HOME=/tmp

[Shreyanand]

LGTM in the current state. The PR adds most of the sdg + training pipeline skeleton. We could merge and iterate over it.