stop workers during online backup

[evgeni]

this ensures more consistent on-disk data

[evgeni]

# hammer job-invocation create --job-template "Run Command - Script Default" --inputs 'command=sleep 120' --search-query "name = $(hostname -f)" --async
Job invocation 5 created

# foreman-maintain backup online /var/tmp/b
Starting backup: 2024-06-07 07:29:04 +0000
Running preparation steps required to run the next scenarios
================================================================================
Make sure Foreman DB is up: 
/ Checking connection to the Foreman DB                               [OK]      
--------------------------------------------------------------------------------
Make sure Candlepin DB is up: 
- Checking connection to the Candlepin DB                             [OK]      
--------------------------------------------------------------------------------
Make sure Pulpcore DB is up: 
\ Checking connection to the Pulpcore DB                              [OK]      
--------------------------------------------------------------------------------


Running Backup
================================================================================
Check for running tasks:                                              [FAIL]
There are 2 active task(s) in the system.
Please wait for these to complete or cancel them from the Monitor tab.
--------------------------------------------------------------------------------
There are multiple steps to proceed:
1) Fetch tasks status and wait till they finish
2) Investigate the tasks via UI
Select step to continue, [n(next), q(quit)] 1
Fetch tasks status and wait till they finish:                                   
| Try checking status of running task(s)                                        
There are 2 running tasks.
| Try checking status of running task(s)                                        
There are 2 running tasks.
\ Try checking status of running task(s)                                        
There are 2 running tasks.
\ Try checking status of running task(s)                                        
There are 2 running tasks.
\ Try checking status of running task(s)                                        
There are 2 running tasks.
\ Try checking status of running task(s)                                        
There are 2 running tasks.
| Try checking status of running task(s)                                        
There are 2 running tasks.
| Try checking status of running task(s)                                        
There are 2 running tasks.
| Try checking status of running task(s)                                        
There are 2 running tasks.
| Try checking status of running task(s)                                        
There are 2 running tasks.
| Try checking status of running task(s)                                        
There are 2 running tasks.
| Try checking status of running task(s)                                        
There are 1 running tasks.
| Try checking status of running task(s)                                        
There are 1 running tasks.
| Try checking status of running task(s)                              [OK]      
--------------------------------------------------------------------------------
Rerunning the check after fix procedure
Check for running tasks:                                              [OK]
--------------------------------------------------------------------------------
Check for running pulpcore tasks:                                     [OK]

[evgeni]

# PULP_SETTINGS=/etc/pulp/settings.py pulpcore-manager shell < lol.py 
/pulp/api/v3/tasks/018ff1cc-1924-7108-9b17-0b7adc330e9b/

# pulp task list --state running
[
  {
    "pulp_href": "/pulp/api/v3/tasks/018ff1cc-1924-7108-9b17-0b7adc330e9b/",
    "pulp_created": "2024-06-07T08:23:55.941268Z",
    "pulp_last_updated": "2024-06-07T08:23:55.941279Z",
    "state": "running",
    "name": "pulpcore.app.tasks.test.sleep",
    "logging_cid": "lol",
    "created_by": "/pulp/api/v3/users/1/",
    "unblocked_at": "2024-06-07T08:23:55.952339Z",
    "started_at": "2024-06-07T08:23:55.972970Z",
    "finished_at": null,
    "error": null,
    "worker": "/pulp/api/v3/workers/018ff195-c89d-70bd-8262-16b5431226bc/",
    "parent_task": null,
    "child_tasks": [],
    "task_group": null,
    "progress_reports": [],
    "created_resources": [],
    "reserved_resources_record": [
      "shared:/pulp/api/v3/domains/018fb72c-f3d2-7e49-9b59-20a4c2f7da44/"
    ]
  }
]

# foreman-maintain backup online /var/tmp/b
Starting backup: 2024-06-07 08:24:01 +0000
Running preparation steps required to run the next scenarios
================================================================================
Make sure Foreman DB is up: 
/ Checking connection to the Foreman DB                               [OK]      
--------------------------------------------------------------------------------
Make sure Candlepin DB is up: 
- Checking connection to the Candlepin DB                             [OK]      
--------------------------------------------------------------------------------
Make sure Pulpcore DB is up: 
\ Checking connection to the Pulpcore DB                              [OK]      
--------------------------------------------------------------------------------


Running Backup
================================================================================
Check for running tasks:                                              [OK]
--------------------------------------------------------------------------------
Check for running pulpcore tasks:                                     [FAIL]
There are 1 active task(s) in the system.
Please wait for these to complete.
--------------------------------------------------------------------------------
Continue with step [Fetch tasks status and wait till they finish]?, [y(yes), n(no), q(quit)] y
Fetch tasks status and wait till they finish:                                   
- waiting for tasks to finish                                                   
There are 1 tasks.
/ Waiting 10 seconds before retry.                                              
There are 1 tasks.
\ Waiting 10 seconds before retry.                                              
There are 1 tasks.
/ Waiting 10 seconds before retry.                                              
There are 1 tasks.
| Waiting 10 seconds before retry.                                              
There are 1 tasks.
- Waiting 10 seconds before retry.                                              
There are 1 tasks.
| Waiting 10 seconds before retry.                                    [OK]      
--------------------------------------------------------------------------------
Rerunning the check after fix procedure
Check for running pulpcore tasks:                                     [OK]

lol.py being:

from django_guid import set_guid 
from pulpcore.tasking.tasks import dispatch 
from pulpcore.app.util import get_url
set_guid('lol') 
task = dispatch("pulpcore.app.tasks.test.sleep", args=(60,)) 
print(get_url(task))

[evgeni]

Right now I've implemented this as a boolean flag, but am wondering if instead we want to give users the ability to configure the timeout. (the current one is 300, so 5 minutes, for both pulp and foreman, so a maximum of 10 minutes waiting)

[ehelms]

Yes - however we could make it explicit (options to the CLI) or "hidden" (environment variables).

[evgeni]

I think that is the only one that is left undecided.
I must admit I kinda find env vars bad for discovery, but on the other hand we'd have to add the cli command to update/upgrade too if we'd go with that (to be consistent).
But I also really have no feeling how often people would actually want to override that.

So, maybe, start with env, and then see how many complaints we get?

[evgeni]

should the whole "wait for tasks before backup" part be split out in an own commit/PR?

[evgeni]

split the big patch into four logical commits, enjoy!

[ehelms]

Needs a rebase!

[evgeni]

got a rebase!

[evgeni]

hm, need to adjust tests. later.