You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
We are using sqoop --num-mappers=40[1] for all Rucio table dump queries. However, some tables like rses, subscriptions are so small tables and their run time can be decreased with --num-mappers=1.
Previous experiences showed that we can run 2 --num-mappers-40 parallel, we did not implement it for Rucio table dumps since we don't have a time pressure. However, we can expect that our users will come with more Rucio table dump request, so we can implement it now.
So, my suggestions are:
Run, at most, 2 Rucio table dumps parallel.
Set --num-mappers=1 for "rses" and "subscriptions" tables.
P.S.: dbs3_full_global.sh is already using parallel dumps and logic can be copied from there.
We are using sqoop
--num-mappers=40
[1] for all Rucio table dump queries. However, some tables likerses, subscriptions
are so small tables and their run time can be decreased with--num-mappers=1
.Previous experiences showed that we can run 2
--num-mappers-40
parallel, we did not implement it for Rucio table dumps since we don't have a time pressure. However, we can expect that our users will come with more Rucio table dump request, so we can implement it now.So, my suggestions are:
P.S.:
dbs3_full_global.sh
is already using parallel dumps and logic can be copied from there.[1] https://github.com/dmwm/CMSMonitoring/blob/master/sqoop/scripts/rucio_table_dumps.sh#L49
The text was updated successfully, but these errors were encountered: