Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Control the concurrency at the time of child table execution #578

Open
bigdatasourav opened this issue Jun 21, 2023 · 1 comment
Open

Control the concurrency at the time of child table execution #578

bigdatasourav opened this issue Jun 21, 2023 · 1 comment
Assignees

Comments

@bigdatasourav
Copy link

As per my understanding, if I execute select * from child_table, internally first, it will execute the parent list method and then execute the child list method concurrently based on the number of results we get from the parent list method. Can we control the Concurrency in this scenario like we do in the case of HydrateConfig?

HydrateConfig: []plugin.HydrateConfig{
			{
				Func:           listUserGroups,
				MaxConcurrency: 5,
				RetryConfig: &plugin.RetryConfig{
					ShouldRetryErrorFunc: shouldRetryError([]string{"429"}),
				},
			},
			{
@bigdatasourav bigdatasourav changed the title Control the Concurrency at the time of child table execution Control the concurrency at the time of child table execution Jun 21, 2023
@heasman-proof
Copy link

I am also running into this issue when developing a custom plugin. While its possible to add a limiter at the .spc level, it would be very helpful to control this at the Hydrate function level.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants