Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add extra sleep time for 429 #19

Open
wants to merge 3 commits into
base: main
Choose a base branch
from
Open

add extra sleep time for 429 #19

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
33120fc
add extra sleep time for 429
jasonhcox Nov 28, 2023
1be7418
Change to exponential backoff
jasonhcox Nov 28, 2023
50a3c0e
change backoff factor
jasonhcox Nov 28, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 8 additions & 1 deletion airflow_provider_hightouch/hooks/hightouch.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,11 +40,13 @@ def __init__(
api_version: str = "v3",
request_max_retries: int = 3,
request_retry_delay: float = 0.5,
request_backoff_factor: float = 10.0
):
self.hightouch_conn_id = hightouch_conn_id
self.api_version = api_version
self._request_max_retries = request_max_retries
self._request_retry_delay = request_retry_delay
self._request_backoff_factor = request_backoff_factor
if self.api_version not in ("v1", "v3"):
raise AirflowException(
"This version of the Hightouch Operator only supports the v1/v3 API."
Expand Down Expand Up @@ -91,14 +93,19 @@ def make_request(
data=data,
headers=headers,
)

resp_dict = response.json()
return resp_dict["data"] if "data" in resp_dict else resp_dict
except AirflowException as e:
self.log.error("Request to Hightouch API failed: %s", e)
if num_retries == self._request_max_retries:
break
num_retries += 1
time.sleep(self._request_retry_delay)
if "429" in str(e):
backoff_delay = self._request_backoff_factor * (2 ** (num_retries - 1))
time.sleep(backoff_delay)
else:
time.sleep(self._request_retry_delay)

raise AirflowException("Exceeded max number of retries.")

Expand Down