Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix: handle Sentry issue for CSV downloading errors #212

Open
wants to merge 6 commits into
base: main
Choose a base branch
from
Open

fix: handle Sentry issue for CSV downloading errors #212

Changes from 1 commit
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
bcd85d2
fix: fix sentry error 145755
bolinocroustibat Oct 30, 2024
491bc70
Merge branch 'main' into fix-sentry-issue-145755
bolinocroustibat Nov 15, 2024
32d7f95
feat: add custom IOException error
bolinocroustibat Nov 15, 2024
1a9e667
fix: fix exception logic
bolinocroustibat Nov 15, 2024
a4e84bb
Merge branch 'main' into fix-sentry-issue-145755
bolinocroustibat Nov 15, 2024
883c695
Merge branch 'main' into fix-sentry-issue-145755
bolinocroustibat Nov 20, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 12 additions & 9 deletions udata_hydra/utils/file.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,15 +52,18 @@ async def download_resource(
async with aiohttp.ClientSession(
headers={"user-agent": config.USER_AGENT}, raise_for_status=True
) as session:
async with session.get(url, allow_redirects=True) as response:
async for chunk in response.content.iter_chunked(chunk_size):
if max_size_allowed is None or i * chunk_size < max_size_allowed:
tmp_file.write(chunk)
else:
tmp_file.close()
log.warning(f"File {url} is too big, skipping")
raise IOError("File too large to download")
i += 1
try:
async with session.get(url, allow_redirects=True) as response:
async for chunk in response.content.iter_chunked(chunk_size):
if max_size_allowed is None or i * chunk_size < max_size_allowed:
tmp_file.write(chunk)
else:
tmp_file.close()
log.warning(f"File {url} is too big, skipping")
raise IOError("File too large to download")
i += 1
except aiohttp.ClientResponseError as e:
raise IOError(f"Error downloading CSV: {e}")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think this makes the error more explicit than previously?
I think the issue is more about marking these as "expected" in sentry

Copy link
Contributor Author

@bolinocroustibat bolinocroustibat Nov 15, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the error is more explicit that way, at least it does describe it in the title. For now in Sentry it's not easy to identify what is the issue is when listing the errors, and that way we can see immediately in list mode that it's an "expected" error.

We could also create a specific exception for this, which would include attributes to identify the ressource, as we did for the custom ParseException. What do you think?

Copy link
Contributor Author

@bolinocroustibat bolinocroustibat Nov 15, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could also create a specific exception for this, which would include attributes to identify the ressource, as we did for the custom ParseException. What do you think?

Update: it has been done in this PR, as a proposal. Please review again!

tmp_file.close()
if magic.from_file(tmp_file.name, mime=True) in [
"application/x-gzip",
Expand Down