Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

More transparent handling of CastErrors #107

Open
micimize opened this issue Aug 8, 2019 · 1 comment
Open

More transparent handling of CastErrors #107

micimize opened this issue Aug 8, 2019 · 1 comment

Comments

@micimize
Copy link

micimize commented Aug 8, 2019

I think exception handling should be done at the processor level, but just giving a way to override the behavior would be good as well

def process(self):
ds = self._process()
try:
for res in ds.res_iter:
collections.deque(res, maxlen=0)
except CastError as e:
for err in e.errors:
logging.error('%s', err)
return ds.dp, ds.merge_stats()

@akariv
Copy link
Member

akariv commented Aug 26, 2019

@micimize
Note that the validate processor provides means to act upon individual casting errors (including whether to drop/pass the failing rows). You can add this processor as the last one in your flow to create special handling for such errors.
I think the error logs in the process method should remain - as an indication to the user that something is not right - but the handling should be done using a processor.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants