Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

better way to handle scripts that export data from log directories #36

Open
doudou opened this issue May 14, 2018 · 0 comments
Open

better way to handle scripts that export data from log directories #36

doudou opened this issue May 14, 2018 · 0 comments

Comments

@doudou
Copy link
Member

doudou commented May 14, 2018

e.g. export streams to CSV files for matlab.

The issue right now is that the file names depend on the deployment choices - they change if someone changes the deployment. We should have a way to do it based on the stream name.

One way to handle this would be to finally normalize log generation to have a single file per stream. This would allow one to easily pick what stream he/she wants and needs and copy only that without complex changes. This is what the syskit-pocolog integration does when normalizing datasets, but we could move it to the logger itself instead. The downside is that it may be more CPU and I/O-intensive (more syscalls since we don't multiplex all streams into a single I/O anymore)

Another way would be to make pocolog (the command line tool) and pocolog (the ruby api) understand folders instead of single files, through e.g. a Pocolog::Directory API.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant