Skip to content
This repository has been archived by the owner on Jun 21, 2022. It is now read-only.

writing TDirectory doesn't properly handle >2GB #462

Open
PerilousApricot opened this issue Mar 4, 2020 · 0 comments
Open

writing TDirectory doesn't properly handle >2GB #462

PerilousApricot opened this issue Mar 4, 2020 · 0 comments

Comments

@PerilousApricot
Copy link

reported on the iris-hep coffea slack @ https://iris-hep.slack.com/archives/CFZ4Z2Q2H/p1583249545026400

Traceback (most recent call last):
  File "/usr/lib64/python3.6/multiprocessing/pool.py", line 119, in worker
    result = (True, func(*args, **kwds))
  File "/usr/lib64/python3.6/multiprocessing/pool.py", line 44, in mapstar
    return list(map(*args))
  File "run.py", line 24, in process_file
    out_file[out_treename] = newtree
  File "/afs/cern.ch/user/g/gstark/pyhf/lib64/python3.6/site-packages/uproot/write/TFile.py", line 100, in __setitem__
    self._rootdir.setkey(newkey)
  File "/afs/cern.ch/user/g/gstark/pyhf/lib64/python3.6/site-packages/uproot/write/TDirectory.py", line 97, in setkey
    self.writekeys(newcursor)
  File "/afs/cern.ch/user/g/gstark/pyhf/lib64/python3.6/site-packages/uproot/write/TDirectory.py", line 76, in writekeys
    self.update()
  File "/afs/cern.ch/user/g/gstark/pyhf/lib64/python3.6/site-packages/uproot/write/TDirectory.py", line 43, in update
    self.cursor.update_fields(self.sink, self._format1, fVersion, self.fDatimeC, fDatimeM, self.fNbytesKeys, self.fNbytesName, self.fSe
ekDir, self.fSeekParent, self.fSeekKeys)
  File "/afs/cern.ch/user/g/gstark/pyhf/lib64/python3.6/site-packages/uproot/write/sink/cursor.py", line 17, in update_fields
    sink.write(format.pack(*args), self.index)
struct.error: 'i' format requires -2147483648 <= number <= 2147483647

It appears that adding a new tree after the file is >2GB, the tdirectory overflows the 32-bit len @ https://github.com/scikit-hep/uproot/blob/6b7e886efc0720a20b3622a5b2840cb1fa0558bc/uproot/write/TDirectory.py#L43

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

2 participants