Skip to content

Commit

Permalink
[justice] i kdyz necachujem, tak ukladame na disk (#296)
Browse files Browse the repository at this point in the history
Je tu takovy problem, ze kdyz ctu primo z otevreny HTTP konexe a udelam
jich hodne, justicni webserver se trochu pomatne, tak je treba k nim
pristupovat setrneji a drzet ty konexe kratce.
  • Loading branch information
kokes authored May 31, 2024
1 parent e414de1 commit d50f5d9
Showing 1 changed file with 13 additions and 6 deletions.
19 changes: 13 additions & 6 deletions data/justice/main.py
Original file line number Diff line number Diff line change
Expand Up @@ -89,15 +89,22 @@ def cached_urlopen(url, timeout=HTTP_TIMEOUT):
return

with urlopen(url, timeout=timeout) as r:
if not CACHE_ENABLED:
yield r
return
fn_tmp = fn_path + ".tmp"
# driv jsme cetli rovnou z webu, ale delalo to problemy,
# tak to holt docasne ukladame a poustime to z disku
with open(fn_tmp, "wb") as f:
shutil.copyfileobj(r, f)
os.rename(fn_tmp, fn_path)
with cached_urlopen(url, timeout) as r:
yield r

# pri vypnuty cache se jen cte s tempfilu a pak se maze
if not CACHE_ENABLED:
with open(fn_tmp, "rb") as f:
yield f
os.remove(fn_tmp)
return

os.rename(fn_tmp, fn_path)
with cached_urlopen(url, timeout) as r:
yield r


def nahraj_ds(url):
Expand Down

0 comments on commit d50f5d9

Please sign in to comment.