上传过大（500MB+）的测试样例 Upload failed && 题目文件导入失败

[Estrella-Explore]

在提交issue之前请

• 认真阅读文档 http://docs.onlinejudge.me/#/
• 搜索和查看历史issues
• 安全类问题请不要在 GitHub 上公布，请发送邮件到 [email protected]，根据漏洞危害程度发送红包感谢。

然后提交issue请写清楚下列事项

 - 进行什么操作的时候遇到了什么问题，最好能有复现步骤
 - 错误提示是什么，如果看不到错误提示，请去data文件夹查看相应log文件。大段的错误提示请包在代码块标记里面。

• 你尝试修复问题的操作
• 页面问题请写清浏览器版本，尽量有截图

---

复现操作：上传一个较大的数据点（测试样例文件）（约 500MB+）：

Upload failed.

./data/backend/log/nginx.log：

2024/03/09 05:23:04 [warn] 106#106: *18746 a client request body is buffered to a temporary file /var/lib/nginx/tmp/client_body/0000000009, client: 61.136.xxx.xxx, server: _, request: "POST /api/admin/test_case HTTP/1.1", host: "47.115.xxx.xxx", referrer: "http://47.115.xxx.xxx/admin/problem/create"
2024/03/09 05:24:07 [error] 106#106: *18759 client intended to send too large body: 401714833 bytes, client: 61.136.xxx.xxx, server: _, request: "POST /api/admin/test_case HTTP/1.1", host: "47.115.xxx.xxx", referrer: "http://47.115.xxx.xxx/admin/problem/create"
2024/03/09 05:24:56 [error] 105#105: *18775 client intended to send too large body: 401714830 bytes, client: 61.136.xxx.xxx, server: _, request: "POST /api/admin/test_case HTTP/1.1", host: "47.115.xxx.xxx", referrer: "http://47.115.xxx.xxx/admin/problem/create"

---

---

---

另一个问题：
导入题目失败：

复现操作：导出题目后题目文件不加改动重新导入，有如下报错：

Invalid problem format, error is {'template': {'C++': {'prepend': [ErrorDetail(string='This field may not be blank.', code='blank')], 'template': [ErrorDetail(string='This field may not be blank.', code='blank')], 'append': [ErrorDetail(string='This field may not be blank.', code='blank')]}}}

./data/backend/log/gunicron.log：（log 提示没有 1/problem.json，但此压缩包中确实存在）

[2024-03-07 09:19:09 +0000] [120] [INFO] Starting gunicorn 20.1.0
[2024-03-07 09:19:09 +0000] [120] [INFO] Listening at: http://127.0.0.1:8080 (120)
[2024-03-07 09:19:09 +0000] [120] [INFO] Using worker: gthread
[2024-03-07 09:19:09 +0000] [124] [INFO] Booting worker with pid: 124
[2024-03-07 09:19:09 +0000] [125] [INFO] Booting worker with pid: 125
[2024-03-08 05:28:23 +0000] [120] [INFO] Handling signal: term
[2024-03-08 05:28:23 +0000] [124] [INFO] Worker exiting (pid: 124)
[2024-03-08 05:28:24 +0000] [125] [INFO] Worker exiting (pid: 125)
[2024-03-08 05:28:24 +0000] [120] [INFO] Shutting down: Master
[2024-03-08 05:29:03 +0000] [103] [INFO] Starting gunicorn 20.1.0
[2024-03-08 05:29:03 +0000] [103] [INFO] Listening at: http://127.0.0.1:8080 (103)
[2024-03-08 05:29:03 +0000] [103] [INFO] Using worker: gthread
[2024-03-08 05:29:03 +0000] [107] [INFO] Booting worker with pid: 107
[2024-03-08 05:29:03 +0000] [108] [INFO] Booting worker with pid: 108
[2024-03-08 05:56:56] - [ERROR] - [root:156]  - "There is no item named '1/problem.json' in the archive"
Traceback (most recent call last):
  File "/app/utils/api/api.py", line 149, in dispatch
    return super(APIView, self).dispatch(request, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/django/views/generic/base.py", line 98, in dispatch
    return handler(request, *args, **kwargs)           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^  File "/app/problem/views/admin.py", line 571, in post    with zip_file.open(f"{i}/problem.json") as f:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^  File "/usr/local/lib/python3.12/zipfile/__init__.py", line 1595, in open     zinfo = self.getinfo(name) 
  File "/usr/local/lib/python3.12/zipfile/__init__.py", line 1524, in getinfo
    raise KeyError(
KeyError: "There is no item named '1/problem.json' in the archive"
[2024-03-08 05:56:57] - [ERROR] - [sentry.errors:686]  - Sentry responded with an error: module 'ssl' has no attribute 'wrap_socket' (url: https://sentry.io/api/263057/store/)
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/raven/transport/threaded.py", line 165, in send_sync
    super(ThreadedHTTPTransport, self).send(url, data, headers)
  File "/usr/local/lib/python3.12/site-packages/raven/transport/http.py", line 38, in send
    response = urlopen(
               ^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/raven/utils/http.py", line 66, in urlopen
    return opener.open(url, data, timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/urllib/request.py", line 515, in open
    response = self._open(req, data)
               ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/urllib/request.py", line 532, in _open
    result = self._call_chain(self.handle_open, protocol, protocol +
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/urllib/request.py", line 492, in _call_chain
    result = func(*args)
             ^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/raven/utils/http.py", line 46, in https_open
    return self.do_open(ValidHTTPSConnection, req)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/urllib/request.py", line 1344, in do_open
    h.request(req.get_method(), req.selector, req.data, headers,
  File "/usr/local/lib/python3.12/http/client.py", line 1331, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/local/lib/python3.12/http/client.py", line 1377, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.12/http/client.py", line 1326, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.12/http/client.py", line 1085, in _send_output
    self.send(msg)
  File "/usr/local/lib/python3.12/http/client.py", line 1029, in send
    self.connect()
  File "/usr/local/lib/python3.12/site-packages/raven/utils/http.py", line 37, in connect
    self.sock = ssl.wrap_socket(
                ^^^^^^^^^^^^^^^
AttributeError: module 'ssl' has no attribute 'wrap_socket'
b"Sentry responded with an error: module 'ssl' has no attribute 'wrap_socket' (url: https://sentry.io/api/263057/store/)"
[2024-03-08 05:56:57] - [ERROR] - [sentry.errors.uncaught:712]  - ['"There is no item named \'1/problem.json\' in the archive"', '  File "utils/api/api.py", line 149, in dispatch', '  File "django/views/generic/base.py", line 98, in dispatch', '  File "problem/views/admin.py", line 571, in post', '  File "zipfile/__init__.py", line 1595, in open', '  File "zipfile/__init__.py", line 1524, in getinfo']
b'[\'"There is no item named \\\'1/problem.json\\\' in the archive"\', \'  File "utils/api/api.py", line 149, in dispatch\', \'  File "django/views/generic/base.py", line 98, in dispatch\', \'  File "problem/views/admin.py", line 571, in post\', \'  File "zipfile/__init__.py", line 1595, in open\', \'  File "zipfile/__init__.py", line 1524, in getinfo\']'
[2024-03-08 05:57:02] - [ERROR] - [root:156]  - not well-formed (invalid token): line 1, column 2
Traceback (most recent call last):
  File "/app/utils/api/api.py", line 149, in dispatch
    return super(APIView, self).dispatch(request, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/django/views/generic/base.py", line 98, in dispatch
    return handler(request, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/problem/views/admin.py", line 683, in post
    problems = FPSParser(tf.name).parse()
               ^^^^^^^^^^^^^^^^^^
  File "/app/fps/parser.py", line 15, in __init__
    self._etree = ET.parse(fps_path).getroot()
                  ^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/xml/etree/ElementTree.py", line 1204, in parse
    tree.parse(source, parser)
  File "/usr/local/lib/python3.12/xml/etree/ElementTree.py", line 569, in parse
    self._root = parser._parse_whole(source)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
xml.etree.ElementTree.ParseError: not well-formed (invalid token): line 1, column 2
[2024-03-08 05:57:02] - [ERROR] - [sentry.errors:686]  - Sentry responded with an error: module 'ssl' has no attribute 'wrap_socket' (url: https://sentry.io/api/263057/store/)
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/raven/transport/threaded.py", line 165, in send_sync
    super(ThreadedHTTPTransport, self).send(url, data, headers)
  File "/usr/local/lib/python3.12/site-packages/raven/transport/http.py", line 38, in send
    response = urlopen(
               ^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/raven/utils/http.py", line 66, in urlopen
    return opener.open(url, data, timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/urllib/request.py", line 515, in open
    response = self._open(req, data)
               ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/urllib/request.py", line 532, in _open
    result = self._call_chain(self.handle_open, protocol, protocol +
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/urllib/request.py", line 492, in _call_chain
    result = func(*args)
             ^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/raven/utils/http.py", line 46, in https_open
    return self.do_open(ValidHTTPSConnection, req)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/urllib/request.py", line 1344, in do_open
    h.request(req.get_method(), req.selector, req.data, headers,
  File "/usr/local/lib/python3.12/http/client.py", line 1331, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/local/lib/python3.12/http/client.py", line 1377, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.12/http/client.py", line 1326, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.12/http/client.py", line 1085, in _send_output
    self.send(msg)
  File "/usr/local/lib/python3.12/http/client.py", line 1029, in send
    self.connect()
  File "/usr/local/lib/python3.12/site-packages/raven/utils/http.py", line 37, in connect
    self.sock = ssl.wrap_socket(
                ^^^^^^^^^^^^^^^
AttributeError: module 'ssl' has no attribute 'wrap_socket'
b"Sentry responded with an error: module 'ssl' has no attribute 'wrap_socket' (url: https://sentry.io/api/263057/store/)"
[2024-03-08 05:57:02] - [ERROR] - [sentry.errors.uncaught:712]  - ['not well-formed (invalid token): line 1, column 2', '  File "utils/api/api.py", line 149, in dispatch', '  File "django/views/generic/base.py", line 98, in dispatch', '  File "problem/views/admin.py", line 683, in post', '  File "fps/parser.py", line 15, in __init__', '  File "xml/etree/ElementTree.py", line 1204, in parse', '  File "xml/etree/ElementTree.py", line 569, in parse']
b'[\'not well-formed (invalid token): line 1, column 2\', \'  File "utils/api/api.py", line 149, in dispatch\', \'  File "django/views/generic/base.py", line 98, in dispatch\', \'  File "problem/views/admin.py", line 683, in post\', \'  File "fps/parser.py", line 15, in __init__\', \'  File "xml/etree/ElementTree.py", line 1204, in parse\', \'  File "xml/etree/ElementTree.py", line 569, in parse\']'
[2024-03-08 05:57:07] - [ERROR] - [root:156]  - not well-formed (invalid token): line 1, column 2
Traceback (most recent call last):
  File "/app/utils/api/api.py", line 149, in dispatch
    return super(APIView, self).dispatch(request, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/django/views/generic/base.py", line 98, in dispatch
    return handler(request, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/problem/views/admin.py", line 683, in post
    problems = FPSParser(tf.name).parse()
               ^^^^^^^^^^^^^^^^^^
  File "/app/fps/parser.py", line 15, in __init__
    self._etree = ET.parse(fps_path).getroot()
                  ^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/xml/etree/ElementTree.py", line 1204, in parse
    tree.parse(source, parser)
  File "/usr/local/lib/python3.12/xml/etree/ElementTree.py", line 569, in parse
    self._root = parser._parse_whole(source)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^

[SongJunxi2000]

您好，请问上传较大数据点会失败后续有解决掉吗？我们目前遇到的问题是20MB的数据点都无法上传。谢谢

[Estrella-Explore]

您好，请问上传较大数据点会失败后续有解决掉吗？我们目前遇到的问题是20MB的数据点都无法上传。谢谢
@SongJunxi2000

Nope.

[zhblue]

如果测试数据是放在特定目录里的，可以考虑用sftp等方式上传。