Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Meaning of parameters in 'configuration.toml' #21

Open
wenlinyao opened this issue Jan 26, 2024 · 9 comments
Open

Meaning of parameters in 'configuration.toml' #21

wenlinyao opened this issue Jan 26, 2024 · 9 comments

Comments

@wenlinyao
Copy link

Hi,

In the 'configuration.toml' file, I see a range of parameters, but not sure what those parameters control. Could you please provide one example config file that can produce the result in Table 1 and Table 2 of the paper? One example for each table will be great! Thank you!

image image
@mrT23
Copy link
Contributor

mrT23 commented Jan 27, 2024 •
edited
Loading

The table shows the results of running:
https://github.com/Codium-ai/AlphaCodium?tab=readme-ov-file#solving-the-entire-dataset

With:

[dataset]
num_iterations=5

Beware - this is quite a long and expensive run...

To maximize the score, add some "randomness" between different iterations, meaning choose somewhat different params for the stages in different iterations, to get the best possible chances for solving.

I am not going to share at this point my full "recipe" for that, as it is just an ensemble detail, which is highly uncommon to share in academic implementations. even without adding randomness, you should get results close to that.

@wenlinyao
Copy link
Author

Thanks for your reply!

When I run it with more iterations, I sometimes can see one error message as follows. Will it create problems in evaluation?

ERROR:asyncio:Task exception was never retrieved
future: <Task finished name='Task-11476' coro=<AsyncClient.aclose() done, defined at /data1/wenlinyao/miniconda3/envs/alphacode/lib/python3.9/site-packages/httpx/_client.py:1996> exception=RuntimeError('Event loop is cl
osed')>
Traceback (most recent call last):
File "/data1/wenlinyao/miniconda3/envs/alphacode/lib/python3.9/site-packages/httpx/_client.py", line 2006, in aclose
await proxy.aclose()
File "/data1/wenlinyao/miniconda3/envs/alphacode/lib/python3.9/site-packages/httpx/_transports/default.py", line 383, in aclose
await self._pool.aclose()
File "/data1/wenlinyao/miniconda3/envs/alphacode/lib/python3.9/site-packages/httpcore/_async/connection_pool.py", line 324, in aclose
await connection.aclose()
File "/data1/wenlinyao/miniconda3/envs/alphacode/lib/python3.9/site-packages/httpcore/_async/http_proxy.py", line 350, in aclose
await self._connection.aclose()
File "/data1/wenlinyao/miniconda3/envs/alphacode/lib/python3.9/site-packages/httpcore/_async/http11.py", line 253, in aclose
await self._network_stream.aclose()
File "/data1/wenlinyao/miniconda3/envs/alphacode/lib/python3.9/site-packages/httpcore/_backends/anyio.py", line 54, in aclose
await self._stream.aclose()
File "/data1/wenlinyao/miniconda3/envs/alphacode/lib/python3.9/site-packages/anyio/streams/tls.py", line 202, in aclose
await self.transport_stream.aclose()
File "/data1/wenlinyao/miniconda3/envs/alphacode/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 1181, in aclose
self._transport.close()
File "/data1/wenlinyao/miniconda3/envs/alphacode/lib/python3.9/asyncio/selector_events.py", line 698, in close
self._loop.call_soon(self._call_connection_lost, None)
File "/data1/wenlinyao/miniconda3/envs/alphacode/lib/python3.9/asyncio/base_events.py", line 751, in call_soon
self._check_closed()
File "/data1/wenlinyao/miniconda3/envs/alphacode/lib/python3.9/asyncio/base_events.py", line 515, in _check_closed
raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed

@mrT23
Copy link
Contributor

mrT23 commented Jan 30, 2024

can you reproduce this error ? does it happen on specific problems ? does it happen always ?

@wenlinyao
Copy link
Author

Yeah, this error is quite random. The output is something like this:

2024-01-29 23:29:54.595 | INFO | alpha_codium.gen.stages.run_initial_solve:run_initial_solve:14 - --initial solve stage--
2024-01-29 23:29:54.600 | INFO | alpha_codium.llm.ai_handler:chat_completion:86 - -----------------
2024-01-29 23:29:54.601 | INFO | alpha_codium.llm.ai_handler:chat_completion:87 - Running inference ...
2024-01-29 23:30:00.540 | INFO | alpha_codium.llm.ai_handler:chat_completion:133 - done
2024-01-29 23:30:00.540 | INFO | alpha_codium.llm.ai_handler:chat_completion:134 - -----------------
2024-01-29 23:30:00.606 | INFO | alpha_codium.code_contests.eval.code_test_runners:print_test_results:111 - test-0 :: status=ProgramStatus.kSuccess, passed=False
2024-01-29 23:30:00.607 | INFO | alpha_codium.code_contests.eval.code_test_runners:print_test_results:116 - =====================================================================
2024-01-29 23:30:00.607 | INFO | alpha_codium.gen.stages.run_initial_code_generation:run_initial_code_generation:61 - Failed to pass tests after 2 attempts, d: 8.0, best_d so far: 5
2024-01-29 23:30:00.607 | INFO | alpha_codium.gen.stages.run_initial_solve:run_initial_solve:14 - --initial solve stage--
2024-01-29 23:30:00.613 | INFO | alpha_codium.llm.ai_handler:chat_completion:86 - -----------------
2024-01-29 23:30:00.613 | INFO | alpha_codium.llm.ai_handler:chat_completion:87 - Running inference ...
ERROR:asyncio:Task exception was never retrieved
future: <Task finished name='Task-24504' coro=<AsyncClient.aclose() done, defined at /data1/wenlinyao/miniconda3/envs/alphacode/lib/python3.9/site-packages/httpx/_client.py:1996> exception=RuntimeError('Event loop is cl
osed')>
Traceback (most recent call last):
File "/data1/wenlinyao/miniconda3/envs/alphacode/lib/python3.9/site-packages/httpx/_client.py", line 2006, in aclose
await proxy.aclose()
File "/data1/wenlinyao/miniconda3/envs/alphacode/lib/python3.9/site-packages/httpx/_transports/default.py", line 383, in aclose
await self._pool.aclose()
File "/data1/wenlinyao/miniconda3/envs/alphacode/lib/python3.9/site-packages/httpcore/_async/connection_pool.py", line 324, in aclose
await connection.aclose()
File "/data1/wenlinyao/miniconda3/envs/alphacode/lib/python3.9/site-packages/httpcore/_async/http_proxy.py", line 350, in aclose
await self._connection.aclose()
File "/data1/wenlinyao/miniconda3/envs/alphacode/lib/python3.9/site-packages/httpcore/_async/http11.py", line 253, in aclose
await self._network_stream.aclose()
File "/data1/wenlinyao/miniconda3/envs/alphacode/lib/python3.9/site-packages/httpcore/_backends/anyio.py", line 54, in aclose
await self._stream.aclose()
File "/data1/wenlinyao/miniconda3/envs/alphacode/lib/python3.9/site-packages/anyio/streams/tls.py", line 202, in aclose
await self.transport_stream.aclose()
File "/data1/wenlinyao/miniconda3/envs/alphacode/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 1181, in aclose
self._transport.close()
File "/data1/wenlinyao/miniconda3/envs/alphacode/lib/python3.9/asyncio/selector_events.py", line 698, in close
self._loop.call_soon(self._call_connection_lost, None)
File "/data1/wenlinyao/miniconda3/envs/alphacode/lib/python3.9/asyncio/base_events.py", line 751, in call_soon
self._check_closed()
File "/data1/wenlinyao/miniconda3/envs/alphacode/lib/python3.9/asyncio/base_events.py", line 515, in _check_closed
raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed
ERROR:asyncio:Task exception was never retrieved
ERROR:asyncio:Task exception was never retrieved
future: <Task finished name='Task-24506' coro=<AsyncClient.aclose() done, defined at /data1/wenlinyao/miniconda3/envs/alphacode/lib/python3.9/site-packages/httpx/_client.py:1996> exception=RuntimeError('Event loop is closed')>
Traceback (most recent call last):
File "/data1/wenlinyao/miniconda3/envs/alphacode/lib/python3.9/site-packages/httpx/_client.py", line 2006, in aclose
await proxy.aclose()
File "/data1/wenlinyao/miniconda3/envs/alphacode/lib/python3.9/site-packages/httpx/_transports/default.py", line 383, in aclose
await self._pool.aclose()
File "/data1/wenlinyao/miniconda3/envs/alphacode/lib/python3.9/site-packages/httpcore/_async/connection_pool.py", line 324, in aclose
await connection.aclose()
File "/data1/wenlinyao/miniconda3/envs/alphacode/lib/python3.9/site-packages/httpcore/_async/http_proxy.py", line 350, in aclose
await self._connection.aclose()
File "/data1/wenlinyao/miniconda3/envs/alphacode/lib/python3.9/site-packages/httpcore/_async/http11.py", line 253, in aclose
await self._network_stream.aclose()
File "/data1/wenlinyao/miniconda3/envs/alphacode/lib/python3.9/site-packages/httpcore/_backends/anyio.py", line 54, in aclose
await self._stream.aclose()
File "/data1/wenlinyao/miniconda3/envs/alphacode/lib/python3.9/site-packages/anyio/streams/tls.py", line 202, in aclose
await self.transport_stream.aclose()
File "/data1/wenlinyao/miniconda3/envs/alphacode/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 1181, in aclose
self._transport.close()
File "/data1/wenlinyao/miniconda3/envs/alphacode/lib/python3.9/asyncio/selector_events.py", line 698, in close
self._loop.call_soon(self._call_connection_lost, None)
File "/data1/wenlinyao/miniconda3/envs/alphacode/lib/python3.9/asyncio/base_events.py", line 751, in call_soon
self._check_closed()
File "/data1/wenlinyao/miniconda3/envs/alphacode/lib/python3.9/asyncio/base_events.py", line 515, in _check_closed
raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed
2024-01-29 23:30:06.222 | INFO | alpha_codium.llm.ai_handler:chat_completion:133 - done
2024-01-29 23:30:06.222 | INFO | alpha_codium.llm.ai_handler:chat_completion:134 - -----------------
2024-01-29 23:30:06.290 | INFO | alpha_codium.code_contests.eval.code_test_runners:print_test_results:111 - test-0 :: status=ProgramStatus.kSuccess, passed=False
2024-01-29 23:30:06.290 | INFO | alpha_codium.code_contests.eval.code_test_runners:print_test_results:116 - =====================================================================
2024-01-29 23:30:06.290 | INFO | alpha_codium.gen.stages.run_initial_code_generation:run_initial_code_generation:61 - Failed to pass tests after 3 attempts, d: 3.0, best_d so far: 5
2024-01-29 23:30:06.291 | INFO | alpha_codium.gen.stages.run_initial_solve:run_initial_solve:14 - --initial solve stage--
2024-01-29 23:30:06.297 | INFO | alpha_codium.llm.ai_handler:chat_completion:86 - -----------------
2024-01-29 23:30:06.297 | INFO | alpha_codium.llm.ai_handler:chat_completion:87 - Running inference ...
2024-01-29 23:30:12.458 | INFO | alpha_codium.llm.ai_handler:chat_completion:133 - done
2024-01-29 23:30:12.458 | INFO | alpha_codium.llm.ai_handler:chat_completion:134 - -----------------
2024-01-29 23:30:12.524 | INFO | alpha_codium.code_contests.eval.code_test_runners:print_test_results:111 - test-0 :: status=ProgramStatus.kFailed, passed=False
2024-01-29 23:30:12.525 | INFO | alpha_codium.code_contests.eval.code_test_runners:print_test_results:116 - =====================================================================
2024-01-29 23:30:12.525 | ERROR | alpha_codium.gen.stages.run_tests:run_tests:36 - failed to run solution
2024-01-29 23:30:12.525 | INFO | alpha_codium.gen.stages.run_initial_code_generation:run_initial_code_generation:61 - Failed to pass tests after 4 attempts, d: 5, best_d so far: 3.0
2024-01-29 23:30:12.526 | INFO | alpha_codium.gen.stages.run_initial_solve:run_initial_solve:14 - --initial solve stage--
2024-01-29 23:30:12.531 | INFO | alpha_codium.llm.ai_handler:chat_completion:86 - -----------------
2024-01-29 23:30:12.532 | INFO | alpha_codium.llm.ai_handler:chat_completion:87 - Running inference ...
2024-01-29 23:30:19.458 | INFO | alpha_codium.llm.ai_handler:chat_completion:133 - done
2024-01-29 23:30:19.458 | INFO | alpha_codium.llm.ai_handler:chat_completion:134 - -----------------

Another interesting thing I found is that when I run gpt-3.5-turbo-0613 on the valid split (iterations=1) twice, the total amount of data instances after evaluation is quite different. One is 9+87=96 instances and another one is 12+ 80=92. I checked the total amount should be 177? Is this due to the above error?

First time run:
problem 8 passed all tests
problem 17 passed all tests
problem 27 passed all tests
problem 31 passed all tests
problem 60 passed all tests
problem 66 passed all tests
problem 73 passed all tests
problem 84 passed all tests
problem 96 passed all tests
total_passed: 9, total_failed: 87
pass rate: 0.09375

Second time run:
problem 4 passed all tests
problem 8 passed all tests
problem 16 passed all tests
problem 17 passed all tests
problem 34 passed all tests
problem 50 passed all tests
problem 60 passed all tests
problem 66 passed all tests
problem 73 passed all tests
problem 83 passed all tests
problem 96 passed all tests
problem 114 passed all tests
total_passed: 12, total_failed: 80
pass rate: 0.13043478260869565

@mrT23
Copy link
Contributor

mrT23 commented Jan 30, 2024 •
edited
Loading

  • The validation set contains 107 examples. i think 3 or 4 are invalid, because the provided generated tests contain illegal input. so you generated responses for 92 out of ~104 valid problems

  • i think that if you ran the script again, with the same file, it will only try to solve problems it skipped (make sure to back up the file).

  • i am not sure why you see the async error. its weird. the log doesn't report a problem in the run itself. maybe some memory issue ? make sure you run with 'sandbox=true'. or try with a machine with larger memory.

  • for 1 or 2 iteration, 13% is a reasonable solve ratio. for more iterations, think of strategies how to create some randomness between each iteration, to increase the chance of pass@5 solve. (hint - try on 1-2 iterations the 'use_direct_solutions' flag. it is somewhat orthogonal, and helps with some problems)

@wenlinyao
Copy link
Author

Thanks again! that is very helpful!

  • I double-checked the validation set. It should be 117 original instances? Two of my "5 iterations" run returns "total_passed + total_failed = 105", so I guess the total valid problems is 105? Just want to make sure my experiment setting is correct. ;)
  • It seems the "async error" can be solved using the original OpenAI API call instead of litellm.
  • That is a good point. But I see 'use_direct_solutions' flag is defined in the 'configuration.toml' file. So how can I run 5 iterations with a partial of them applying 'use_direct_solutions=true'?

@mrT23
Copy link
Contributor

mrT23 commented Feb 1, 2024

"So how can I run 5 iterations with a partial of them applying 'use_direct_solutions=true'?"

edit the file https://github.com/Codium-ai/AlphaCodium/blob/main/alpha_codium/gen/dataset_solver.py#L75

if iteration <2 :
...
else:
...

@huanhuan6666
Copy link

@wenlinyao
Hello, I've encountered a similar issue and would greatly appreciate your guidance:

ERROR:asyncio:Task exception was never retrieved
future: <Task finished name='Task-479' coro=<AsyncClient.aclose() done, defined at /data/hzhang/.conda/envs/alpha/lib/python3.10/site-packages/httpx/_client.py:1996> exception=RuntimeError('Event loop is closed')>
Traceback (most recent call last):
  File "/data/hzhang/.conda/envs/alpha/lib/python3.10/site-packages/httpx/_client.py", line 2006, in aclose
    await proxy.aclose()
  File "/data/hzhang/.conda/envs/alpha/lib/python3.10/site-packages/httpx/_transports/default.py", line 383, in aclose
    await self._pool.aclose()
  File "/data/hzhang/.conda/envs/alpha/lib/python3.10/site-packages/httpcore/_async/connection_pool.py", line 324, in aclose
    await connection.aclose()
  File "/data/hzhang/.conda/envs/alpha/lib/python3.10/site-packages/httpcore/_async/http_proxy.py", line 350, in aclose
    await self._connection.aclose()

Could you please provide detailed guidance on how to address this issue? I'm eager to learn and make the necessary modifications.🥰

@wenlinyao
Copy link
Author

@huanhuan6666 I just saw your message. I am not sure how to solve this issue thoroughly. But you can try to replace litellm LLM call with the OpenAI official API call.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@mrT23 @wenlinyao @huanhuan6666 and others