Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: Broken ComputeContainer GQL #3042

Merged
merged 13 commits into from
Nov 12, 2024
Merged

Conversation

jopemachine
Copy link
Member

@jopemachine jopemachine commented Nov 7, 2024

Currently, when using ComputeContainer GQL, the following error occurs.

2024-11-07 13:57:19.382 ERROR ai.backend.manager.api.admin [4278] ADMIN.GQL Exception: {'message': "greenlet_spawn has not been called; can't call await_only() here. Was IO attempted in an unexpected place? (Background on this error at: https://sqlalche.me/e/14/xd2s)", 'locations': [{'line': 2, 'column': 3}], 'path': ['compute_container']}
2024-11-07 13:57:19.397 DEBUG ai.backend.manager.api.admin [4278] Traceback (most recent call last):
  File "/Users/jopemachine/Desktop/backend.ai/dist/export/python/virtualenvs/python-default/3.12.6/lib/python3.12/site-packages/graphql/execution/execute.py", line 530, in await_result
    return_type, field_nodes, info, path, await result
                                          ^^^^^^^^^^^^
  File "/Users/jopemachine/Desktop/backend.ai/src/ai/backend/manager/models/base.py", line 1060, in wrapped
    return await resolve_func(root, info, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jopemachine/Desktop/backend.ai/src/ai/backend/manager/models/gql.py", line 2164, in resolve_compute_container
    return await loader.load(id)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jopemachine/Desktop/backend.ai/dist/export/python/virtualenvs/python-default/3.12.6/lib/python3.12/site-packages/aiodataloader.py", line 215, in dispatch_queue_batch
    values = await batch_future
             ^^^^^^^^^^^^^^^^^^
  File "/Users/jopemachine/Desktop/aiotools/src/aiotools/func.py", line 23, in wrapped
    return await coro(*args, *cargs, **kwargs, **ckwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jopemachine/Desktop/backend.ai/src/ai/backend/manager/models/kernel.py", line 1125, in batch_load_detail
    return await batch_result(
           ^^^^^^^^^^^^^^^^^^^
  File "/Users/jopemachine/Desktop/backend.ai/src/ai/backend/manager/models/base.py", line 871, in batch_result
    objs_per_key[key_getter(row)] = obj_type.from_row(graph_ctx, row)
                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jopemachine/Desktop/backend.ai/src/ai/backend/manager/models/kernel.py", line 949, in from_row
    props = cls.parse_row(ctx, row)
            ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jopemachine/Desktop/backend.ai/src/ai/backend/manager/models/kernel.py", line 922, in parse_row
    "image_object": ImageNode.from_row(row.image_row),
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jopemachine/Desktop/backend.ai/src/ai/backend/manager/models/gql_models/image.py", line 367, in from_row
    aliases=[alias_row.alias for alias_row in row.aliases],
                                              ^^^^^^^^^^^
  File "/Users/jopemachine/Desktop/backend.ai/dist/export/python/virtualenvs/python-default/3.12.6/lib/python3.12/site-packages/sqlalchemy/orm/attributes.py", line 487, in __get__
    return self.impl.get(state, dict_)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jopemachine/Desktop/backend.ai/dist/export/python/virtualenvs/python-default/3.12.6/lib/python3.12/site-packages/sqlalchemy/orm/attributes.py", line 959, in get
    value = self._fire_loader_callables(state, key, passive)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jopemachine/Desktop/backend.ai/dist/export/python/virtualenvs/python-default/3.12.6/lib/python3.12/site-packages/sqlalchemy/orm/attributes.py", line 995, in _fire_loader_callables
    return self.callable_(state, passive)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jopemachine/Desktop/backend.ai/dist/export/python/virtualenvs/python-default/3.12.6/lib/python3.12/site-packages/sqlalchemy/orm/strategies.py", line 912, in _load_for_state
    return self._emit_lazyload(
           ^^^^^^^^^^^^^^^^^^^^
  File "/Users/jopemachine/Desktop/backend.ai/dist/export/python/virtualenvs/python-default/3.12.6/lib/python3.12/site-packages/sqlalchemy/orm/strategies.py", line 1046, in _emit_lazyload
    result = session.execute(
             ^^^^^^^^^^^^^^^^
  File "/Users/jopemachine/Desktop/backend.ai/dist/export/python/virtualenvs/python-default/3.12.6/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 1717, in execute
    result = conn._execute_20(statement, params or {}, execution_options)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jopemachine/Desktop/backend.ai/dist/export/python/virtualenvs/python-default/3.12.6/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1710, in _execute_20
    return meth(self, args_10style, kwargs_10style, execution_options)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jopemachine/Desktop/backend.ai/dist/export/python/virtualenvs/python-default/3.12.6/lib/python3.12/site-packages/sqlalchemy/sql/elements.py", line 334, in _execute_on_connection
    return connection._execute_clauseelement(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jopemachine/Desktop/backend.ai/dist/export/python/virtualenvs/python-default/3.12.6/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1577, in _execute_clauseelement
    ret = self._execute_context(
          ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jopemachine/Desktop/backend.ai/dist/export/python/virtualenvs/python-default/3.12.6/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1953, in _execute_context
    self._handle_dbapi_exception(
  File "/Users/jopemachine/Desktop/backend.ai/dist/export/python/virtualenvs/python-default/3.12.6/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 2138, in _handle_dbapi_exception
    util.raise_(exc_info[1], with_traceback=exc_info[2])
  File "/Users/jopemachine/Desktop/backend.ai/dist/export/python/virtualenvs/python-default/3.12.6/lib/python3.12/site-packages/sqlalchemy/util/compat.py", line 211, in raise_
    raise exception
  File "/Users/jopemachine/Desktop/backend.ai/dist/export/python/virtualenvs/python-default/3.12.6/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1910, in _execute_context
    self.dialect.do_execute(
  File "/Users/jopemachine/Desktop/backend.ai/dist/export/python/virtualenvs/python-default/3.12.6/lib/python3.12/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute
    cursor.execute(statement, parameters)
  File "/Users/jopemachine/Desktop/backend.ai/dist/export/python/virtualenvs/python-default/3.12.6/lib/python3.12/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 479, in execute
    self._adapt_connection.await_(
  File "/Users/jopemachine/Desktop/backend.ai/dist/export/python/virtualenvs/python-default/3.12.6/lib/python3.12/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 61, in await_only
    raise exc.MissingGreenlet(
graphql.error.graphql_error.GraphQLError: greenlet_spawn has not been called; can't call await_only() here. Was IO attempted in an unexpected place? (Background on this error at: https://sqlalche.me/e/14/xd2s)

GraphQL request:2:3
1 | query($id: UUID!) {
2 |   compute_container (id:$id) {
  |   ^
3 |     session_id

This PR fixes this bug by removing part of the optimization in ComputeContainer.batch_load_detail() that retrieves KernelRow.

Checklist: (if applicable)

  • Milestone metadata specifying the target backport version

@jopemachine jopemachine self-assigned this Nov 7, 2024
@github-actions github-actions bot added comp:manager Related to Manager component size:XS ~10 LoC labels Nov 7, 2024
@jopemachine jopemachine added type:bug Reports about that are not working and removed comp:manager Related to Manager component size:XS ~10 LoC labels Nov 7, 2024
@jopemachine jopemachine added this to the 23.09 milestone Nov 7, 2024
@jopemachine jopemachine added comp:manager Related to Manager component size:XS ~10 LoC labels Nov 7, 2024
@jopemachine jopemachine marked this pull request as ready for review November 7, 2024 05:26
Copy link
Member

@fregataa fregataa left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This cannot resolve query like below

query ComputeContainer {
    compute_container (
        id: "KERNEL_ID"
    ){
        id
        image_object {
            id
            name
        }
    }
}

@jopemachine
Copy link
Member Author

This cannot resolve query like below

query ComputeContainer {
    compute_container (
        id: "KERNEL_ID"
    ){
        id
        image_object {
            id
            name
        }
    }
}

Resolved in c7c7707

@jopemachine jopemachine added action:on hold Hold it. Wait for the restart. and removed action:on hold Hold it. Wait for the restart. labels Nov 9, 2024
@achimnol achimnol added this pull request to the merge queue Nov 12, 2024
Merged via the queue into main with commit c3b2ed8 Nov 12, 2024
23 checks passed
@achimnol achimnol deleted the fix/compute-container-gql-not-working branch November 12, 2024 00:11
lablup-octodog pushed a commit that referenced this pull request Nov 12, 2024
Co-authored-by: Joongi Kim <[email protected]>
Backported-from: main (24.12)
Backported-to: 24.09
Backport-of: 3042
lablup-octodog pushed a commit that referenced this pull request Nov 12, 2024
Co-authored-by: Joongi Kim <[email protected]>
Backported-from: main (24.12)
Backported-to: 24.03
Backport-of: 3042
lablup-octodog pushed a commit that referenced this pull request Nov 12, 2024
Co-authored-by: Joongi Kim <[email protected]>
Backported-from: main (24.12)
Backported-to: 23.09
Backport-of: 3042
github-merge-queue bot pushed a commit that referenced this pull request Nov 12, 2024
Co-authored-by: Gyubong Lee <[email protected]>
Co-authored-by: Joongi Kim <[email protected]>
github-merge-queue bot pushed a commit that referenced this pull request Nov 12, 2024
Co-authored-by: Gyubong Lee <[email protected]>
Co-authored-by: Joongi Kim <[email protected]>
achimnol added a commit that referenced this pull request Nov 17, 2024
Co-authored-by: Joongi Kim <[email protected]>
Backported-from: main (24.12)
Backported-to: 23.09
Backport-of: 3042
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
comp:manager Related to Manager component size:XS ~10 LoC type:bug Reports about that are not working
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants