KFP charm integration tests fail when deploying `mysql-k8s` #316

phoevos · 2023-09-04T08:28:14Z

Bug Description

Deploying mysql-k8s as part of our charm-specific integration tests (i.e. kfp-api and kfp-persistence) fails both locally and on the GH runners. The following error is produced by python-libjuju:

TypeError: unhashable type: 'dict'

Looking more carefully into the logs (attached below), it seems that the error for both of the charms is raised when attempting to deploy mysql-k8s from 8.0/edge.

To Reproduce

In order to reproduce you can run the integration tests of the kfp-api or the kfp-persistence charms, e.g.

cd charms/kfp-api
tox -e integration

Environment

This affects the main branch, which deploys mysql-k8s from 8.0/edge on juju (i.e. agent and python library) 2.9.44.

Relevant Log Output

_______________________________________________________ TestCharm.test_relational_db_relation_with_mysql_relation _______________________________________________________
Traceback (most recent call last):
  File "/home/phoevos/canonical/kubeflow/pipelines/charms/kfp-api/.tox/integration/lib/python3.8/site-packages/_pytest/runner.py", line 341, in from_call
    result: Optional[TResult] = func()
  File "/home/phoevos/canonical/kubeflow/pipelines/charms/kfp-api/.tox/integration/lib/python3.8/site-packages/_pytest/runner.py", line 262, in <lambda>
    lambda: ihook(item=item, **kwds), when=when, reraise=reraise
  File "/home/phoevos/canonical/kubeflow/pipelines/charms/kfp-api/.tox/integration/lib/python3.8/site-packages/pluggy/_hooks.py", line 433, in __call__
    return self._hookexec(self.name, self._hookimpls, kwargs, firstresult)
  File "/home/phoevos/canonical/kubeflow/pipelines/charms/kfp-api/.tox/integration/lib/python3.8/site-packages/pluggy/_manager.py", line 112, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "/home/phoevos/canonical/kubeflow/pipelines/charms/kfp-api/.tox/integration/lib/python3.8/site-packages/pluggy/_callers.py", line 155, in _multicall
    return outcome.get_result()
  File "/home/phoevos/canonical/kubeflow/pipelines/charms/kfp-api/.tox/integration/lib/python3.8/site-packages/pluggy/_result.py", line 108, in get_result
    raise exc.with_traceback(exc.__traceback__)
  File "/home/phoevos/canonical/kubeflow/pipelines/charms/kfp-api/.tox/integration/lib/python3.8/site-packages/pluggy/_callers.py", line 80, in _multicall
    res = hook_impl.function(*args)
  File "/home/phoevos/canonical/kubeflow/pipelines/charms/kfp-api/.tox/integration/lib/python3.8/site-packages/_pytest/runner.py", line 177, in pytest_runtest_call
    raise e
  File "/home/phoevos/canonical/kubeflow/pipelines/charms/kfp-api/.tox/integration/lib/python3.8/site-packages/_pytest/runner.py", line 169, in pytest_runtest_call
    item.runtest()
  File "/home/phoevos/canonical/kubeflow/pipelines/charms/kfp-api/.tox/integration/lib/python3.8/site-packages/_pytest/python.py", line 1788, in runtest
    self.ihook.pytest_pyfunc_call(pyfuncitem=self)
  File "/home/phoevos/canonical/kubeflow/pipelines/charms/kfp-api/.tox/integration/lib/python3.8/site-packages/pluggy/_hooks.py", line 433, in __call__
    return self._hookexec(self.name, self._hookimpls, kwargs, firstresult)
  File "/home/phoevos/canonical/kubeflow/pipelines/charms/kfp-api/.tox/integration/lib/python3.8/site-packages/pluggy/_manager.py", line 112, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "/home/phoevos/canonical/kubeflow/pipelines/charms/kfp-api/.tox/integration/lib/python3.8/site-packages/pluggy/_callers.py", line 155, in _multicall
    return outcome.get_result()
  File "/home/phoevos/canonical/kubeflow/pipelines/charms/kfp-api/.tox/integration/lib/python3.8/site-packages/pluggy/_result.py", line 108, in get_result
    raise exc.with_traceback(exc.__traceback__)
  File "/home/phoevos/canonical/kubeflow/pipelines/charms/kfp-api/.tox/integration/lib/python3.8/site-packages/pluggy/_callers.py", line 80, in _multicall
    res = hook_impl.function(*args)
  File "/home/phoevos/canonical/kubeflow/pipelines/charms/kfp-api/.tox/integration/lib/python3.8/site-packages/_pytest/python.py", line 194, in pytest_pyfunc_call
    result = testfunction(**testargs)
  File "/home/phoevos/canonical/kubeflow/pipelines/charms/kfp-api/.tox/integration/lib/python3.8/site-packages/pytest_asyncio/plugin.py", line 532, in inner
    _loop.run_until_complete(task)
  File "/home/phoevos/.pyenv/versions/3.8.16/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/home/phoevos/canonical/kubeflow/pipelines/charms/kfp-api/tests/integration/test_charm.py", line 80, in test_relational_db_relation_with_mysql_relation
    await ops_test.model.deploy(
  File "/home/phoevos/canonical/kubeflow/pipelines/charms/kfp-api/.tox/integration/lib/python3.8/site-packages/juju/model.py", line 1686, in deploy
    resources = await self._add_charmhub_resources(res.app_name,
  File "/home/phoevos/canonical/kubeflow/pipelines/charms/kfp-api/.tox/integration/lib/python3.8/site-packages/juju/model.py", line 1819, in _add_charmhub_resources
    res = await charm_facade.CharmInfo(entity_url)
  File "/home/phoevos/canonical/kubeflow/pipelines/charms/kfp-api/.tox/integration/lib/python3.8/site-packages/juju/client/facade.py", line 502, in wrapper
    result = cls.from_json(reply['response'])
  File "/home/phoevos/canonical/kubeflow/pipelines/charms/kfp-api/.tox/integration/lib/python3.8/site-packages/juju/client/facade.py", line 673, in from_json
    return cls(**d)
  File "/home/phoevos/canonical/kubeflow/pipelines/charms/kfp-api/.tox/integration/lib/python3.8/site-packages/juju/client/_definitions.py", line 4962, in __init__
    meta_ = CharmMeta.from_json(meta) if meta else None
  File "/home/phoevos/canonical/kubeflow/pipelines/charms/kfp-api/.tox/integration/lib/python3.8/site-packages/juju/client/facade.py", line 673, in from_json
    return cls(**d)
  File "/home/phoevos/canonical/kubeflow/pipelines/charms/kfp-api/.tox/integration/lib/python3.8/site-packages/juju/client/_definitions.py", line 5474, in __init__
    assumes_expr_ = ExpressionTree.from_json(assumes_expr) if assumes_expr else None
  File "/home/phoevos/canonical/kubeflow/pipelines/charms/kfp-api/.tox/integration/lib/python3.8/site-packages/juju/client/facade.py", line 687, in from_json
    cls.splitEntries(i, d)
  File "/home/phoevos/canonical/kubeflow/pipelines/charms/kfp-api/.tox/integration/lib/python3.8/site-packages/juju/client/facade.py", line 713, in splitEntries
    result_dict[entry] = ''
TypeError: unhashable type: 'dict'

Additional Context

Given that we haven't made any changes to the affected charms recently, we assume that it's an issue with mysql-k8s. More specifically, we can see the integration tests running successfully 2 weeks ago (last commit in main): https://github.com/canonical/kfp-operators/actions/runs/5968203791

Inspecting the above run logs we see it deploys mysql-k8s rev95. Since then, there have been 4 new revisions on 8.0/edge with the latest (i.e. 99) coming out 3 days ago. At the moment, it is unclear to me which revision could have introduced the change affecting us here.

Update

Our updated understanding is summarised here.

Relevant Issue:

Support nested assumes juju/python-libjuju#938

The text was updated successfully, but these errors were encountered:

phoevos · 2023-09-04T09:46:53Z

Update

Deploying directly (same way we do in our tests) using the Juju CLI works:

juju deploy mysql-k8s --config profile=testing --channel 8.0/edge --revision 99 --trust kfp-db

The above also works from inside our tests, even with the latest revision:

diff --git a/charms/kfp-persistence/tests/integration/test_charm.py b/charms/kfp-persistence/tests/integration/test_charm.py
index 65cbf40..8f61ac9 100644
--- a/charms/kfp-persistence/tests/integration/test_charm.py
+++ b/charms/kfp-persistence/tests/integration/test_charm.py
@@ -36,12 +36,18 @@ class TestCharm:
             trust=True,
         )
 
-        await ops_test.model.deploy(
-            entity_url="mysql-k8s",
-            application_name="kfp-db",
-            config={"profile": "testing"},
-            channel="8.0/edge",
-            trust=True,
+        await ops_test.juju(
+            "deploy",
+            "mysql-k8s",
+            "--config",
+            "profile=testing",
+            "--channel",
+            "8.0/edge",
+            "--revision",
+            "99",
+            "--trust",
+            "kfp-db",
+            check=True,
         )
 
         await ops_test.model.deploy(

phoevos · 2023-09-04T10:36:24Z

This looks like something introduced in revision 96 that exposes a libjuju issue.
More specifically, revision 96 introduced these assumptions in metadata.yaml.

These metadata entries are parsed by libjuju here (it is this line that raises the unhashable field error).

It seems that nested assumes are introduced by this PR, tackling the linked issue:

@cderici, when can we expect the above to be included in a patch release?

Upgrade python-libjuju to 2.9.44.1 which fixes issues in our integration tests that deploy `mysql-k8s` from `8.0/edge`. Closes #316 Signed-off-by: Phoevos Kalemkeris <[email protected]>

phoevos added bug Something isn't working 23.10 Should be fixed by 23.10 labels Sep 4, 2023

phoevos self-assigned this Sep 4, 2023

This was referenced Sep 4, 2023

feat: get images on main #312

Merged

fix: Patch KFP Profile Controller service port #318

Merged

fix: Bump libjuju to 2.9.44.1 #319

Merged

phoevos closed this as completed in #319 Sep 7, 2023

phoevos added a commit that referenced this issue Sep 7, 2023

fix: Bump libjuju to 2.9.44.1 (#319)

b6b0931

Upgrade python-libjuju to 2.9.44.1 which fixes issues in our integration tests that deploy `mysql-k8s` from `8.0/edge`. Closes #316 Signed-off-by: Phoevos Kalemkeris <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

KFP charm integration tests fail when deploying `mysql-k8s` #316

KFP charm integration tests fail when deploying `mysql-k8s` #316

phoevos commented Sep 4, 2023 •

edited

Loading

phoevos commented Sep 4, 2023

phoevos commented Sep 4, 2023

KFP charm integration tests fail when deploying mysql-k8s #316

KFP charm integration tests fail when deploying mysql-k8s #316

Comments

phoevos commented Sep 4, 2023 • edited Loading

Bug Description

To Reproduce

Environment

Relevant Log Output

Additional Context

Update

phoevos commented Sep 4, 2023

Update

phoevos commented Sep 4, 2023

KFP charm integration tests fail when deploying `mysql-k8s` #316

KFP charm integration tests fail when deploying `mysql-k8s` #316

phoevos commented Sep 4, 2023 •

edited

Loading