Storage: Show correct instance root disk size on API #14511

hamistao · 2024-11-22T13:42:25Z

This aims to get instance root disk sizes correctly, as it currently often shows as 0 or null on the API, either though /1.0/instances?recursion=2 or /1.0/instances/<instance_name>/state, as reported by #14277.

This is still not a perfect solution but is a significant improvement as it:

Fixes not returning the disk total size when getting the disk usage is not supported by the driver (relevant for dir and lvm)
Returns default block size for block typed/based instances when the disk device does not have the size key set.
Returns disk size -1 for instances that have unbounded storage and have access to the entire pool storage.

hamistao · 2024-11-27T00:43:49Z

There are two situations that aren't fixed by this PR:

Since we do not yet support live resizing of block volumes, setting the disk size key for a VM or a container on a block based driver with the instance running will only take effect after the instance is restarted. But the API will show the size specified on the size key from the moment it is set, thus being inconsistent until instance restart. This will be partly mitigated by Storage: Add support for online disk growing of zfs and lvm block volumes (from Incus) #14211. Depending on how our plans for live resizing develop in the near future, we can consider developing more intricate solutions to prevent showing incorrect disk sizes in these scenarios.
When the instance size was influenced by volume.size on the pool level. Refer to '/1.0/instances?recursion=2' Endpoint has missing information. #14277 (comment) for more information on this limitation.

roosterfish

There seems to be one more usage of DefaultBlockSize in the instance scriptlet https://github.com/canonical/lxd/blob/main/lxd/scriptlet/instance_placement.go#L224.

Can we do something similar to https://github.com/canonical/lxd/blob/main/lxd/project/limits/permissions.go#L1472 to load the storage driver and fetch the respective DefaultBlockSize? Not sure if it will cause an import loop though.

lxd/storage/backend_lxd.go

test/suites/storage.sh

shared/api/instance_state.go

doc/rest-api.yaml

hamistao · 2024-11-28T14:19:10Z

@roosterfish I believe everything was addressed. I did something different from what is done on https://github.com/canonical/lxd/blob/main/lxd/project/limits/permissions.go#L1472 to get DefaultBlockSize on InstancePlacementRun, but I think you will like this approach.

Tests for VMs will follow on LXD CI.

hamistao · 2024-11-29T18:51:38Z

Going back to Draft to add support for the situations described in #14511 (comment).

hamistao · 2024-12-06T08:34:08Z

This includes the simpler fixes related to showing accurate disk sizes on the API. To address the use cases described in #14511 (comment), a follow up draft PR was opened to unblock these changes from getting merged: #14511 (comment).

hamistao · 2024-12-09T07:21:56Z

@tomponline This includes the simpler fixes for getting correct root disk sizes and don't address the scenarios described in #14511 (comment). These are addressed in a follow up PR: #14595.

The tests for these changes are included in the follow up PR, but we can check that they are passing on the follow up's test runs.

All that said, this is ready for a review :D

shared/api/instance_state.go

lxc/info.go

This includes some improvements that were originally a part of the fix implemented in #14511, but ended up not being needed for that fix. They are still nice to have and thus were included here.

When driver does not support getting the volume usage, we are returning prematurely and abstaining from getting the disk total size. Signed-off-by: hamistao <[email protected]>

This proposes using -1 as a disk size to indicate we do not impose limits on the volume level and thus it has access to the entire pool storage. Signed-off-by: hamistao <[email protected]>

Signed-off-by: hamistao <[email protected]>

Previously, when usage was not supported, we returned null root disk and thus did not show disk total size, even when it was possible. Now we represent not supporting showing disk usage with usage=-1. Signed-off-by: hamistao <[email protected]>

Signed-off-by: hamistao <[email protected]>

tomponline

Thanks!

tomponline · 2024-12-11T09:48:39Z

@edlerd please can you see whether this helps with storage reporting in the UI. Thanks

fixed

github-actions bot added the Documentation Documentation needs updating label Nov 22, 2024

hamistao force-pushed the instance_metrics_on_list branch from 752fbf9 to 0ad2e26 Compare November 22, 2024 14:04

github-actions bot removed the Documentation Documentation needs updating label Nov 22, 2024

hamistao force-pushed the instance_metrics_on_list branch from 0ad2e26 to 760adc8 Compare November 26, 2024 03:37

github-actions bot added Documentation Documentation needs updating API Changes to the REST API labels Nov 26, 2024

hamistao force-pushed the instance_metrics_on_list branch 2 times, most recently from 0a5d38d to 4ccdd4c Compare November 26, 2024 13:12

hamistao changed the title ~~Show instance root disk size correctly on instance list~~ lxd: Show correct instance root disk size on API Nov 26, 2024

hamistao force-pushed the instance_metrics_on_list branch 3 times, most recently from f3a8f96 to 47d7fe1 Compare November 27, 2024 00:39

hamistao marked this pull request as ready for review November 27, 2024 02:20

hamistao requested review from roosterfish and tomponline November 27, 2024 02:20

roosterfish previously requested changes Nov 28, 2024

View reviewed changes

lxd/storage/backend_lxd.go Show resolved Hide resolved

test/suites/storage.sh Outdated Show resolved Hide resolved

shared/api/instance_state.go Outdated Show resolved Hide resolved

doc/rest-api.yaml Outdated Show resolved Hide resolved

hamistao force-pushed the instance_metrics_on_list branch 2 times, most recently from aed4e82 to 6d75211 Compare November 28, 2024 13:19

hamistao marked this pull request as draft November 29, 2024 18:50

hamistao force-pushed the instance_metrics_on_list branch 7 times, most recently from 61cba04 to e422622 Compare December 5, 2024 22:59

hamistao mentioned this pull request Dec 6, 2024

Storage: Define per-pool default block size #14594

Merged

hamistao force-pushed the instance_metrics_on_list branch 2 times, most recently from 7db2378 to b964ab2 Compare December 6, 2024 07:30

hamistao mentioned this pull request Dec 6, 2024

Storage: Show correct disk size #14595

Draft

hamistao force-pushed the instance_metrics_on_list branch from b964ab2 to c239c75 Compare December 9, 2024 06:19

tomponline reviewed Dec 9, 2024

View reviewed changes

shared/api/instance_state.go Outdated Show resolved Hide resolved

tomponline reviewed Dec 9, 2024

View reviewed changes

lxc/info.go Show resolved Hide resolved

hamistao marked this pull request as ready for review December 9, 2024 13:38

hamistao added 8 commits December 9, 2024 13:06

lxd/storage: Get disk total size regardless of previous error

9ffc5d8

When driver does not support getting the volume usage, we are returning prematurely and abstaining from getting the disk total size. Signed-off-by: hamistao <[email protected]>

lxd/storage: Return -1 disk size for containers without size limits

26b8d73

This proposes using -1 as a disk size to indicate we do not impose limits on the volume level and thus it has access to the entire pool storage. Signed-off-by: hamistao <[email protected]>

shared/api: Update instance state comments

164a78f

Signed-off-by: hamistao <[email protected]>

doc: make update-api

87823c8

Signed-off-by: hamistao <[email protected]>

lxc/info: Add disk total to instance info

2b64a00

Signed-off-by: hamistao <[email protected]>

i18n: Update translation templates.

bb584df

Signed-off-by: hamistao <[email protected]>

i18n: Update translations.

4f17d26

Signed-off-by: hamistao <[email protected]>

hamistao force-pushed the instance_metrics_on_list branch from c239c75 to 4f17d26 Compare December 9, 2024 16:07

hamistao mentioned this pull request Dec 10, 2024

VM disk size tests canonical/lxd-ci#367

Draft

tomponline approved these changes Dec 11, 2024

View reviewed changes

tomponline merged commit d56d152 into canonical:main Dec 11, 2024
27 checks passed

hamistao deleted the instance_metrics_on_list branch December 11, 2024 18:43

tomponline changed the title ~~lxd: Show correct instance root disk size on API~~ Storage: Show correct instance root disk size on API Dec 12, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Storage: Show correct instance root disk size on API #14511

Storage: Show correct instance root disk size on API #14511

hamistao commented Nov 22, 2024 •

edited

Loading

hamistao commented Nov 27, 2024

roosterfish left a comment

hamistao commented Nov 28, 2024

hamistao commented Nov 29, 2024

hamistao commented Dec 6, 2024

hamistao commented Dec 9, 2024

tomponline left a comment

tomponline commented Dec 11, 2024

Storage: Show correct instance root disk size on API #14511

Storage: Show correct instance root disk size on API #14511

Conversation

hamistao commented Nov 22, 2024 • edited Loading

hamistao commented Nov 27, 2024

roosterfish left a comment

Choose a reason for hiding this comment

hamistao commented Nov 28, 2024

hamistao commented Nov 29, 2024

hamistao commented Dec 6, 2024

hamistao commented Dec 9, 2024

tomponline left a comment

Choose a reason for hiding this comment

tomponline commented Dec 11, 2024

hamistao commented Nov 22, 2024 •

edited

Loading