Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Flatten & unify component structure #961

Merged
merged 2 commits into from
Oct 8, 2024
Merged

Conversation

sharnoff
Copy link
Member

@sharnoff sharnoff commented Jun 13, 2024

This basically encompasses two logical changes:

  1. More fully integrating neonvm into the rest of the repository; and
  2. Using a consistent directory structure for all components

The new structure is:

  • All components get their own top-level directory (e.g. autoscaler-agent, neonvm-controller, vm-builder)
  • Inside the directory, there is:
    • cmd/ - directory containing main.go for the component
    • Dockerfile - for building the component
    • *.yaml - various YAML files for deploying, if applicable

Additionally, all code outside the cmd package are moved into the pkg/ subdirectory. In particular, there is now pkg/neonvm/.

Also of note: neonvm/hack/kernel has been relocated to the new top-level directory neonvm-kernel/.

Under this structure, the user-visible changes are:

  1. Introduction of new YAML: neonvm-controller.yaml
  2. Introduction of new YAML: neonvm-vxlan-controller.yaml
  3. neonvm.yaml now is mostly just the CRD.

The primary reasons for this change are:

  1. Because it's been long enough since we moved the neonvm repository here that we might as well make it more well-integrated.
  2. Because as we look to potentially add components in other languages, we need a directory structure that supports it.

Notes for review: Splitting the YAMLs will be somewhat tricky to manage safely. I think we should have a clear idea of how we'll do that before merging.

If we go with this approach, we should aim to cut a release soon after merging.

@sharnoff sharnoff requested a review from Omrigan June 13, 2024 18:36
Copy link
Contributor

@Omrigan Omrigan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Regardless of the comment, I support this change, let's get it done and merged!

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This move of neonvm files into common pkg/ (vs keeping it in neonvm/pkg) hints towards erasing the boundary between neonvm and autoscaling. Personally, I am for that change, as I believe it might eventually help us to simplify the architecture and reduce the number of components.

But, we should agree on this explicitly.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm +1 for "erasing the boundary between neonvm and autoscaling", so probably it's a consensus already :)

@sharnoff
Copy link
Member Author

Will be affected by #988 (and also #989, if we go that route). I plan to rebase after those are merged.

@sharnoff sharnoff force-pushed the sharnoff/flatten-components branch from 878304e to 1a77f56 Compare August 27, 2024 23:06
@sharnoff sharnoff marked this pull request as ready for review August 28, 2024 05:41
@sharnoff

This comment was marked as resolved.

ARCHITECTURE.md Outdated Show resolved Hide resolved
ARCHITECTURE.md Outdated Show resolved Hide resolved
README-NeonVM.md Outdated
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why this move?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NeonVM is broader than just the CRD (e.g., neonvm-controller now has its own top-level directory). So I figured that the README for it should be at the highest common directory.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess the reasoning the same as for vxlan controller -> so that there is less stuff in top level dir.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am tempted to suggest neonvm-vxlan-controller -> neonvm/vxlan-controller

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you elaborate? (or, why just neonvm-vxlan-controller and not also neonvm-controller ?)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because it has an insignificant amount of code and could tucked away from top-level.

@sharnoff sharnoff mentioned this pull request Aug 29, 2024
Copy link
Member

@petuhovskiy petuhovskiy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm +1 for everything, but not sure how to ensure that all docs/yamls/scripts/tests won't have a broker path in them. Let's merge this sooner rather than later.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm +1 for "erasing the boundary between neonvm and autoscaling", so probably it's a consensus already :)

@sharnoff sharnoff force-pushed the sharnoff/flatten-components branch 3 times, most recently from 86fdd14 to 49cc225 Compare September 7, 2024 15:27
@sharnoff sharnoff force-pushed the sharnoff/flatten-components branch from 49cc225 to 18ecebb Compare September 8, 2024 21:51
@sharnoff sharnoff force-pushed the sharnoff/flatten-components branch 2 times, most recently from 97d4462 to 8ecaf63 Compare October 8, 2024 00:07
This basically encompasses two logical changes:

1. More fully integrating neonvm into the rest of the repository; and
2. Using a consistent directory structure for all components

The new structure is:

- All components get their own top-level directory (e.g.
  'autoscaler-agent', 'neonvm-controller', 'vm-builder')
- Inside the directory, there is:
  - cmd/ - directory containing main.go for the component
  - Dockerfile - for building the component
  - *.yaml - various YAML files for deploying, if applicable

Additionally, all dependencies outside the 'cmd' package are moved into
the 'pkg/' subdirectory. In particular, there is now 'pkg/neonvm/'.

Also of note: 'neonvm/hack/kernel' has been relocated to the new
top-level directory 'neonvm-kernel/'.

Under this structure, the user-visible changes are:

1. Introduction of new YAML: neonvm-controller.yaml
2. Introduction of new YAML: neonvm-vxlan-controller.yaml
3. neonvm.yaml now is *mostly* just the CRD.

The primary reasons for this change are:

1. Because it's been long enough since we moved the neonvm repository
   here that we might as well make it more well-integrated.
2. Because as we look to add components in other languages, we need a
   directory structure that supports it.

... and now that Autoscaling is GA, we can make the bigger changes we
were holding off on :)
@sharnoff sharnoff force-pushed the sharnoff/flatten-components branch from 8c4e1c6 to 7daf447 Compare October 8, 2024 19:24
Copy link

github-actions bot commented Oct 8, 2024

Merging this branch will increase overall coverage

Impacted Packages Coverage Δ 🤖
github.com/neondatabase/autoscaling/autoscale-scheduler/cmd 0.00% (ø)
github.com/neondatabase/autoscaling/autoscaler-agent/cmd 0.00% (ø)
github.com/neondatabase/autoscaling/neonvm-controller/cmd 0.00% (ø)
github.com/neondatabase/autoscaling/neonvm-runner/cmd 0.00% (ø)
github.com/neondatabase/autoscaling/neonvm-vxlan-controller/cmd 0.00% (ø)
github.com/neondatabase/autoscaling/pkg/neonvm/controllers 11.51% (+11.51%) 🎉
github.com/neondatabase/autoscaling/pkg/neonvm/controllers/buildtag 0.00% (ø)
github.com/neondatabase/autoscaling/pkg/neonvm/controllers/failurelag 100.00% (+100.00%) 🌟
github.com/neondatabase/autoscaling/pkg/neonvm/controllers/functests 0.00% (ø)
github.com/neondatabase/autoscaling/pkg/neonvm/ipam 0.00% (ø)
github.com/neondatabase/autoscaling/pkg/neonvm/ipam/demo 0.00% (ø)
github.com/neondatabase/autoscaling/vm-builder 0.00% (ø)

Coverage by file

Changed files (no unit tests)

Changed File Coverage Δ Total Covered Missed 🤖
github.com/neondatabase/autoscaling/autoscale-scheduler/cmd/main.go 0.00% (ø) 30 (+30) 0 30 (+30)
github.com/neondatabase/autoscaling/autoscaler-agent/cmd/main.go 0.00% (ø) 39 (+39) 0 39 (+39)
github.com/neondatabase/autoscaling/neonvm-controller/cmd/main.go 0.00% (ø) 130 (+130) 0 130 (+130)
github.com/neondatabase/autoscaling/neonvm-runner/cmd/main.go 0.00% (ø) 787 (+787) 0 787 (+787)
github.com/neondatabase/autoscaling/neonvm-vxlan-controller/cmd/main.go 0.00% (ø) 134 (+134) 0 134 (+134)
github.com/neondatabase/autoscaling/pkg/neonvm/controllers/buildtag/nodelete_false.go 0.00% (ø) 0 0 0
github.com/neondatabase/autoscaling/pkg/neonvm/controllers/buildtag/nodelete_true.go 0.00% (ø) 0 0 0
github.com/neondatabase/autoscaling/pkg/neonvm/controllers/buildtag/tagnames.go 0.00% (ø) 0 0 0
github.com/neondatabase/autoscaling/pkg/neonvm/controllers/catch_panic.go 0.00% (ø) 8 (+8) 0 8 (+8)
github.com/neondatabase/autoscaling/pkg/neonvm/controllers/config.go 0.00% (ø) 0 0 0
github.com/neondatabase/autoscaling/pkg/neonvm/controllers/failurelag/tracker.go 100.00% (+100.00%) 36 (+36) 36 (+36) 0 🌟
github.com/neondatabase/autoscaling/pkg/neonvm/controllers/metrics.go 6.67% (+6.67%) 45 (+45) 3 (+3) 42 (+42) 👍
github.com/neondatabase/autoscaling/pkg/neonvm/controllers/vm_controller.go 25.23% (+25.23%) 662 (+662) 167 (+167) 495 (+495) 🌟
github.com/neondatabase/autoscaling/pkg/neonvm/controllers/vm_qmp_queries.go 0.00% (ø) 380 (+380) 0 380 (+380)
github.com/neondatabase/autoscaling/pkg/neonvm/controllers/vmmigration_controller.go 0.00% (ø) 346 (+346) 0 346 (+346)
github.com/neondatabase/autoscaling/pkg/neonvm/controllers/webhook.go 0.00% (ø) 36 (+36) 0 36 (+36)
github.com/neondatabase/autoscaling/pkg/neonvm/ipam/allocate.go 0.00% (ø) 65 (+65) 0 65 (+65)
github.com/neondatabase/autoscaling/pkg/neonvm/ipam/client.go 0.00% (ø) 29 (+29) 0 29 (+29)
github.com/neondatabase/autoscaling/pkg/neonvm/ipam/demo/ipam.go 0.00% (ø) 36 (+36) 0 36 (+36)
github.com/neondatabase/autoscaling/pkg/neonvm/ipam/ipam.go 0.00% (ø) 469 (+469) 0 469 (+469)
github.com/neondatabase/autoscaling/pkg/neonvm/ipam/types.go 0.00% (ø) 3 (+3) 0 3 (+3)
github.com/neondatabase/autoscaling/vm-builder/main.go 0.00% (ø) 208 (+208) 0 208 (+208)

Please note that the "Total", "Covered", and "Missed" counts above refer to code statements instead of lines of code. The value in brackets refers to the test coverage of that file in the old version of the code.

Changed unit test files

  • github.com/neondatabase/autoscaling/pkg/neonvm/controllers/failurelag/tracker_test.go
  • github.com/neondatabase/autoscaling/pkg/neonvm/controllers/functests/suite_test.go
  • github.com/neondatabase/autoscaling/pkg/neonvm/controllers/functests/vm_controller_test.go
  • github.com/neondatabase/autoscaling/pkg/neonvm/controllers/vm_controller_unit_test.go
  • github.com/neondatabase/autoscaling/pkg/neonvm/controllers/vm_qmp_test.go

HTML Report

Click to open

@sharnoff sharnoff merged commit a2f53e3 into main Oct 8, 2024
22 checks passed
@sharnoff sharnoff deleted the sharnoff/flatten-components branch October 8, 2024 19:50
sharnoff added a commit that referenced this pull request Oct 10, 2024
This updates some comments from changes in #961 and #1073.
sharnoff added a commit that referenced this pull request Oct 10, 2024
This updates some comments from changes in #961 and #1073.
sharnoff added a commit that referenced this pull request Oct 14, 2024
Must've missed this in #1090 -- after #961, the common structure is that
main.go sits inside a cmd subdir for the component.
sharnoff added a commit that referenced this pull request Oct 14, 2024
Must've missed this in #1090 — after #961, the common structure is that
main.go sits inside a cmd subdir for the component.
Omrigan added a commit that referenced this pull request Oct 18, 2024
Since last release, vm-builder directory was moved into root
of the repo. Because of this, we can't download vm-builder during
release pipeline, this name is already occupied by a dir.

This was caused by the following commit:
a2f53e3 Flatten & unify component structure (#961)

Signed-off-by: Oleg Vasilev <[email protected]>
Omrigan added a commit that referenced this pull request Oct 18, 2024
This was done in the following commit:
a2f53e3 Flatten & unify component structure (#961)

Signed-off-by: Oleg Vasilev <[email protected]>
Omrigan added a commit that referenced this pull request Oct 19, 2024
Since last release, vm-builder directory was moved into root
of the repo. Because of this, we can't download vm-builder during
release pipeline, this name is already occupied by a dir.

This was caused by the following commit:
a2f53e3 Flatten & unify component structure (#961)

Signed-off-by: Oleg Vasilev <[email protected]>
Omrigan added a commit that referenced this pull request Oct 19, 2024
This was done in the following commit:
a2f53e3 Flatten & unify component structure (#961)

Signed-off-by: Oleg Vasilev <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants