Slm #421

gparmer · 2021-06-16T17:06:08Z

Summary of this Pull Request (PR)

Initial slm implementation that is designed to

simplify the current sl codebase by removing hierarchical scheduling,
removing allocation from the core code, and
decomposing the system into modules more cleanly.

This is a work in progress, and the tests are in implementation/tests/slm/main.c. Trello encodes many of the next steps, but they include:

getting multicore working again which involves bootup, and communication (queues, and asnds),
determine the proper design of the periodic blocking API,
adding the blockpoint logic, and
test the heck out of it with existing tests.

Intent for your PR

Choose one (Mandatory):

This PR is for a code-review and is intended to get feedback, but not to be pulled yet.
This PR is mature, and ready to be integrated into the repo.

Reviewers (Mandatory):

(Specify @<github.com username(s)> of the reviewers. Ex: @user1, @user2)

@WenyuanShao @msdx321 @betahxy

Code Quality

As part of this pull request, I've considered the following:

Style:

Comments adhere to the Style Guide (SG)
Spacing adhere's to the SG
Naming adhere's to the SG
All other aspects of the SG are adhered to, or exceptions are justified in this pull request
I have run the auto formatter on my code before submitting this PR (see doc/auto_formatter.md for instructions)

Code Craftsmanship:

I've made an attempt to remove all redundant code
I've considered ways in which my changes might impact existing code, and cleaned it up
I've formatted the code in an effort to make it easier to read (proper error handling, function use, etc...)
I've commented appropriately where code is tricky
I agree that there is no "throw-away" code, and that code in this PR is of high quality

Testing

I've tested the code using the following test programs (provide list here):

slmtests

gparmer

Some notes on things to fix.

gparmer · 2021-06-16T17:36:36Z

src/components/lib/slm/slm_policy_timer.h

@@ -0,0 +1,5 @@
+#ifndef SLM_POLICY_TIMER_H
+#define SLM_POLICY_TIMER_H
+


Remove this file.

gparmer · 2021-06-16T17:36:57Z

src/components/lib/slm/slm_timer.h

+#ifndef SLM_TIMER_H
+#define SLM_TIMER_H
+
+struct slm_timer_thd {


REmove this file.

gparmer · 2021-06-16T17:37:33Z

src/components/implementation/tests/slm/doc.md

@@ -0,0 +1,9 @@
+## tests
+
+This is the skeleton interface used by the `mkcomponent.sh` script to aid in the creation of a new component.


TODO: add documentation here.

gparmer · 2021-06-16T17:42:19Z

src/components/implementation/tests/slm/main.c

+};
+
+struct slm_timer_thd *
+slm_thd_timer_policy(struct slm_thd *t)


Replace with slm_container.h (see below).

Create a set of macros to enable the scheduler to easily mix and match different policies (edf/fprr/fifo), timers (quantum/oneshot), and memory allocators (static/dynamic).

gparmer · 2021-06-16T17:46:51Z

src/components/lib/slm/doc.md

@@ -0,0 +1,9 @@
+## slmin
+
+This is the skeleton library used by the `mklib.sh` script to aid in the creation of a new library.


Add actual documentation.

gparmer · 2021-06-16T17:49:45Z

src/components/lib/slm/slm_mod_sched.h

+
+/* Thread allocation and indexing API */
+
+struct slm_thd *slm_policy_thd_alloc(void);


FIXME: make all of this consistent with the above fprr APIs, or remove.

gparmer · 2021-06-16T17:51:45Z

src/components/lib/component/cos_component.h

+ * `ret`. This assumes that ret == 0 is success, otherwise jump to the
+ * label.
+ */
+#define ERR_CHECK(expr, label)			\


Fix indentation.

- -no-pic and -no-pie are not valid flags in ld - update needed env deps

- it does not make sense to specify this section manually and set a PHDR (.interp segment) for it, this may cause ld reporting errors because some Alloc type sections might be tried to put into the interp segment by the linker, but we don't have this segment in the final PHDR set.

- fix composer allocating captbl_end using BOOT_CAPTBL_FREE, it is 52 now. This can be used by both i386 and x86_64 - fix pgtbl_lkup_lvl to return/set page table entry flags correctly - fix cos compinfo initialiaztion problem

still needs higher level API/library to call still needs to be tested

- x86_64 ABI requires a 16-byte alignment for fetching data and when calling a function, because some instructions like movabs and sse require data to be 16-byte aligned. Otherwise, it will case a #GP

- .boottext section needs to be aligned to generate correct bootable image - the stack in loader.S needs to be set to .data section, the original .comm will generate a .bss section that could have random address after compiling

- fix the linker script so it can correctly compile a bootable image - add -nmagic flag to $(LD) in the Makefile so $(LD) does not add additional blank information to the bootable image

- enable smp booting into long mode and jump into kmain

…nd validation of composition scripts, and added access to AS representation in the composer's state. The overall goal is to add full initargs support for address spaces, going all the way back to a specification for address spaces from the composition scripts. Steps: 1. DONE: parsing and validation of composition scripts. 2. DONE: add an AS representation to the pass structure of the composer for later pass access. 3. Order ASes by the parent relationship. 4. Assign virtual addresses based on ASes and the parent relation. 5. Synthesize the initargs based on the ASes and addresses. 6. Use the initargs and the AS specification in the booter.

…zation (with containing components, and exclusive AS components) Steps: 1. DONE: parsing and validation of composition scripts. 2. DONE: add an AS representation to the pass structure of the composer for later pass access. 3. -> DONE: Order ASes by the parent relationship. 4. Assign virtual addresses based on ASes and the parent relation. 5. Synthesize the initargs based on the ASes and addresses. 6. Use the initargs and the AS specification in the booter.

1. DONE: parsing and validation of composition scripts. 2. DONE: add an AS representation to the pass structure of the composer for later pass access. 3. DONE: Order ASes by the parent relationship. 4. -> DONE: Assign virtual addresses based on ASes and the parent relation. This is *not yet integrated into the output*. That's next commit. 5. Synthesize the initargs based on the ASes and addresses. 6. Use the initargs and the AS specification in the booter.

1. DONE: parsing and validation of composition scripts. 2. DONE: add an AS representation to the pass structure of the composer for later pass access. 3. DONE: Order ASes by the parent relationship. 4. DONE: Assign virtual addresses based on ASes and the parent relation. -> Now integrated into the output! 5. Synthesize the initargs based on the ASes and addresses. 6. Use the initargs and the AS specification in the booter.

…fication. Steps: 1. DONE: parsing and validation of composition scripts. 2. DONE: add an AS representation to the pass structure of the composer for later pass access. 3. DONE: Order ASes by the parent relationship. 4. DONE: Assign virtual addresses based on ASes and the parent relation. 5. -> DONE: Synthesize the initargs based on the ASes and addresses. 6. Use the initargs and the AS specification in the booter.

Steps: 1. DONE: parsing and validation of composition scripts. 2. DONE: add an AS representation to the pass structure of the composer for later pass access. 3. DONE: Order ASes by the parent relationship. 4. DONE: Assign virtual addresses based on ASes and the parent relation. 5. DONE: Synthesize the initargs based on the ASes and addresses. 6. -> DONE: Use the initargs and the AS specification in the booter. Pending: BUG for split ASes that don't properly maintain non-overlapping ASes.

Steps: 1. DONE: parsing and validation of composition scripts. 2. DONE: add an AS representation to the pass structure of the composer for later pass access. 3. DONE: Order ASes by the parent relationship. 4. DONE: Assign virtual addresses based on ASes and the parent relation. 5. DONE: Synthesize the initargs based on the ASes and addresses. 6. DONE: Use the initargs and the AS specification in the booter.

Official NIC/DPDK component support to Composite

Virtual address space specification, assignment, and booting.

…eduler

gparmer requested a review from WenyuanShao June 16, 2021 17:06

gparmer commented Jun 16, 2021

View reviewed changes

betahxy and others added 27 commits October 8, 2021 09:36

add deps setting script for 64 bit composite

cde63ba

fix compiling problem

0a0c290

- -no-pic and -no-pie are not valid flags in ld - update needed env deps

update env deps script

55f6b40

fix booter switch into components

f9b1d6e

- fix composer allocating captbl_end using BOOT_CAPTBL_FREE, it is 52 now. This can be used by both i386 and x86_64 - fix pgtbl_lkup_lvl to return/set page table entry flags correctly - fix cos compinfo initialiaztion problem

fix simple pingpong sinv call

3793443

fix pingpong sinv call stubs

a3b9a4a

infrastructure for creating shared pgtbl node, cons

bfb0991

still needs higher level API/library to call still needs to be tested

finish fixing pingpong tests

4f69f5d

revert back to use gwsystem ps repo

4642559

fix compiling warnings

d2c1ef8

fix run code within kvm

62b200b

- x86_64 ABI requires a 16-byte alignment for fetching data and when calling a function, because some instructions like movabs and sse require data to be 16-byte aligned. Otherwise, it will case a #GP

add some comments

e0ac0e6

fix -O3 optimization errors and some warnings

8a03f0a

fix i386 port pingpong

90107bc

fix pingpong test in x86_64 port

484da5c

improve cos script

d59f58c

improve build scripts

c0aba92

fix i386 bug

cd4111c

- .boottext section needs to be aligned to generate correct bootable image - the stack in loader.S needs to be set to .data section, the original .comm will generate a .bss section that could have random address after compiling

remove unnecessary file

f542dfd

fix x86_64 compiling problems

3ac4f70

- fix the linker script so it can correctly compile a bootable image - add -nmagic flag to $(LD) in the Makefile so $(LD) does not add additional blank information to the bootable image

improve init_env script

7e9aaea

Merge branch 'gwsystems:loaderarm' into loaderarm

54eaeaf

remove unnecessary files

d2f6658

multi-core bootup support in cos script

c650c72

fix smp booting in loader.S

267e596

- enable smp booting into long mode and jump into kmain

fix code format of loader.S

cfc219f

betahxy and others added 30 commits May 2, 2022 00:51

add init_env dependencies

5b8fa8e

minor fix

8b67159

keep posix legacy as its original content

cc8e98c

add lwip submodule

8c34fa5

add lwip submodule

ba66857

Merge branch 'main' of github.com:betahxy/composite

e307303

add lwip submodule and successfully compile it

0950f94

enable avx feature

0868465

test lwip within nic component

f1c74a1

enable lwip debug message

8bd5569

WARNING: does not build. Transient commit

7915890

WARNING: does not build. Transient commit

052b284

add DPDK bench test and some minor changes

92585ca

change lwip submodule url to gwsystem

c1fb10c

minor fix

e68d887

fix format

ca2f9f2

minor fixes

22d5f9a

Merge pull request #451 from betahxy/main

b42e20c

Official NIC/DPDK component support to Composite

Merge branch 'main' of github.com:gparmer/composite.local into main

2f68b0a

Comment update

6748528

Merge branch 'main' into composer_vas

b94fe2c

Merge pull request #453 from gparmer/composer_vas

0da657c

Virtual address space specification, assignment, and booting.

Merge conflict resolution: removing symbol issues in the sl-based sch…

7182995

…eduler

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Slm #421

Slm #421

gparmer commented Jun 16, 2021 •

edited by betahxy

Loading

gparmer left a comment

gparmer Jun 16, 2021

gparmer Jun 16, 2021

gparmer Jun 16, 2021

gparmer Jun 16, 2021

gparmer Jun 16, 2021

gparmer Jun 16, 2021

gparmer Jun 16, 2021

		@@ -0,0 +1,5 @@
		#ifndef SLM_POLICY_TIMER_H
		#define SLM_POLICY_TIMER_H

		@@ -0,0 +1,9 @@
		## tests

		This is the skeleton interface used by the `mkcomponent.sh` script to aid in the creation of a new component.

		@@ -0,0 +1,9 @@
		## slmin

		This is the skeleton library used by the `mklib.sh` script to aid in the creation of a new library.


		/* Thread allocation and indexing API */

		struct slm_thd *slm_policy_thd_alloc(void);

Slm #421

Are you sure you want to change the base?

Slm #421

Conversation

gparmer commented Jun 16, 2021 • edited by betahxy Loading

Summary of this Pull Request (PR)

Intent for your PR

Reviewers (Mandatory):

Code Quality

Testing

gparmer left a comment

Choose a reason for hiding this comment

gparmer Jun 16, 2021

Choose a reason for hiding this comment

gparmer Jun 16, 2021

Choose a reason for hiding this comment

gparmer Jun 16, 2021

Choose a reason for hiding this comment

gparmer Jun 16, 2021

Choose a reason for hiding this comment

gparmer Jun 16, 2021

Choose a reason for hiding this comment

gparmer Jun 16, 2021

Choose a reason for hiding this comment

gparmer Jun 16, 2021

Choose a reason for hiding this comment

gparmer commented Jun 16, 2021 •

edited by betahxy

Loading