fix parallel mode: add missing pkg dep & missing path creations

[udance4ever]

Two main categories:

1. missing package dependency declarations
2. missing directory paths

Gaps became apparent while debugging buildroot’s experimental PARALLEL_BUILD mode which uses PER_PACKAGE directories: https://pastebin.com/cLynrDqZ

Also verified these commits work for small builds like bcm2836

I am able to get both x86_64 and bcm2836 builds to completion (serial, only bcm2836 works in parallel, x86_64 is super close!)

[nadenislamarre]

some are strange,
python-pyxel can't required evmapy.

[nadenislamarre]

libglvnd depends on nvidia, so on an other package or at least on x86_64

[udance4ever]

thank you for taking the time to review the changes.

some are strange, python-pyxel can't required evmapy.

you’re right - I fixed it so it doesn’t depend on evmapy - it was the evmapy path that needs to be created if it doesn’t exist (the 2nd major class of changes)

libglvnd depends on nvidia, so on an other package or at least on x86_64

is there a change that needs to happen here?

just completed a parallel build using these .mk/Config.in files for bcm2836 last night - all packages compiled (except for mangohud and fixed this just now).

Firing off a clean build (I clobbered my build during finalization by accident) and will do x86_64 after that.

[nadenislamarre]

@dmanlfc when back, can you read this pr too please, you'll be a better feedback than me

[nadenislamarre]

mainly on libglvnd, i've a doubt on the test to do, even if x86_64 should do the job

[udance4ever]

bcm2836 parallel clean build 67 min (w ccache). final images in 86 min. M1 MBP 16GB to external NVMe (not Thunderbolt)

firing off x86_64 parallel build in the background.

[nadenislamarre]

you manage to make buildroot parallel mode working with batocera ?

[udance4ever]

you manage to make buildroot parallel mode working with batocera ?

yes, i like torturing myself :)

what happened is when I starting learning how to build - I used parallel build mode not knowing it was broken. so i’ve been getting to bottom of what breaks it.

I’ll have to benchmark with a zeroed out ccache, it is probably ~4.5h (dl already downloaded, pre-finalize) to build all the code for bcm2836. all 8 cores stay nicely slammed.

[nadenislamarre]

it would be wonderfull if we could merge to parallel builds
i wait dman however, he masterize a lot builds and dependancies

[udance4ever]

ok - long story short, I can only get parallel builds to work for bcm2836.

when I try to build the same codebase and target x86_64, the build finishes and Batocera boots (I can ssh in just fine) but the ES interface doesn’t initialize correctly (photo I posted to Discord w the white squares)

For my own sanity, I’m building x86_64 in serial mode on the same exact codebase so I’m 100% sure it is the parallelization that is introducing some weird artifact.

Given the number of SBCs we support, it’ll still be nice if all the SBC builds were parallelizable.

I don’t have any other SBC hardware to test this theory on unfortunately so hope the rest of crew can help test this out.

[nadenislamarre]

where in discord, i don't see it ?

[nadenislamarre]

do you have a pr to enable parallel builds ? i don't remember exactly what to do

[udance4ever]

ok - the jury’s in. I finished a serial build with the exact same config and codebase and it boots properly and ES initializes correctly. this is good news but a bit of a bummer parallel builds are only restricted to boards like bcm2836 for now.

I have two complete build directories side-by-side if anyone has suggestions on what to compare.

where in discord, i don't see it ?

I don’t see a way to link to a message in Discord. here’s the photo:

do you have a pr to enable parallel builds ? i don't remember exactly what to do

happy to file a PR if it makes a difference. I jumped into this not intending to get parallel builds to work - I was just trying to get my first build to work so getting the bcm2836 parallel build working was a side effect :)

[nadenislamarre]

i have an idea for this one.
one package is providing xinitrc,
one of bacera too.
/etc/X11/xinit/xinitrc or something like it

the original xinitrc open 3 terms like in your capture
the batocera one opens emulationstation.

this one
package/batocera/emulationstation/batocera-emulationstation/xorg/xinitrc
just replace the file and it should fix.
and in .mk we should do something to not copy the original one

[dmanlfc]

Please mark as draft until completion. ibglvnd is not x86 exclusive. Therefore change the logic to add the dependency if package BR2_PACKAGE_XORG7 is selected.

I also see dependencies being added to the .mk files but not the corresponding Config.in for packages. Please ensure they're in sync.

Finally squash the commits and force update accordingly as you work through the draft so it's one commit, thanks.

[udance4ever]

Please mark as draft until completion.

whoops - my bad - I didn’t realize renaming a branch ends up closing a PR. reopening as draft.

libglvnd is not x86 exclusive. Therefore change the logic to add the dependency if package BR2_PACKAGE_XORG7 is selected.

thanks for the direction - I just noticed bcm2836 didn’t need it so this explains why.

I also see dependencies being added to the .mk files but not the corresponding Config.in for packages. Please ensure they're in sync.

ok - I’ll clean this up. Any insight why buildroot complains about missing selects and other times doesn’t?

Finally squash the commits and force update accordingly as you work through the draft so it's one commit, thanks.

I debated whether or not to do this and intentionally left them atomic so you could follow new changes I discovered as I was debugging the x86_64 build further. (all the reason for this to be a draft branch). I will squash my commits before pulling my pull request out of draft if this is okay.

and just for the record, I’ve reached out to the developer of the br-pbuilder project as he has figured out how to do parallel builds without per-package directories. If it actually works, it might be the key to getting our x86_64 build parallelized as I don’t understand how to get to the root of why parallelism introduces the glitch in the photo above.

[udance4ever]

@dmanlfc made changes you requested. Buildroot isn’t happy about syncing Config.in and .mk files (though I totally agree with you this should be done!) - it complains about 3 classes of errors:

1. circular dependencies
2. BR2_PACKAGE_GSTREAMER1 is unselected
3. vkd3d-proton - BR2_PACKAGE_HOST_ZSTD

https://pastebin.com/a7pD6CTg

I’m not surprised about the 2nd class - I had to create a .patch file for webkitgtk to disable using gstreamer1 so v41-dev would build without errors:

https://github.com/udance4ever/batocera.buildroot/blob/udance4ever/workaround/webkitgtk-transcoder/package/webkitgtk/0002-GStreamerDefinitions-disable-USE_GSTREAMER_TRANSCODER.patch

3rd class we can ignore for now as this appears related to ironing out wine build on aarch64

it’s the 1st class (circular dependencies) I’m not clear how we go about resolving.

[dmanlfc]

looking good, how are you tracking?