problem with generator on 64 bit compters #65
Comments
|
here the valgrind output on the arm64 porter box from debian |
|
here my generator |
|
And indeed te code which call the generator |
|
Ok. The first thing to know is that I wrote generator most to see if I could. I was quite surprised how well it worked out, but I don't know that I'd really call it ready for serious use. But let's see what I can work out.. I developed and tested the module mostly on x86_64, so it's certainly not broken on all 64-bit systems in all circumstances. My guess is simply that you're running out of space on the generator's stack - 64-bit code will typically consume more stack space, because of the 64-bit pointers it needs to store there. Currently generator always allocates a fixed - and tiny (~8kiB) - stack for each generator, meaning you're pretty limited in what you can do inside the actual generator function. Your return type is also a reasonably sized structure, so if you're manipulating similar structures within your generator code that could consume stack pretty quickly. I had some fairly ambitious plans to improve the stack handling in coroutine and generator, but I didn't get very far with it before being distracted by other projects and I've never had a chance to go back to it. So first step is to see if that actually is the problem: go into generator.c and change DEFAULT_STACK_SIZE to something much larger (say a couple of megabytes) and see if the problem goes away. If it does, we can work together on improving the out of the box behaviour. |
|
Thanks a lot for your reply, I can confirm that using 4 times the current stack_size (8192*4), make the problem vanish for my use case. Now it is time to improve the out of box behaviour ;) |
|
It vanish on the arm64 architecture, but even if I increase a lot the stack size, on amd64 it still faill |
|
the stack trace is a bit different |
|
the sefgault arise when I define the generator. generator_def(trajectory_gen, struct Engine, struct Trajectory, tconfig)
{
switch(tconfig.tag){
case TRAJECTORY_HKL_FROM_TO:
{
uint i;
double dh = (tconfig.hklfromto.h1 - tconfig.hklfromto.h0) / (tconfig.hklfromto.n);
double dk = (tconfig.hklfromto.k1 - tconfig.hklfromto.k0) / (tconfig.hklfromto.n);
double dl = (tconfig.hklfromto.l1 - tconfig.hklfromto.l0) / (tconfig.hklfromto.n);
for(i=0; i<tconfig.hklfromto.n + 1; ++i){
double h = i * dh + tconfig.hklfromto.h0;
double k = i * dk + tconfig.hklfromto.k0;
double l = i * dl + tconfig.hklfromto.l0;
struct Engine econfig = EngineHkl(h, k, l, tconfig.hklfromto.mode);
generator_yield(econfig);
}
}
break;
}
} |
|
Here the valgrind output on amd64 The stack switch information worried me. |
|
Ok. So, the first step is altering generator to use coroutine_stack_alloc() instead of coroutine_stack_init(). The former is safer, more flexible, and if it fails is much more likely to fail with a relatively clear SEGV, instead of corrupting unrelated memory. The complication is that the generator code makes a temporary allocation at the "bottom" of the stack area (that is, at the far end of the buffer from where the stack itself will first occupy). That's used to pass the initial parameters into the generator code. So far, this requires manually managing the stack buffer, because coroutine_stack_*() doesn't provide a built in way to make such an allocation. So we either need to extend coroutine with a way to make such temporary allocations, or use a different means of passing the arguments in. Let's worry about x86_64 first - there may be further complications with arm64. The "switching stack" warnings are kind of expected - the generator code does indeed switch stacks, that's how it works. I have some valgrind calls which are supposed to tell it what's going on, but apparently they're not really supported any more, so we might just have to live with valgrind not understanding this code. |
|
Hello, I can not be of a great help here except if you want me to test patches :), have a good weekend |
|
Just for information it works with gcc6, on an amd64 computer :~/hkl/hkl$ gcc --version So there is something different with gcc-7 |
|
My guess would just be that gcc7 is doing some better optimizations that result in less stack usage, and so we're not hitting the overrun that we did before. |
|
Hello, I reveive this today https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=889878#26 so it seems that this fantastic guy found the culprite :)) |
It seems that this patch solve our problem with generator on all arch supported by Debian.
Hello, I am using generator and coroutine on an i386 computer in order to build my hkl project.
It works fine.
The problem arise when I switch to 64 bits computer and can be reproduce using the current debian unstable source package from here
https://packages.debian.org/source/unstable/hkl
(the problematic version is this one https://tracker.debian.org/news/930678)
It generates this kind of failure via a Segfault.
https://buildd.debian.org/status/fetch.php?pkg=hkl&arch=amd64&ver=5.0.0.2447-1&stamp=1518020479&raw=0
I tryed to debug the problem via gdb and I got this
If we look at the generator line,we get this
I do not understand why it work great on 32 bits computer and not on 64 bits computer.
can you help me find the problem. I am ready to test this until it works on 64 bits computer also :))
Cheers
Frederic
The text was updated successfully, but these errors were encountered: