Add initial shader module #1196

TristanCacqueray · 2024-10-18T14:40:43Z

This change adds the loadShader, to be used like this:

let {uniforms} = await loadShader`
  uniform float iColorOffset;
  uniform vec3 iColor;

  void mainImage( out vec4 fragColor, in vec2 fragCoord ) {
    vec2 uv = fragCoord / iResolution.xy;
    vec3 col = 0.5 + 0.5*cos(iColorOffset+iColor+uv.xyx+vec3(0,2,4));
    fragColor = vec4(col, 1);
  }
`
let smooth = (desired) => (value) => value + ((desired - value) / 20)
setcpm(20)
$: s("bd hh [sd -] hh").gain("<1 .5>").onTrigger((_, _hap, now) => {
  // Bump a random color component
  uniforms.iColor.incr(.5, Math.floor(now))
  // Slowly change the palette offset
  uniforms.iColorOffset.set(cur => smooth(cur + 1))
}, false)

… as presented in https://club.tidalcycles.org/t/adding-webgl-shader-uniform-modulation/5481

TristanCacqueray · 2024-10-18T14:42:56Z

When I run pnpm install, the lockfile is updated to version 9 with a massive diff. I tried to setup pnpm version 6, but pnpm install failed with WARN GET https://registry.npmjs.org/acorn error (ERR_INVALID_THIS). May I propose a pnpm bump?

TristanCacqueray · 2024-10-18T14:45:01Z

This PR looks like this:

This change adds the `loadShader` and `shader` function, to be used like this: ```strudel await loadShader` // The modulation targets uniform float iColor; void mainImage( out vec4 fragColor, in vec2 fragCoord ) { vec2 uv = fragCoord / iResolution.xy; vec3 col = 0.5 + 0.5*cos(iColor+uv.xyx+vec3(0,2,4)); fragColor = vec4(col, 0); } ` $: s("bd").shader({uniform: 'iColor'}) ```

felixroos · 2024-10-18T21:27:06Z

hey, thanks for the PR, being able to directly control shaders is a great feature imo. I am trying to test it, but for some reason, the canvas is not visible, without getting an error.. is there anything else i need to do? I've cloned your branch and tried to test with your first basic example.

felixroos · 2024-10-18T21:28:45Z

When I run pnpm install, the lockfile is updated to version 9 with a massive diff. I tried to setup pnpm version 6, but pnpm install failed with WARN GET https://registry.npmjs.org/acorn error (ERR_INVALID_THIS). May I propose a pnpm bump?

yes the lockfile really needs updating, probably best to do it in a separate PR, i can do that

The attribute for the draw call is called drawFrame, not draw.

TristanCacqueray · 2024-10-18T22:23:57Z

@felixroos That's great to hear! I just pushed a fix that prevented the code to be reloaded (though refreshing the page should have displayed the canvas).

I guess this should be marked as draft until the shader parameters support pattern strings (the values presently need to be single quoted). Also, it would be nice to figure out how to control the modulations, like the decay value which is presently hardcoded. Though I think that PR is already quite usable.

The shader module should also support texture input, for now it's just a plain fragment shader.

felixroos · 2024-10-18T22:46:05Z

ah turns out the first example did set the alpha channel to 0 so it was always transparent :) I'm still not sure how the pattern interacts with the uniforms defined in the shader function, could you clarify that? The iColor doesn't seem to react to the pattern

TristanCacqueray · 2024-10-18T23:16:55Z

Interesting, on Firefox the alpha doesn't seem to be enabled. The iColor should be incremented by one each time a note is played, the effect is a bit dim, you can increase it by adding 4.0*iColor in the shader, or there is a gain attribute you can pass to the shader options, like so: {uniform: 'iColor', gain: 4}.

I picked 1 because the haps in onTrigger don't seem to have a velocity, but it would be great to use the sound intensity.

Also note that this PR only reacts to noteOn events, I think we would need to analyze the sound to get a more accurate modulation source, though this simple solution works great for percussion sounds.

TristanCacqueray · 2024-10-18T23:19:52Z

Lastly, something I haven't done yet but I think would be really nice, is to pass the cpm as uniform to sync to sin values in the shader to better match the composition. Perhaps we could define some useful macros in the shader headers...

felixroos · 2024-10-18T23:25:05Z

Interesting, on Firefox the alpha doesn't seem to be enabled. The iColor should be incremented by one each time a note is played, the effect is a bit dim, you can increase it by adding 4.0*iColor in the shader, or there is a gain attribute you can pass to the shader options, like so: {uniform: 'iColor', gain: 4}.

I picked 1 because the haps in onTrigger don't seem to have a velocity, but it would be great to use the sound intensity.

Also note that this PR only reacts to noteOn events, I think we would need to analyze the sound to get a more accurate modulation source, though this simple solution works great for percussion sounds.

This idea might be half baked, but wouldn't it be more flexible to allow the user to fill in the mapping between hap values and uniforms? Something like note("c a f e").pan(sine).shader({myUniformName: 'pan'}) . this would move more logic into the pattern realm and less hidden logic into the shader function. One more radical alternative would be to automatically define all hap values as uniforms, e.g note("c a f e").pan(sine).shader() would automatically define note and pan. although i'm not quite sure how to handle string values like notes

felixroos · 2024-10-18T23:35:37Z

some more ideas:

it should make sense to move this to a separate package called "shader", as it depends on another package (picogl) + the shader file does not import anything from the draw package anyway
it seems a bit arbitrary to me to have the canvas on the top right. I'd rather treat it like the hydra canvas and lay it behind the code in full size. There could potentially also be options to customize the size / move the canvas if you want that look.
we could potentially get rid of picogl ?! i've done some experiments with webgl and it's not a lot of code that's needed to spin up a shader: https://github.com/felixroos/schattenspiel having less dependencies is less to worry about, but also not that big of a deal

what do you think?

TristanCacqueray · 2024-10-18T23:40:29Z

As explained in the tidal club post, I'm not really familiar with strudel, so I'm looking forward your feedback to improve the implementations.

About the mapping, well, I couldn't find examples for sending events from one track to another, so I just re-used the code from the Midi API. I guess we could also attach the shader destination directly to a signal or pattern like so: pan(sine.shader('uniformName')). The shader destination should also be a pattern to pick different uniforms or array position, e.g. shader("<uniform1 uniform2>").

Your other ideas sound great to me, I can update the PR tomorrow. I picked picogl because it handle the context loss, it's not implemented yet here, but it might be useful to avoid blank screen. Otherwise you are right, we could totally do it by hand.

felixroos · 2024-10-18T23:55:36Z

I guess we could also attach the shader destination directly to a signal or pattern like so: pan(sine.shader('uniformName')). The shader destination should also be a pattern to pick different uniforms or array position, e.g. shader("").

would that make it possible to call shader multiple times to populate multiple uniforms? If yes, i'd rather call it uniform, so you could do:

$: note("c a f e").pan(sine.uniform("iPan"))
$: s("hh*8").gain(".5 1".uniform("iGain"))

The shader function probably wouldn't be needed, as we're calling loadShader anyway, which could already trigger the rendering. It would only be needed if multiple shaders should be rendered in parallel, which is maybe out of scope. I'm still not sure if I like the separated definitions, so let's compare that to a single uniform mapping:

$: note("c a f e").pan(sine)
$: s("hh*8").gain(".5 1")
all(x=>x.uniforms({iPan: "pan", iGain: "gain"))

In this notation, the uniforms function maps uniform names to hap value keys. The all function is a way to run a function on all patterns (what you probably mean by tracks). Maybe both notations could also coexist..

Your other ideas sound great to me, I can update the PR tomorrow. I picked picogl because it handle the context loss, it's not implemented yet here, but it might be useful to avoid blank screen. Otherwise you are right, we could totally do it by hand.

Maybe let's keep it for now. If it's a separate package the dependency is fine i think

…into shader

felixroos · 2024-10-19T13:14:03Z

just formatted the code (using pnpm codeformat) to pass the tests (hopefully) + it's easier to track changes this way

TristanCacqueray · 2024-10-19T13:17:36Z

With the latest commit, the pretty demo now looks like this:

let truchetFTW = await fetch('https://raw.githubusercontent.com/TristanCacqueray/shaders/refs/heads/main/shaders/Truchet%20%2B%20Kaleidoscope%20FTW.glsl').then((res) => res.text())
await loadShader(truchetFTW)

$: s("bd -".uniform("moveFWD")).bank("RolandTR909")
$: s("- sd".uniform("rotations:seq")).bank("RolandTR909")
$:
 n("<[0 2 3 - 7 8 9 -] [4 2 - 7 3 7 9 -] [9 8 6 3 - 9 4 -]>")
.uniform("modulations:pitch")
.anchor("<C4 A3 C4 G3>/2")
.chord("<Gm7 <C>>/2").voicing()
.transpose(12)
.sound("<sine, z_sine triangle>")
.decay("<.6 1 2 .5 .1>")
.gain("<.2 .2 .3 .4 .5 .1>")
.room("<.6 .5 .4 .3>")
.pan(sine.range(0, 1))
.jux(rev).roomsize(2).delay(.4)
.gain(.2)

In the uniform.mjs module you can find the uniform param documentation. I think this is starting to look great.
Now I'm not sure how to handle signals like sine, in the onTrigger handler, I only get their value at the noteOn event, not the continuous one.

Lastly, I would like to expose the "smoothing" function to pre-process the value, I think we should have:

direct, e.g. to pass signal value as-is,
incr(time), e.g. to smooth change and only increment the uniform, as it is presently hardcoded
adsr, e.g. to let the value go up and down

I though about adding an extra component to the uniform string syntax, e.g. "iColor:decay(100)", but perhaps this should be expressed differently.

PS: Sorry I erased the codeformat, I'll setup the an auto formatter next.

TristanCacqueray · 2024-10-20T12:40:50Z

I've been studying the pattern and signal module and I'll update the PR with with the following pattern methods:

uniform("name"): using pat.withValue to pass the current number value
uniformTrigger("name"): using pat.onTrigger to pass the current note/sound event

It doesn't seem like we can check in advance if a pattern is numerical, so using two methods looks cleaner.

To smooth the value, it looks easier to use an extra argument, e.g. uniform("time", "incr:50"). Though we could also use an object argument, e.g. uniform({name: "time", smooth: "incr:50"}), but that sounds a bit more complicated because we need to evaluate the nested patterns.

Then I'll look into the uniforms method to set multiple uniform values at once, as you suggested with the all functions, but I'd like to figure out the simpler methods beforehand.

TristanCacqueray · 2024-10-20T14:00:15Z

Here is a new example:

await loadShader`
  uniform float iColorOffset;
  uniform float iColors[3];

  void mainImage( out vec4 fragColor, in vec2 fragCoord ) {
    vec2 uv = fragCoord / iResolution.xy;
    vec3 bcol = vec3(0.+iColors[0],2.+iColors[1],4.+iColors[2]);
    vec3 col = 0.5 + 0.5*cos(iColorOffset+uv.xyx+bcol);
    fragColor = vec4(col, 1);
  }
`
setcpm(20)
$:
 s("bd hh [sd -] hh")
.pan(sine.slow(6).uniform("iColorOffset"))
.uniformTrigger("iColors")

felixroos · 2024-10-20T18:20:44Z

I am struggling to understand the syntax passed to uniform / uniformTrigger, could you maybe elaborate more on what the different options do? Some functionality could maybe also expressed with patterns, which would add less complexity. Moving logic to pattern land would also allow people to implement their own functions from within the repl.

TristanCacqueray · 2024-10-20T19:18:15Z

Yeah, I'm not happy with the uniform syntax, but I couldn't figure out a better way to expose the following options:

gain:N: a number to multiply the value before setting the uniform.
slow:N: a number to change the value slowly: 1 is instantaneous, 50 is slow.
index:N: the destination position for uniform's arrays. This can be a number, or seq to pick a different position each time, or random for a random position.

For uniformTrigger, the default index is based on the note or sound.

Here are some example usages:

let truchetFTW = await fetch('https://raw.githubusercontent.com/TristanCacqueray/shaders/refs/heads/main/shaders/Truchet%20%2B%20Kaleidoscope%20FTW.glsl').then((res) => res.text())
await loadShader(truchetFTW)

setcpm(96)

$: s("bd").bank("RolandTR808")
   .gain(2).dist("<1 .7 .7 .7>")
   .mask("<1@30 0@2>")
   // each kick rotate a different ring
   .uniformTrigger("rotations:index:seq:gain:.2")
$: sound("hh*4").bank("RolandTR808")
   .room(.3).gain(".25 .3 .4")
   .mask("<0@8 1@32>")
   // the hats change the color rapidely
   .uniformTrigger("icolor:gain:0.5:slow:5")
$: s("cp/8").bank("RolandTR808")
   .hpf(500).hpa(.8).hpenv("<-3 -2 -3 -2 -1>/8")
   .room(0.5).roomsize(7).rlp(5000).gain(.2)
   // the snare advances the tunnel
   .uniformTrigger("moveFWD:slow:20:gain:4")
$: note("<C D G A Bb D C A G D Bb A>*[2,2.02]")
  .clip(1.1)
  .transpose("<-12 -24 -12 0>/8")
  // .sound("sawtooth")
  .sound("triangle")
  .cutoff(perlin.slow(5).range(20,1200))
  .room(.8).roomsize(.6)
  .gain(.4)
  // each piano note modulate one of the 8 truchet patterns
  .uniformTrigger("modulations:gain:.5:slow:20")

TristanCacqueray · 2024-10-20T19:53:06Z

I meant I couldn't figure how to use the existing functions like gain or slow, because they get applied to the audio pattern. The uniform pattern is a bit different since it capture the values and send them to the shader. In that situation, how could we re-use the existing functions like gain but for the uniform pattern?

TristanCacqueray · 2024-10-20T22:58:12Z

I think I get how to handle the uniform's option as pattern. So the API would look like this:
uniform({name: "iColors", gain: "<2 1 1 1>", slow: "10"}). The uniform method can stack the patterns from the options, and by using context I can get their values for each outer hap.

With the last commit 1baa6da, here is how the demo looks like:

let truchetFTW = await fetch('https://raw.githubusercontent.com/TristanCacqueray/shaders/refs/heads/main/shaders/Truchet%20%2B%20Kaleidoscope%20FTW.glsl').then((res) => res.text())
await loadShader(truchetFTW)

setcpm(96)

$: s("bd").bank("RolandTR808")
   .gain(2).dist("<1 .7 .7 .7>")
   .mask("<1@30 0@2>")
   .uniform({dest: "rotations:seq", gain: "<0.1 0.3>", onTrigger: true})
$: sound("hh*4").bank("RolandTR808")
   .room(.3).gain(".25 .3 .4")
   .mask("<0@8 1@32>")
   .uniform({dest: 'icolor', gain: 0.5, slow:"5", onTrigger: true})
$: s("cp/8").bank("RolandTR808")
   .uniform({dest: 'moveFWD', slow: "<40 20>", gain: "<8 4>", onTrigger: true})
   .hpf(500).hpa(.8).hpenv("<-3 -2 -3 -2 -1>/8")
   .room(0.5).roomsize(7).rlp(5000).gain(.2)
$: note("<C D G A Bb D C A G D Bb A>*[2,2.02]")
  .clip(1.1)
  .transpose("<-12 -24 -12 0>/8")
  // .sound("sawtooth")
  .sound("triangle")
  .cutoff(perlin.slow(5).range(20,1200))
  .room(.8).roomsize(.6)
  .gain(.4)
  .uniform({dest: "modulations", gain: 1.5, slow: 50, onTrigger: true})

This change makes the following demo works: `s("bd").uniform("rot")`

felixroos · 2024-10-21T20:29:28Z

looks pretty neat! I think the difficulty for using pattern methods is that the uniform values are stateful, also expected to retain state between evaluations. It is theoretically possible with patterns to do a similar thing:

https://strudel.cc/#bGV0IGNvdW50ID0gMTsKCiQ6IHMoImJkKjw0IDY%2BIikKICAub25UcmlnZ2VyKCgpID0%2BIGNvdW50KyssIGZhbHNlKQogIC5zcGVlZChzaWduYWwoKCkgPT4gY291bnQlOCsyKSkK

I'm not sure if it's a good idea though.. Just playing a bit with the idea, you could maybe also pass uniforms as the second argument of loadShader:

let iColorOffset = 1;

await loadShader(`
  uniform float iColorOffset;

  void mainImage( out vec4 fragColor, in vec2 fragCoord ) {
    vec2 uv = fragCoord / iResolution.xy;
    vec3 bcol = vec3(0.1,2.1,4.2);
    vec3 col = 0.5 + 0.5*cos(iColorOffset+uv.xyx+bcol);
    fragColor = vec4(col, 1);
  }
`, () => ({
  iColorOffset
}))

$: s("bd*<4 6>")
  .onTrigger(() => iColorOffset++, false)

This would allow implement any custom stateful logic in userland. What do you think?

TristanCacqueray · 2024-10-21T21:43:35Z

Thank you for following up on this feature; I appreciate your feedback!

I think there is room for experimentation as it feels like we are in uncharted territory. I am not sure how many uniform variables could be used in a meaningful way, and perhaps a skillful shader artist could use every pattern to create pleasing visuals. So I hope this feature will be as flexible as possible to avoid limiting explorations.

What I have proposed so far is based on my work with animation-fractal which mostly uses NoteOn events. In this project, I found that it is important to control the rate of change to avoid unpleasing stroboscopic effects and this is why I made this the default behavior with the slow option. I haven't tried using pattern values, like from the pan or the dist, but that is already supported.

Using a regular strudel pattern to activate variables sounds ideal because we can leverage the existing modifiers. For example: .uniform("rotations".mask("<1@3 0@1>")). Though I find the proposed solution for setting the slow and gain to be somewhat clunky. Also it seems that I'm not using stack correctly and the uniform mask gets wrongly applied to the outer pattern.

I like how explicit your uniform argument var example looks like. I guess we can also globalThis the uniforms so that they are available by default. I can update the PR later this week, and I'd be happy to remove the uniform function as it doesn't have to be part of this initial change.

TristanCacqueray · 2024-10-27T13:44:19Z

I think the difficulty for using pattern methods is that the uniform values are stateful, also expected to retain state between evaluations.

Right, using withValue or withHap doesn't work as I expected: the callback is not synced and we can't use that for setting uniform values. Could we have a withCurrentValue to get the synced value, e.g. the one that is highlighted in the code with a white box? I tried using setTimeout and the strudeltick, but I can't tell if and how that would work...

So I removed the uniform pattern function for now, and changed loadShader to return the uniforms so that they can be manually set on trigger as you suggested. Please find the updated demos in the top comment, let me know what do you think.

TristanCacqueray · 2024-11-20T15:26:24Z

we could potentially get rid of picogl ?! i've done some experiments with webgl and it's not a lot of code that's needed to spin up a shader: https://github.com/felixroos/schattenspiel having less dependencies is less to worry about, but also not that big of a deal

I started to remove picogl, that's better indeed, and that should let us try the feature directly on the strudel's version currently deployed.

TristanCacqueray · 2024-12-22T13:56:36Z

Trying to keep the ball rolling, I made the module standalone and uploaded a demo here: https://strudel.cc/?ErKvnkz3z_3p . Next I'll address the todo list, please let me know if the new design looks good to you.

TristanCacqueray force-pushed the shader branch 2 times, most recently from a58f1a4 to 30de163 Compare October 18, 2024 15:14

TristanCacqueray force-pushed the shader branch from 30de163 to f0ddb2d Compare October 18, 2024 15:40

Fix shader reload

1b8998a

The attribute for the draw call is called drawFrame, not draw.

felixroos added 2 commits October 19, 2024 01:14

Merge remote-tracking branch 'origin/main' into shader

46b4aff

codeformat

13168bb

TristanCacqueray and others added 4 commits October 19, 2024 06:29

Move shader module into it's own package

735d531

Extract the uniform helper to its own module

5c983ce

codeformat

bf3a095

Refactor the uniform function and add documentation

b0149f8

TristanCacqueray force-pushed the shader branch from bf3a095 to b0149f8 Compare October 19, 2024 13:10

felixroos added 2 commits October 19, 2024 15:12

Merge branch 'shader' of https://github.com/TristanCacqueray/strudel …

d3e624f

…into shader

codeformat

fcd550b

felixroos added 2 commits October 19, 2024 15:15

fix: test runtime for uniform function

8d1180f

snapshot

67c0eba

Update the shader todos

9b4675f

Use pat.withValue and add uniformTrigger

b85a5a9

Update uniform value and add tests

cad11e1

felixroos and others added 2 commits October 20, 2024 21:07

Merge branch 'main' into shader

e6fbef9

Fix default uniformTrigger indexs

9e0b840

TristanCacqueray added 2 commits October 20, 2024 19:55

Update uniform method to take an option object

1baa6da

Add uniform pattern function doc

f8a2414

TristanCacqueray force-pushed the shader branch from d536807 to f8a2414 Compare October 21, 2024 02:06

TristanCacqueray added 2 commits October 21, 2024 09:23

Fix lint error

36d4d89

Automatically set onTrigger when the hap value is not a number

65cc317

This change makes the following demo works: `s("bd").uniform("rot")`

Remove uniform pattern function and expose the shader instance

77bbb96

TristanCacqueray force-pushed the shader branch from 2cf77b0 to 77bbb96 Compare October 27, 2024 14:17

Improve the API when using a frame modifier

0d67176

Remove PicoGL dependency

fc335c9

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add initial shader module #1196

Add initial shader module #1196

TristanCacqueray commented Oct 18, 2024 •

edited

Loading

TristanCacqueray commented Oct 18, 2024

TristanCacqueray commented Oct 18, 2024

felixroos commented Oct 18, 2024

felixroos commented Oct 18, 2024

TristanCacqueray commented Oct 18, 2024

felixroos commented Oct 18, 2024

TristanCacqueray commented Oct 18, 2024

TristanCacqueray commented Oct 18, 2024

felixroos commented Oct 18, 2024 •

edited

Loading

felixroos commented Oct 18, 2024

TristanCacqueray commented Oct 18, 2024

felixroos commented Oct 18, 2024

felixroos commented Oct 19, 2024 •

edited

Loading

TristanCacqueray commented Oct 19, 2024

TristanCacqueray commented Oct 20, 2024

TristanCacqueray commented Oct 20, 2024

felixroos commented Oct 20, 2024

TristanCacqueray commented Oct 20, 2024

TristanCacqueray commented Oct 20, 2024

TristanCacqueray commented Oct 20, 2024 •

edited

Loading

felixroos commented Oct 21, 2024

TristanCacqueray commented Oct 21, 2024

TristanCacqueray commented Oct 27, 2024 •

edited

Loading

TristanCacqueray commented Nov 20, 2024

TristanCacqueray commented Dec 22, 2024

Add initial shader module #1196

Are you sure you want to change the base?

Add initial shader module #1196

Conversation

TristanCacqueray commented Oct 18, 2024 • edited Loading

TristanCacqueray commented Oct 18, 2024

TristanCacqueray commented Oct 18, 2024

felixroos commented Oct 18, 2024

felixroos commented Oct 18, 2024

TristanCacqueray commented Oct 18, 2024

felixroos commented Oct 18, 2024

TristanCacqueray commented Oct 18, 2024

TristanCacqueray commented Oct 18, 2024

felixroos commented Oct 18, 2024 • edited Loading

felixroos commented Oct 18, 2024

TristanCacqueray commented Oct 18, 2024

felixroos commented Oct 18, 2024

felixroos commented Oct 19, 2024 • edited Loading

TristanCacqueray commented Oct 19, 2024

TristanCacqueray commented Oct 20, 2024

TristanCacqueray commented Oct 20, 2024

felixroos commented Oct 20, 2024

TristanCacqueray commented Oct 20, 2024

TristanCacqueray commented Oct 20, 2024

TristanCacqueray commented Oct 20, 2024 • edited Loading

felixroos commented Oct 21, 2024

TristanCacqueray commented Oct 21, 2024

TristanCacqueray commented Oct 27, 2024 • edited Loading

TristanCacqueray commented Nov 20, 2024

TristanCacqueray commented Dec 22, 2024

TristanCacqueray commented Oct 18, 2024 •

edited

Loading

felixroos commented Oct 18, 2024 •

edited

Loading

felixroos commented Oct 19, 2024 •

edited

Loading

TristanCacqueray commented Oct 20, 2024 •

edited

Loading

TristanCacqueray commented Oct 27, 2024 •

edited

Loading