Waypoint job list receives large message, errors

[catsby]

After your Waypoint server has accumulated enough jobs, the CLI command waypoint job list will start to fail:

$ waypoint job list 
! grpc: received message larger than max (14681194 vs. 4194304)

The limit option is performed client side, so unfortunately it is no help:

$ waypoint job list -limit=1 
! grpc: received message larger than max (14759844 vs. 4194304)

We should provide a better error message here if possible, or better, expand the job list to do the limiting server side. Bonus points for pagination, but that's probably much more non-trivial

[briancain]

BTW - this will be true for any endpoint that returns a lot of data. Eventually we should be handling this through pagination, but a temp fix might be piping through a limit param for any List endpoints and doing the trim on the server side before returning the data to the client.

Another thing that would be good is to detect the size of the response and return an error if it's too large so we can give a better error to the user when we exceed the grpc size limit.

[const-tmp]

exactly the same issue

➜  ~ waypoint version
CLI: v0.10.5
Server: v0.10.5

➜  ~ waypoint task list
! grpc: received message larger than max (16748239 vs. 4194304)

➜  ~ waypoint job list
! grpc: received message larger than max (7242920 vs. 4194304)

Server logs

2023-02-11T12:22:20.659Z [INFO]  waypoint.server.grpc: /hashicorp.waypoint.Waypoint/GetVersionInfo request
2023-02-11T12:22:20.659Z [INFO]  waypoint.server.grpc: /hashicorp.waypoint.Waypoint/GetVersionInfo response: error=<nil> duration=446.936µs
2023-02-11T12:22:20.689Z [INFO]  waypoint.server.grpc: /hashicorp.waypoint.Waypoint/ListTask request
2023-02-11T12:22:21.026Z [INFO]  waypoint.server.grpc: /hashicorp.waypoint.Waypoint/ListTask response: error=<nil> duration=336.896064ms

But a couple of days ago it worked as expected, and I even deployed an app. But now it just stopped working.

[rastakajakwanna]

Hi, it is supposed to be fixed in the latest Waypoint version (https://github.com/hashicorp/waypoint/releases/tag/v0.11.0):

core: waypoint job list will now retrieve paginated list of jobs to avoid grpc data limits per request [https://github.com//issues/4271]

but ...

[16:43:30](develop) ~/git $ waypoint version
CLI: v0.11.0 (e92d6fbe0)
Server: v0.11.0

[16:43:31](develop) ~/git $ waypoint job list
! grpc: received message larger than max (16441917 vs. 4194304)

Huh? Am I missing something ? 😃

[radriaanse]

The issue seems to be that the actual pagination isn't yet implemented on the server side.

It's possible to workaround this by setting the grpc.MaxRecvMsgSizeCallOption to a suitable size on the ListJobs request and building Waypoint from source; I used this to be able to cancel some hanging jobs.

[rastakajakwanna]

Still an issue 😭

$ waypoint task inspect -run-job-id=01H8259FS3VYQDFS011BXR99RP

panic: runtime error: index out of range [0] with length 0

goroutine 1 [running]:
github.com/hashicorp/waypoint/internal/cli.(*TaskInspectCommand).Run(0xc000ec5260, {0xc000d35970?, 0xffffffffffffffff?, 0xb?})
	/home/runner/work/waypoint/waypoint/internal/cli/task_inspect.go:40 +0x1172
github.com/mitchellh/cli.(*CLI).Run(0xc000d3f040)
	/home/runner/go/pkg/mod/github.com/mitchellh/[email protected]/cli.go:262 +0x5f8
github.com/hashicorp/waypoint/internal/cli.Main({0xc000072080?, 0x31201a0?, 0xc0000061a0?})
	/home/runner/work/waypoint/waypoint/internal/cli/main.go:127 +0x56d
main.main()
	/home/runner/work/waypoint/waypoint/cmd/waypoint/main.go:14 +0x77

$ waypoint task list
! grpc: received message larger than max (40698419 vs. 4194304)

$ waypoint version
CLI: v0.11.4 (7128fba)
Server: v0.11.3