Panic: EOF During Syncing "public"."c" Table in make sync Command

[salilsaifuddin]

When running the make sync command, the process fails with a panic: EOF error while syncing the "public"."c" table. The command successfully syncs other tables before encountering this issue. Below is the detailed output leading up to the error:

Steps to Reproduce
Execute make sync in the project directory.
Wait for the process to sync tables from PostgreSQL.
Observe the panic error during the syncing of "public"."c".

Env added
BEMIDB_PORT=54321
BEMIDB_DATABASE=bemidb
BEMIDB_USER=salil
BEMIDB_PASSWORD=salil
BEMIDB_LOG_LEVEL=DEBUG

Local storage

BEMIDB_STORAGE_TYPE=LOCAL
BEMIDB_STORAGE_PATH=../iceberg

PG_DATABASE_URL=postgres://postgres:postgres@:5432/

Observed Error Logs
bash
Copy code
2024/11/24 23:49:57 [INFO] Syncing "public"."c"...
2024/11/24 23:49:57 [DEBUG] Copied 1 row(s) into /tmp/"public"."c"3084334934
panic: EOF

goroutine 1 [running]:
main.PanicIfError({0x2bc12a0?, 0x3a39aa0?}, {0x0?, 0x6?, 0x3a77ff8?})
/home/test/BemiDB/src/utils.go:19 +0x9c
main.(*Syncer).syncFromPgTable(0xc00013c1b0, 0xc00055c900, {{0xc00011cd2a, 0x6}, {0x3a77ff8, 0x1}, {0x0, 0x0}})
/home/test/BemiDB/src/syncer.go:109 +0x3bd
main.(*Syncer).SyncFromPostgres(0xc00013c1b0)
/home/test/BemiDB/src/syncer.go:45 +0x225
main.syncFromPg(0x3ad7500)
/home/test/BemiDB/src/main.go:74 +0x1d
main.main()
/home/test/BemiDB/src/main.go:38 +0xac
exit status 2

Full error
devbox run --env-file .env "cd src && go run . sync"
2024/11/24 23:49:57 [INFO] Syncing "public"."a"...
2024/11/24 23:49:57 [DEBUG] Copied 1 row(s) into /tmp/"public"."a"2901591137
2024/11/24 23:49:57 [DEBUG] Parquet schema: {"Fields":[{"Tag":"name=hl7_edi, type=BYTE_ARRAY, repetitiontype=OPTIONAL, fieldid=3, convertedtype=UTF8"},{"Tag":"name=file_name, type=BYTE_ARRAY, repetitiontype=OPTIONAL, fieldid=4, convertedtype=UTF8"},{"Tag":"name=hl7_json, type=BYTE_ARRAY, repetitiontype=OPTIONAL, fieldid=5, convertedtype=UTF8"},{"Tag":"name=status, type=BYTE_ARRAY, repetitiontype=OPTIONAL, fieldid=12, convertedtype=UTF8"}],"Tag":"name=root"}
2024/11/24 23:49:57 [DEBUG] Parquet row: {"file_name":"processed_2.2_ECMC_5ec3bcda-b319-429d-a54c-56a21f5c9077_test-Abc-ChannelReader_0_006399387197.hl7","hl7_edi":"n"}}}}, "hl7v2_data_segments": [{}]}","status":null}
2024/11/24 23:49:57 [DEBUG] Parquet file with 1 record(s) created at: /home/test/BemiDB/iceberg/public/a/data/00000-0-47cd70c3-64c0-4543-a441-6ca6f72c20a0.parquet
2024/11/24 23:49:57 [DEBUG] Manifest file created at: /home/test/BemiDB/iceberg/public/a/metadata/47cd70c3-64c0-4543-a441-6ca6f72c20a0-m0.avro
2024/11/24 23:49:57 [DEBUG] Manifest list file created at: /home/test/BemiDB/iceberg/public/a/metadata/snap-1732510197888684133-0-47cd70c3-64c0-4543-a441-6ca6f72c20a0.avro
2024/11/24 23:49:57 [DEBUG] Metadata file created at: /home/test/BemiDB/iceberg/public/a/metadata/v1.metadata.json
2024/11/24 23:49:57 [DEBUG] Version hint file created at: /home/test/BemiDB/iceberg/public/a/metadata/version-hint.text
2024/11/24 23:49:57 [INFO] Syncing "public"."c"...
2024/11/24 23:49:57 [DEBUG] Copied 1 row(s) into /tmp/"public"."c"3084334934
panic: EOF

goroutine 1 [running]:
main.PanicIfError({0x2bc12a0?, 0x3a39aa0?}, {0x0?, 0x6?, 0x3a77ff8?})
/home/test/BemiDB/src/utils.go:19 +0x9c
main.(*Syncer).syncFromPgTable(0xc00013c1b0, 0xc00055c900, {{0xc00011cd2a, 0x6}, {0x3a77ff8, 0x1}, {0x0, 0x0}})
/home/test/BemiDB/src/syncer.go:109 +0x3bd
main.(*Syncer).SyncFromPostgres(0xc00013c1b0)
/home/test/BemiDB/src/syncer.go:45 +0x225
main.syncFromPg(0x3ad7500)
/home/test/BemiDB/src/main.go:74 +0x1d
main.main()
/home/test/BemiDB/src/main.go:38 +0xac
exit status 2
Error: error running script "cd src && go run . sync" in Devbox: exit status 1

[exAspArk]

Hey @salilsaifuddin — thank you for submitting this issue!

Would you be able to share the schema (and potentially anonymized data) from your database table that failed to sync? For example, you could execute this query in your Postgres database:

SELECT *
FROM information_schema.columns
WHERE table_schema = 'public' AND table_name = 'c'
ORDER BY ordinal_position;