-
Notifications
You must be signed in to change notification settings - Fork 125
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
SNOW-1660331: Should SnowflakeFileTransferOptions.getFilesToStream be exported member? #1207
Comments
hi and thanks for reporting this issue, taking a look |
thank you so much for the example you provided, it helped a lot in troubleshooting this issue. I setup a stage, uploaded a test.csv with following content
Then confirmed indeed
Fixing it in a local fork (only this single field) then # cat main.go
package main
import (
"bytes"
// "compress/gzip"
"context"
"database/sql"
"fmt"
"io"
"log"
"os"
"github.com/snowflakedb/gosnowflake"
)
func main(){
cfg, err := gosnowflake.GetConfigFromEnv([]*gosnowflake.ConfigParam{
{Name: "Account", EnvName: "SNOWFLAKE_TEST_ACCOUNT", FailOnMissing: true},
{Name: "User", EnvName: "SNOWFLAKE_TEST_USER", FailOnMissing: true},
{Name: "Password", EnvName: "SNOWFLAKE_TEST_PASSWORD", FailOnMissing: true},
})
dsn, err := gosnowflake.DSN(cfg)
if err != nil {
log.Fatalf("failed to create DSN from Config: %v, err: %v", cfg, err)
}
db, err := sql.Open(
"snowflake",
dsn,
)
if err != nil {
log.Fatal(err)
}
if err := db.Ping(); err != nil {
log.Fatal(err)
}
var streamBuf bytes.Buffer
ctx := gosnowflake.WithFileTransferOptions(context.Background(), &gosnowflake.SnowflakeFileTransferOptions{GetFileToStream: true})
ctx = gosnowflake.WithFileGetStream(ctx, &streamBuf)
// observe local file nomenclature, it's 'file://<path>' so in case '/tmp' is the dest path, then it's 3 '/'s
if _, err := db.ExecContext(ctx, "GET @test_db.public.mystage file:///tmp/"); err != nil {
log.Fatal(err)
}
br := bytes.NewReader(streamBuf.Bytes())
/*
gr, err := gzip.NewReader(&streamBuf)
if err != nil {
log.Fatal(err)
}
defer gr.Close()
*/
//io.Copy(os.Stdout, gr)
io.Copy(os.Stdout, br)
fmt.Println()
} now it gives:
Hope it helps you. About the unexported fields; we definitely need to look at them again because I'm sure some of them should be accessible from outside the module by their main purpose 😅 like |
PR in draft #1208 |
Thank you for investigating the issue. I'm glad we were able to identify issues other than getFilesToStream. I look forward to the resolution of the problems. |
change is merged and will be part of the next upcoming release cycle |
released with gosnowflake v1.11.2, in September 2024 release cycle |
Please answer these questions before submitting your issue.
In order to accurately debug the issue this information is required. Thanks!
v1.11.1
macOS 13.2.1
arm64
run
go version
in your console1.21.1
4.Server version:* E.g. 1.90.1
You may get the server version by running a query:
8.33.1
What did you do?
If possible, provide a recipe for reproducing the error.
A complete runnable program is good.
I would like to retrieve the files stored in the internal stage using the GET operation, which was supported in version 1.11.1, and output the contents of the files to the standard output. I referred to the example in the gosnowflake godocs, but when using gosnowflake as an external module, I found that I cannot access the getFilesToStream member of the SnowflakeFileTransferOptions struct.
What did you expect to see?
What should have happened and what happened instead?
I expected the content of the file obtained via the GET operation to be streamed to the standard output, but the contents of streamBytes only contained EOF. If it is necessary to set getFilesToStream to true, then a means to access it is required.
Can you set logging to DEBUG and collect the logs?
https://community.snowflake.com/s/article/How-to-generate-log-file-on-Snowflake-connectors
What is your Snowflake account identifier, if any? (Optional)
The text was updated successfully, but these errors were encountered: