Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Bandwidth Exhausted #411

Open
ronykrell opened this issue May 25, 2020 · 7 comments
Open

Bandwidth Exhausted #411

ronykrell opened this issue May 25, 2020 · 7 comments

Comments

@ronykrell
Copy link

ronykrell commented May 25, 2020 •
edited
Loading

Expected behavior

firestore-import working correctly - i.e. importing contents of json file created by firestore-export back into firestore.

Actual behavior

The import works partially - a few seconds after starting I get these errors

Writing documents for dev/CPUqFGRmPqsctsCovM7n/keys
Writing documents for dev/CPUqFGRmPqsctsCovM7n/programCounters
Writing documents for dev/CPUqFGRmPqsctsCovM7n/sheets
Writing documents for dev/CPUqFGRmPqsctsCovM7n/varieties
Error: 8 RESOURCE_EXHAUSTED: Bandwidth exhausted
    at Object.callErrorFromStatus (/home/ec2-user/.nvm/versions/node/v13.3.0/lib/node_modules/node-firestore-import-export/node_modules/@grpc/grpc-js/build/src/call.js:30:26)
    at Object.onReceiveStatus (/home/ec2-user/.nvm/versions/node/v13.3.0/lib/node_modules/node-firestore-import-export/node_modules/@grpc/grpc-js/build/src/client.js:175:52)
    at Object.onReceiveStatus (/home/ec2-user/.nvm/versions/node/v13.3.0/lib/node_modules/node-firestore-import-export/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:341:141)
    at Object.onReceiveStatus (/home/ec2-user/.nvm/versions/node/v13.3.0/lib/node_modules/node-firestore-import-export/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:304:181)
    at Http2CallStream.outputStatus (/home/ec2-user/.nvm/versions/node/v13.3.0/lib/node_modules/node-firestore-import-export/node_modules/@grpc/grpc-js/build/src/call-stream.js:116:74)
    at Http2CallStream.maybeOutputStatus (/home/ec2-user/.nvm/versions/node/v13.3.0/lib/node_modules/node-firestore-import-export/node_modules/@grpc/grpc-js/build/src/call-stream.js:155:22)
    at Http2CallStream.endCall (/home/ec2-user/.nvm/versions/node/v13.3.0/lib/node_modules/node-firestore-import-export/node_modules/@grpc/grpc-js/build/src/call-stream.js:141:18)
    at ClientHttp2Stream.<anonymous> (/home/ec2-user/.nvm/versions/node/v13.3.0/lib/node_modules/node-firestore-import-export/node_modules/@grpc/grpc-js/build/src/call-stream.js:403:22)
    at ClientHttp2Stream.emit (events.js:219:5)
    at emitErrorCloseNT (internal/streams/destroy.js:76:8) {
  code: 8,
  details: 'Bandwidth exhausted',
  metadata: Metadata { internalRepr: Map {}, options: {} }
}
Error: 13 INTERNAL: Received RST_STREAM with code 2
    at Object.callErrorFromStatus (/home/ec2-user/.nvm/versions/node/v13.3.0/lib/node_modules/node-firestore-import-export/node_modules/@grpc/grpc-js/build/src/call.js:30:26)
    at Object.onReceiveStatus (/home/ec2-user/.nvm/versions/node/v13.3.0/lib/node_modules/node-firestore-import-export/node_modules/@grpc/grpc-js/build/src/client.js:175:52)
    at Object.onReceiveStatus (/home/ec2-user/.nvm/versions/node/v13.3.0/lib/node_modules/node-firestore-import-export/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:341:141)
    at Object.onReceiveStatus (/home/ec2-user/.nvm/versions/node/v13.3.0/lib/node_modules/node-firestore-import-export/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:304:181)
    at Http2CallStream.outputStatus (/home/ec2-user/.nvm/versions/node/v13.3.0/lib/node_modules/node-firestore-import-export/node_modules/@grpc/grpc-js/build/src/call-stream.js:116:74)
    at Http2CallStream.maybeOutputStatus (/home/ec2-user/.nvm/versions/node/v13.3.0/lib/node_modules/node-firestore-import-export/node_modules/@grpc/grpc-js/build/src/call-stream.js:155:22)
    at Http2CallStream.endCall (/home/ec2-user/.nvm/versions/node/v13.3.0/lib/node_modules/node-firestore-import-export/node_modules/@grpc/grpc-js/build/src/call-stream.js:141:18)
    at ClientHttp2Stream.<anonymous> (/home/ec2-user/.nvm/versions/node/v13.3.0/lib/node_modules/node-firestore-import-export/node_modules/@grpc/grpc-js/build/src/call-stream.js:403:22)
    at ClientHttp2Stream.emit (events.js:219:5)
    at emitErrorCloseNT (internal/streams/destroy.js:76:8) {
  code: 13,
  details: 'Received RST_STREAM with code 2',
  metadata: Metadata { internalRepr: Map {}, options: {} }
}
Error: 13 INTERNAL: Received RST_STREAM with code 2
    at Object.callErrorFromStatus (/home/ec2-user/.nvm/versions/node/v13.3.0/lib/node_modules/node-firestore-import-export/node_modules/@grpc/grpc-js/build/src/call.js:30:26)
    at Object.onReceiveStatus (/home/ec2-user/.nvm/versions/node/v13.3.0/lib/node_modules/node-firestore-import-export/node_modules/@grpc/grpc-js/build/src/client.js:175:52)
    at Object.onReceiveStatus (/home/ec2-user/.nvm/versions/node/v13.3.0/lib/node_modules/node-firestore-import-export/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:341:141)
    at Object.onReceiveStatus (/home/ec2-user/.nvm/versions/node/v13.3.0/lib/node_modules/node-firestore-import-export/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:304:181)
    at Http2CallStream.outputStatus (/home/ec2-user/.nvm/versions/node/v13.3.0/lib/node_modules/node-firestore-import-export/node_modules/@grpc/grpc-js/build/src/call-stream.js:116:74)
    at Http2CallStream.maybeOutputStatus (/home/ec2-user/.nvm/versions/node/v13.3.0/lib/node_modules/node-firestore-import-export/node_modules/@grpc/grpc-js/build/src/call-stream.js:155:22)
    at Http2CallStream.endCall (/home/ec2-user/.nvm/versions/node/v13.3.0/lib/node_modules/node-firestore-import-export/node_modules/@grpc/grpc-js/build/src/call-stream.js:141:18)
    at ClientHttp2Stream.<anonymous> (/home/ec2-user/.nvm/versions/node/v13.3.0/lib/node_modules/node-firestore-import-export/node_modules/@grpc/grpc-js/build/src/call-stream.js:403:22)
    at ClientHttp2Stream.emit (events.js:219:5)
    at emitErrorCloseNT (internal/streams/destroy.js:76:8) {
  code: 13,
  details: 'Received RST_STREAM with code 2',
  metadata: Metadata { internalRepr: Map {}, options: {} }
}
Error: 8 RESOURCE_EXHAUSTED: Bandwidth exhausted
    at Object.callErrorFromStatus (/home/ec2-user/.nvm/versions/node/v13.3.0/lib/node_modules/node-firestore-import-export/node_modules/@grpc/grpc-js/build/src/call.js:30:26)
    at Object.onReceiveStatus (/home/ec2-user/.nvm/versions/node/v13.3.0/lib/node_modules/node-firestore-import-export/node_modules/@grpc/grpc-js/build/src/client.js:175:52)
    at Object.onReceiveStatus (/home/ec2-user/.nvm/versions/node/v13.3.0/lib/node_modules/node-firestore-import-export/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:341:141)
    at Object.onReceiveStatus (/home/ec2-user/.nvm/versions/node/v13.3.0/lib/node_modules/node-firestore-import-export/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:304:181)
    at Http2CallStream.outputStatus (/home/ec2-user/.nvm/versions/node/v13.3.0/lib/node_modules/node-firestore-import-export/node_modules/@grpc/grpc-js/build/src/call-stream.js:116:74)
    at Http2CallStream.maybeOutputStatus (/home/ec2-user/.nvm/versions/node/v13.3.0/lib/node_modules/node-firestore-import-export/node_modules/@grpc/grpc-js/build/src/call-stream.js:155:22)
    at Http2CallStream.endCall (/home/ec2-user/.nvm/versions/node/v13.3.0/lib/node_modules/node-firestore-import-export/node_modules/@grpc/grpc-js/build/src/call-stream.js:141:18)
    at ClientHttp2Stream.<anonymous> (/home/ec2-user/.nvm/versions/node/v13.3.0/lib/node_modules/node-firestore-import-export/node_modules/@grpc/grpc-js/build/src/call-stream.js:403:22)
    at ClientHttp2Stream.emit (events.js:219:5)
    at emitErrorCloseNT (internal/streams/destroy.js:76:8) {
  code: 8,
  details: 'Bandwidth exhausted',
  metadata: Metadata { internalRepr: Map {}, options: {} }
}
Error: 8 RESOURCE_EXHAUSTED: Bandwidth exhausted
    at Object.callErrorFromStatus (/home/ec2-user/.nvm/versions/node/v13.3.0/lib/node_modules/node-firestore-import-export/node_modules/@grpc/grpc-js/build/src/call.js:30:26)
    at Object.onReceiveStatus (/home/ec2-user/.nvm/versions/node/v13.3.0/lib/node_modules/node-firestore-import-export/node_modules/@grpc/grpc-js/build/src/client.js:175:52)
    at Object.onReceiveStatus (/home/ec2-user/.nvm/versions/node/v13.3.0/lib/node_modules/node-firestore-import-export/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:341:141)
    at Object.onReceiveStatus (/home/ec2-user/.nvm/versions/node/v13.3.0/lib/node_modules/node-firestore-import-export/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:304:181)
    at Http2CallStream.outputStatus (/home/ec2-user/.nvm/versions/node/v13.3.0/lib/node_modules/node-firestore-import-export/node_modules/@grpc/grpc-js/build/src/call-stream.js:116:74)
    at Http2CallStream.maybeOutputStatus (/home/ec2-user/.nvm/versions/node/v13.3.0/lib/node_modules/node-firestore-import-export/node_modules/@grpc/grpc-js/build/src/call-stream.js:155:22)
    at Http2CallStream.endCall (/home/ec2-user/.nvm/versions/node/v13.3.0/lib/node_modules/node-firestore-import-export/node_modules/@grpc/grpc-js/build/src/call-stream.js:141:18)
    at ClientHttp2Stream.<anonymous> (/home/ec2-user/.nvm/versions/node/v13.3.0/lib/node_modules/node-firestore-import-export/node_modules/@grpc/grpc-js/build/src/call-stream.js:403:22)
    at ClientHttp2Stream.emit (events.js:219:5)
    at emitErrorCloseNT (internal/streams/destroy.js:76:8) {
  code: 8,
  details: 'Bandwidth exhausted',
  metadata: Metadata { internalRepr: Map {}, options: {} }
}
Error: 8 RESOURCE_EXHAUSTED: Bandwidth exhausted
    at Object.callErrorFromStatus (/home/ec2-user/.nvm/versions/node/v13.3.0/lib/node_modules/node-firestore-import-export/node_modules/@grpc/grpc-js/build/src/call.js:30:26)
    at Object.onReceiveStatus (/home/ec2-user/.nvm/versions/node/v13.3.0/lib/node_modules/node-firestore-import-export/node_modules/@grpc/grpc-js/build/src/client.js:175:52)
    at Object.onReceiveStatus (/home/ec2-user/.nvm/versions/node/v13.3.0/lib/node_modules/node-firestore-import-export/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:341:141)
    at Object.onReceiveStatus (/home/ec2-user/.nvm/versions/node/v13.3.0/lib/node_modules/node-firestore-import-export/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:304:181)
    at Http2CallStream.outputStatus (/home/ec2-user/.nvm/versions/node/v13.3.0/lib/node_modules/node-firestore-import-export/node_modules/@grpc/grpc-js/build/src/call-stream.js:116:74)
    at Http2CallStream.maybeOutputStatus (/home/ec2-user/.nvm/versions/node/v13.3.0/lib/node_modules/node-firestore-import-export/node_modules/@grpc/grpc-js/build/src/call-stream.js:155:22)
    at Http2CallStream.endCall (/home/ec2-user/.nvm/versions/node/v13.3.0/lib/node_modules/node-firestore-import-export/node_modules/@grpc/grpc-js/build/src/call-stream.js:141:18)
    at ClientHttp2Stream.<anonymous> (/home/ec2-user/.nvm/versions/node/v13.3.0/lib/node_modules/node-firestore-import-export/node_modules/@grpc/grpc-js/build/src/call-stream.js:403:22)
    at ClientHttp2Stream.emit (events.js:219:5)
    at emitErrorCloseNT (internal/streams/destroy.js:76:8) {
  code: 8,
  details: 'Bandwidth exhausted',
  metadata: Metadata { internalRepr: Map {}, options: {} }
}
Error: 13 INTERNAL: Received RST_STREAM with code 2
    at Object.callErrorFromStatus (/home/ec2-user/.nvm/versions/node/v13.3.0/lib/node_modules/node-firestore-import-export/node_modules/@grpc/grpc-js/build/src/call.js:30:26)
    at Object.onReceiveStatus (/home/ec2-user/.nvm/versions/node/v13.3.0/lib/node_modules/node-firestore-import-export/node_modules/@grpc/grpc-js/build/src/client.js:175:52)
    at Object.onReceiveStatus (/home/ec2-user/.nvm/versions/node/v13.3.0/lib/node_modules/node-firestore-import-export/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:341:141)
    at Object.onReceiveStatus (/home/ec2-user/.nvm/versions/node/v13.3.0/lib/node_modules/node-firestore-import-export/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:304:181)
    at Http2CallStream.outputStatus (/home/ec2-user/.nvm/versions/node/v13.3.0/lib/node_modules/node-firestore-import-export/node_modules/@grpc/grpc-js/build/src/call-stream.js:116:74)
    at Http2CallStream.maybeOutputStatus (/home/ec2-user/.nvm/versions/node/v13.3.0/lib/node_modules/node-firestore-import-export/node_modules/@grpc/grpc-js/build/src/call-stream.js:155:22)
    at Http2CallStream.endCall (/home/ec2-user/.nvm/versions/node/v13.3.0/lib/node_modules/node-firestore-import-export/node_modules/@grpc/grpc-js/build/src/call-stream.js:141:18)
    at ClientHttp2Stream.<anonymous> (/home/ec2-user/.nvm/versions/node/v13.3.0/lib/node_modules/node-firestore-import-export/node_modules/@grpc/grpc-js/build/src/call-stream.js:403:22)
    at ClientHttp2Stream.emit (events.js:219:5)
    at emitErrorCloseNT (internal/streams/destroy.js:76:8) {
  code: 13,
  details: 'Received RST_STREAM with code 2',
  metadata: Metadata { internalRepr: Map {}, options: {} }
}
Error: 13 INTERNAL: Received RST_STREAM with code 2
    at Object.callErrorFromStatus (/home/ec2-user/.nvm/versions/node/v13.3.0/lib/node_modules/node-firestore-import-export/node_modules/@grpc/grpc-js/build/src/call.js:30:26)
    at Object.onReceiveStatus (/home/ec2-user/.nvm/versions/node/v13.3.0/lib/node_modules/node-firestore-import-export/node_modules/@grpc/grpc-js/build/src/client.js:175:52)
    at Object.onReceiveStatus (/home/ec2-user/.nvm/versions/node/v13.3.0/lib/node_modules/node-firestore-import-export/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:341:141)
    at Object.onReceiveStatus (/home/ec2-user/.nvm/versions/node/v13.3.0/lib/node_modules/node-firestore-import-export/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:304:181)
    at Http2CallStream.outputStatus (/home/ec2-user/.nvm/versions/node/v13.3.0/lib/node_modules/node-firestore-import-export/node_modules/@grpc/grpc-js/build/src/call-stream.js:116:74)
    at Http2CallStream.maybeOutputStatus (/home/ec2-user/.nvm/versions/node/v13.3.0/lib/node_modules/node-firestore-import-export/node_modules/@grpc/grpc-js/build/src/call-stream.js:155:22)
    at Http2CallStream.endCall (/home/ec2-user/.nvm/versions/node/v13.3.0/lib/node_modules/node-firestore-import-export/node_modules/@grpc/grpc-js/build/src/call-stream.js:141:18)
    at ClientHttp2Stream.<anonymous> (/home/ec2-user/.nvm/versions/node/v13.3.0/lib/node_modules/node-firestore-import-export/node_modules/@grpc/grpc-js/build/src/call-stream.js:403:22)
    at ClientHttp2Stream.emit (events.js:219:5)
    at emitErrorCloseNT (internal/streams/destroy.js:76:8) {
  code: 13,
  details: 'Received RST_STREAM with code 2',
  metadata: Metadata { internalRepr: Map {}, options: {} }
}
All done 🎉

Some data is imported into the firestore database, but other data is not imported.

Steps to reproduce the behavior

Using version 1.1.0

Running command
/home/ec2-user/.nvm/versions/node/v13.3.0/bin/firestore-import -a ~/pc-test-firebase-adminsdk-5relp-daf7fa32cf.json -b backup-202005251534.json

The Firestore project is using the Blaze plan
@jontybrook
Copy link

I'm also facing this issue

@3rcok
Copy link

3rcok commented Sep 2, 2020

I had the same issue

@kisonay
Copy link

kisonay commented Mar 20, 2021

anyone able to resolve this?

@MdFarhanRaja
Copy link

I'm also facing same. Please please please help...

@ghost
Copy link

ghost commented Nov 15, 2021

Same please help

@tilmanschweitzer
Copy link

I think this issue can be fixed by limiting the concurrency. If a limitation is necessary seems to depend on the content (depth, collection size etc). I'll open a PR later.

tilmanschweitzer pushed a commit to tilmanschweitzer/node-firestore-import-export that referenced this issue Dec 7, 2021
Add parameter --maxConcurrency to prevent different load errors.
d5ab713 
Depending on the content different load errors like 'Bandwidth exhausted'
or 'Received RST_STREAM with code 2' occur and cause the import to be incomplete.
Limiting the concurrent imported batches seems to prevents issues.
The default behaviour is still without any limit.

see jloosli#411
@tilmanschweitzer tilmanschweitzer mentioned this issue Dec 7, 2021
Open
@nonoumasy
Copy link

nonoumasy commented Nov 17, 2022 •
edited
Loading

Just wondering if this issue has been resolved.
I'm also getting the 'Bandwidth exhausted' error. cheers

@nonoumasy nonoumasy mentioned this issue Apr 2, 2023
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
7 participants
@tilmanschweitzer @jontybrook @3rcok @ronykrell @kisonay @nonoumasy @MdFarhanRaja