Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Allow large Parse File uploads #9286

Open
wants to merge 27 commits into
base: alpha
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
734d85e
feat: adjust the router for large buffers
dalyaidan1 Aug 17, 2024
088c669
feat: adjust adapter for large blobs
dalyaidan1 Aug 17, 2024
3385d2f
test: test file uploads
dalyaidan1 Aug 17, 2024
0a2c904
fix: don't depend on axios
dalyaidan1 Aug 17, 2024
d2c89af
feat: clean up files
dalyaidan1 Aug 17, 2024
216c078
fix: file default data and adapter comment
dalyaidan1 Aug 17, 2024
a559453
test: split test and move to better file
dalyaidan1 Aug 17, 2024
39face2
fix: remove unwanted line
dalyaidan1 Aug 18, 2024
aedd984
fix typo
mtrezza Aug 18, 2024
c814b30
Merge branch 'alpha' into parse-file-large-upload
mtrezza Aug 18, 2024
0cf1391
fix: remove unneeded server reconfigure
dalyaidan1 Aug 18, 2024
92b5ec5
fix: actually upload the full files
dalyaidan1 Aug 18, 2024
5bd46b6
fix: encryption chunking
dalyaidan1 Aug 19, 2024
01cce43
test: adjust to test blobs too
dalyaidan1 Aug 19, 2024
f296828
use pipes
dalyaidan1 Aug 19, 2024
73bbdb0
fix: remove web readable
dalyaidan1 Aug 19, 2024
d43a8ad
fix: test errors and clean up pipes
dalyaidan1 Aug 20, 2024
f817c20
fix: remove unused dep
dalyaidan1 Aug 20, 2024
aabae8b
fix: comment typo
dalyaidan1 Aug 20, 2024
07098f9
feat: ternary iv
dalyaidan1 Aug 20, 2024
f09037f
Merge branch 'alpha' into parse-file-large-upload
mtrezza Sep 4, 2024
5c298b3
remove Blob avail check
mtrezza Oct 8, 2024
e276df3
remove blob check
mtrezza Oct 8, 2024
fad46b3
Merge branch 'alpha' into parse-file-large-upload
mtrezza Oct 8, 2024
18157be
change file router to only mongo
dalyaidan1 Nov 2, 2024
a87a452
Merge branch 'alpha' into parse-file-large-upload
mtrezza Nov 3, 2024
7e23668
Merge branch 'alpha' into parse-file-large-upload
mtrezza Nov 7, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
79 changes: 79 additions & 0 deletions spec/FilesRouter.spec.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,79 @@
const fs = require('fs');
const path = require('path');

describe('FilesRouter', () => {
describe_only_db('mongo')('File Uploads', () => {

beforeEach(async () => {
// Set the maxUploadSize to 1GB
await reconfigureServer({
maxUploadSize: '1GB',
});
});

const V8_STRING_LIMIT_BYTES = 536_870_912;

/**
* Quick helper function to upload the file to the server via the REST API
* We do this because creating a Parse.File object with a file over 512MB
* will try to use the Web FileReader API, which will fail the test
*
* @param {string} fileName the name of the file
* @param {string} filePath the path to the file locally
* @returns
*/
const postFile = async (fileName, filePath) => {
const url = `${Parse.serverURL}/files/${fileName}`;
const headers = {
'X-Parse-Application-Id': Parse.applicationId,
'X-Parse-Master-Key': Parse.masterKey,
'Content-Type': 'multipart/form-data',
};

const fileStream = fs.createReadStream(filePath);

// Send the request
const response = await fetch(url, {
method: 'POST',
headers,
body: fileStream,
duplex: 'half' // This is required to send a stream
});

return response;
};

it('should allow Parse.File uploads under 512MB', async done => {
const filePath = path.join(__dirname, 'file.txt');
await fs.promises.writeFile(filePath, Buffer.alloc(1024 * 1024));

const response = await postFile('file.txt', filePath);
expect(response.ok).toBe(true);

fs.unlinkSync(filePath);
done();
});

it('should allow Parse.File uploads exactly 512MB', async done => {
const filePath = path.join(__dirname, 'file.txt');
await fs.promises.writeFile(filePath, Buffer.alloc(V8_STRING_LIMIT_BYTES));

const response = await postFile('file.txt', filePath);
expect(response.ok).toBe(true);

fs.unlinkSync(filePath);
done();
});

it('should allow Parse.File uploads over 512MB', async done => {
const filePath = path.join(__dirname, 'file.txt');
await fs.promises.writeFile(filePath, Buffer.alloc(V8_STRING_LIMIT_BYTES + 50 * 1024 * 1024));

const response = await postFile('file.txt', filePath);
expect(response.ok).toBe(true);

fs.unlinkSync(filePath);
done();
});
});
});
Loading