Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Insert all on a collection with the same timestamp #2712

Open
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

jyeshe
Copy link
Member

@jyeshe jyeshe commented Nov 25, 2024

Description

This PR changes the inserted_at set when calling Repo.insert_all to have as per requirement the same value for all records.
In order to adhere to the order given by the user on the insert, the pagination and cursor are changed to use a sequence number.

Closes #2711

Validation steps

  1. Send a POST to /collections/new-collection-name with a JSON body having the items field.
  2. Send a GET to /collections/new-collection-name and check that createdAt are equal and the order of the keys is the same as on the insert.

AI Usage

Please disclose how you've used AI in this work (it's cool, we just want to know!):

  • Code generation (copilot but not intellisense)
  • Learning or fact checking
  • Strategy / design
  • Optimisation / refactoring
  • Translation / spellchecking / doc gen
  • Other
  • I have not used AI

You can read more details in our Responsible AI Policy

Pre-submission checklist

  • I have performed a self-review of my code.
  • I have implemented and tested all related authorization policies. (e.g., :owner, :admin, :editor, :viewer)
  • I have updated the changelog.
  • I have ticked a box in "AI usage" in this PR

@jyeshe jyeshe force-pushed the paginate-by-serial branch 2 times, most recently from b8cc228 to 9dfe78f Compare November 25, 2024 16:26
@jyeshe jyeshe changed the title Add all records with same timestamp Insert all on a collection with the same timestamp Nov 25, 2024
@jyeshe jyeshe force-pushed the paginate-by-serial branch 3 times, most recently from b5fbea6 to f456fc8 Compare November 25, 2024 16:36
@jyeshe
Copy link
Member Author

jyeshe commented Nov 25, 2024

Test failed due to unrelated flaky
image

@jyeshe jyeshe marked this pull request as ready for review November 25, 2024 16:48
@jyeshe
Copy link
Member Author

jyeshe commented Nov 26, 2024

@stuartc thanks for asking, here are the stats for inserting 300 rounds of 2 parallel put_all (2 collections) with 5000 items each put_all:

Average: 516.225ms
Std Deviation: 363.52131213314027ms

database docker (localhost):

docker create --cpus=2.0 --memory=8g --name lightning-postgres --mount source=lightning-postgres-data,target=/var/lib/postgresql/data   --publish 5432:5432   -e POSTGRES_PASSWORD=postgres   postgres:15.3-alpine

disk perf:

$ dd if=/dev/zero of=/tmp/test.img bs=10MB count=100
100+0 records in
100+0 records out
1000000000 bytes (1,0 GB, 954 MiB) copied, 3,26877 s, 306 MB/s

Without the two last indexes (on updated_at and id) not much difference on a standalone, without any other load:

Average: 474.08666666666664ms
Std Deviation: 323.6109894439448ms

Copy link

codecov bot commented Nov 26, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 91.05%. Comparing base (7fcc9cb) to head (a7824f0).

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #2712      +/-   ##
==========================================
- Coverage   91.06%   91.05%   -0.01%     
==========================================
  Files         333      333              
  Lines       11708    11708              
==========================================
- Hits        10662    10661       -1     
- Misses       1046     1047       +1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: New Issues
Development

Successfully merging this pull request may close these issues.

Paginate collection indirectly by insertion using a serial
1 participant