Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

(Source) - Couchbase connector 🎉 #1

Draft
wants to merge 2,369 commits into
base: master
Choose a base branch
from
Draft

(Source) - Couchbase connector 🎉 #1

wants to merge 2,369 commits into from

Conversation

teetangh
Copy link
Collaborator

@teetangh teetangh commented Sep 24, 2024 •
edited
Loading

What

  • Implementation of a Couchbase source connector that enables data extraction from Couchbase databases
  • Supports both full refresh and incremental sync modes
  • Handles authentication and connection management with Couchbase clusters
  • Implements efficient querying using SQL++ with pagination support

How

  • Uses Python CDK's AbstractSource for the main connector implementation
  • Implements connection checking using Couchbase SDK's cluster connection
  • Utilizes SQL++ queries with parameterized statements for secure data access
  • Handles incremental sync using configurable cursor fields

Review guide

  1. source.py - Main connector implementation, connection handling, and stream configuration
  2. streams.py - Stream implementations for full refresh and incremental sync modes
  3. queries.py - SQL++ query construction and execution logic
  4. Integration tests for verifying connector behavior with actual Couchbase instance
  5. Unit tests covering core functionality and edge cases

User Impact

  • Users can reliably sync data from Couchbase databases to any Airbyte destination
  • Clear error messages help with troubleshooting
  • Incremental sync support reduces data transfer and improves efficiency

Can this PR be safely reverted and rolled back?

  • YES 💚
  • The connector is self-contained and doesn't have dependencies on other system components
  • No database migrations or schema changes are required
  • Reverting would simply disable the Couchbase source functionality without affecting other connectors

@teetangh teetangh self-assigned this Sep 24, 2024
@teetangh teetangh requested a review from shyam-cb September 24, 2024 14:04
teetangh and others added 30 commits December 16, 2024 09:23
@teetangh
Merge branch 'master' into DA-547-airbyte-couchbase-source
25c5759
@williamkaper @maxi297
✨ Source-Stripe: Add new payout_balance_transactions incremental stre…
7d91ed7 
…am (airbytehq#46499)

Co-authored-by: maxi297 <[email protected]>
@rodireich @tomnz @octavia-squidington-iii
Simplify CTID_TABLE_BLOCK_SIZE query for Postgres integration (airbyt…
3af71ab 
…ehq#48126) (airbytehq#49469)

Co-authored-by: Tom <[email protected]>
Co-authored-by: Octavia Squidington III <[email protected]>
@subodh1810
fix: iceberg timestamp value handling (airbytehq#49807)
369cde7
@johnny-schmidt
Bulk Load CDK: Queue between process records & batch; fixed num batch… (
4f24507 
airbytehq#49372)
@airbyteio
🐙 source-google-analytics-data-api: run up-to-date pipeline [2024-12-…
052be6f 
…14] (airbytehq#48649)
@williamkaper
🐛Source-Stripe - Fix for Customer_Balance_Transaction records occasio…
559cac7 
…nally getting missed (airbytehq#49811)
@ChristoGrab @matteogp
docs: add enterprise docs for incubating service-now connector (airby…
24f95a9 
…tehq#49821)

Co-authored-by: matteogp <[email protected]>
@aldogonzalez8
✨feat(source-s3): Increase individual file size limit to 1.5 GB (airb…
db035e1 
…ytehq#49824)
@aldogonzalez8
✨feat(source-sftp-bulk): increase individual file size limit to 1.5 GB (
078b2f8 
airbytehq#49826)
@LouisAuneau @marcosmarxm
✨ source-pennylane: update customer_invoices stream schema (airbyte…
5ddf487 
…hq#48892)

Co-authored-by: Marcos Marx <[email protected]>
@darynaishchenko
feat(source-pipedrive): update cdk to enable concurrency (airbytehq#4…
c27161b 
…8615)
@rodireich
Send a correct fallback value for autogenerated boolean (airbytehq#49830
dfcdea8 
)
@jdpgrailsdev
fix: return correct language tag when using Kotlin build script (airb…
6d8a3a2 
…ytehq#49835)
@jdpgrailsdev
chore: Skeleton project for destination MSSQL V2 (airbytehq#49460)
4dc13b3
@ChristoGrab @justbeez @DanyloGL
feat(source-pardot): 23 new streams, fixed auth flow (airbytehq#49424)
b7e25c9 
Co-authored-by: Justin Beasley <[email protected]>
Co-authored-by: Danylo Jablonski <[email protected]>
@yardencarmeli
Updating CDC sources documentation to clarify incremental sync method…
6341ead 
… limitation. (airbytehq#49823)
@alafanechere
base-images: release a base image for our java connectors (airbytehq#…
3f4e4d6 
…49831)
@alafanechere
airbyte-ci: use the base image to build java connectors (airbytehq#49832
8ad2776 
)
@ian-at-airbyte
Inline code uses correct color in a table's header row (airbytehq#49828)
9f819a1
@johnny-schmidt
S3V2: Increase memory overhead ratio (airbytehq#48812)
9bff367
@btkcodedev @octavia-squidington-iii
@ChristoGrab @DanyloGL
Source Intercom: Migrate to manifest only format with components (air…
39abff0 
…bytehq#47240)

Co-authored-by: Octavia Squidington III <[email protected]>
Co-authored-by: ChristoGrab <[email protected]>
Co-authored-by: Danylo Jablonski <[email protected]>
Co-authored-by: Christo Grabowski <[email protected]>
@postamar
source-mysql: better datatype integration tests (airbytehq#49844)
b9ea48e
@brian-pinwheel @marcosmarxm
fix(source-zendesk-support) check start_date exists during check op…
28c7e9a 
…eration (airbytehq#48893)

Co-authored-by: Marcos Marx <[email protected]>
Co-authored-by: marcosmarxm <[email protected]>
@JoeriSmits @natikgadzhi
source-hoorayhr contribution from JoeriSmits (airbytehq#49836)
c0f8679 
Co-authored-by: Natik Gadzhi <[email protected]>
@airbyteio
🐙 source-mixmax: run up-to-date pipeline [2024-12-14] (airbytehq#49604)
8ba7ecb
@johnny-schmidt
Load CDK: S3V2 processes in processRecords, uploads in processBatch (a…
6118419 
…irbytehq#49819)
@alafanechere
source-postgres: Use airbyte/java-connector-base:1.0.0 (airbytehq#49838)
1d54d2a
@alafanechere
source-mssql: Use airbyte/java-connector-base:1.0.0 (airbytehq#49840)
4be7f1a
@teetangh
Merge branch 'master' into DA-547-airbyte-couchbase-source
eb1c6b0
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@shyam-cb shyam-cb Awaiting requested review from shyam-cb

Assignees

@teetangh teetangh

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.