Skip to content

Commit

Permalink
add replication snippet
Browse files Browse the repository at this point in the history
  • Loading branch information
flarco committed Jul 1, 2024
1 parent 2035ce9 commit ea3fca5
Showing 1 changed file with 44 additions and 24 deletions.
68 changes: 44 additions & 24 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,37 +21,57 @@ sling run --src-conn MY_PG --src-stream myschema.mytable \
Or passing a yaml/json string or file

```shell
sling run -c '
source:
conn: MY_PG
stream: myschema.mytable
target:
conn: YOUR_SNOWFLAKE
object: yourschema.yourtable
mode: full-refresh
'
# OR
sling run -c /path/to/config.json
cat '
source: MY_POSTGRES
target: MY_SNOWFLAKE
# default config options which apply to all streams
defaults:
mode: full-refresh
object: new_schema.{stream_schema}_{stream_table}
streams:
my_schema.*:
' > /path/to/replication.yaml

sling run -r /path/to/replication.yaml
```

### From Lib

Run a replication from file:

```python
from sling import Sling
import yaml
from sling import Replication

config = {
'source': {
'conn': 'MY_PG',
'stream': "select * from my_table",
},
'target': {
'conn': "s3://my_bucket/my_folder/new_file.csv",
},
}
with open('path/to/replication.yaml') as file:
config = yaml.load(file, Loader=yaml.FullLoader)

replication = Replication(**config)

replication.run()
```

Build a replication dynamically:

Sling(**config).run()
```python
from sling import Replication, ReplicationStream

# build sling replication
streams = {}
for (folder, table_name) in list(folders):
streams[folder] = ReplicationStream(mode='full-refresh', object=table_name, primary_key='_hash_id')

replication = Replication(
source='aws_s3',
target='snowflake',
streams=streams,
env=dict(SLING_STREAM_URL_COLUMN='true', SLING_LOADED_AT_COLUMN='true'),
debug=True,
)

replication.run()
```

## Config Schema
Expand Down

0 comments on commit ea3fca5

Please sign in to comment.