-
-
Notifications
You must be signed in to change notification settings - Fork 563
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
docs: add vercel example to show a portal to private snowflake api
- Loading branch information
1 parent
5277ba0
commit 12c7109
Showing
7 changed files
with
320 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,2 @@ | ||
*.ticket | ||
.vercel |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,114 @@ | ||
# Call private APIs in Vercel | ||
|
||
![Architecture](./diagram.png) | ||
|
||
|
||
## Snowflake: Setup a private API in Snowpark Container Services | ||
|
||
- [Follow](https://quickstarts.snowflake.com/guide/build_a_private_custom_api_in_python/index.html?index=..%2F..index#0) to setup a private API in Snowpark Container Services | ||
|
||
- Create a database, table and insert some data | ||
|
||
```sql | ||
-- Use the accountadmin role to create the database, table and insert some data | ||
USE ROLE ACCOUNTADMIN; | ||
|
||
-- Grant the DATA_API_ROLE the necessary permissions to use the database and schema | ||
GRANT USAGE ON DATABASE TEST_DATABASE TO ROLE DATA_API_ROLE; | ||
GRANT USAGE ON SCHEMA TEST_DATABASE.PUBLIC TO ROLE DATA_API_ROLE; | ||
|
||
-- Create the table | ||
CREATE OR REPLACE TABLE TEST_DATABASE.PUBLIC.PRODUCTS ( | ||
id INTEGER PRIMARY KEY, | ||
product_name VARCHAR(255), | ||
price NUMBER(10,2) | ||
); | ||
|
||
-- Grant the DATA_API_ROLE the necessary permissions to use the table | ||
GRANT SELECT, INSERT, UPDATE, DELETE ON TABLE TEST_DATABASE.PUBLIC.PRODUCTS TO ROLE DATA_API_ROLE; | ||
|
||
|
||
-- Insert sample data | ||
INSERT INTO TEST_DATABASE.PUBLIC.PRODUCTS (id, product_name, price) | ||
VALUES (1, 'Ergonomic Keyboard', 10.99); | ||
|
||
INSERT INTO TEST_DATABASE.PUBLIC.PRODUCTS (id, product_name, price) | ||
VALUES (2, 'Wireless Mouse', 20.99); | ||
|
||
INSERT INTO TEST_DATABASE.PUBLIC.PRODUCTS (id, product_name, price) | ||
VALUES (3, 'LED Monitor 27"', 30.99); | ||
``` | ||
|
||
- Create two new endpoints `/projects` and `/products/update`. Update `connector.py` to include the new endpoints. | ||
|
||
```python | ||
@connector.route('/products') | ||
def get_products(): | ||
sql = ''' | ||
SELECT id, product_name, price | ||
FROM TEST_DATABASE.public.products | ||
''' | ||
try: | ||
res = conn.cursor(DictCursor).execute(sql) | ||
return make_response(jsonify(res.fetchall())) | ||
except: | ||
abort(500, "Error reading from Snowflake. Check the logs for details.") | ||
|
||
@connector.route('/products/update', methods=['POST']) | ||
def update_products(): | ||
sql = ''' | ||
UPDATE TEST_DATABASE.public.products | ||
SET price = ROUND(UNIFORM(10, 100, RANDOM()), 2) | ||
''' | ||
try: | ||
conn.cursor().execute(sql) | ||
conn.commit() | ||
return make_response(jsonify({"message": "Products prices updated successfully"}), 200) | ||
except: | ||
conn.rollback() | ||
abort(500, "Error updating prices in Snowflake. Check the logs for details.") | ||
``` | ||
|
||
- Rebuild and publish the dataapi image (`/api/private/api/dataapi`) with new endpoints. Drop and create `API.PRIVATE.API` service with new image. | ||
|
||
|
||
## Vercel: Setup Vercel Serverless functions to access the private API. | ||
|
||
### Download latest ockam binary | ||
|
||
- Download ockam binary (x86_64-unknown-linux-gnu) from the [Ockam](https://github.com/build-trust/ockam/releases) github repository and place it in the `data/linux-x86_64` directory. Rename the binary to `ockam`. | ||
|
||
### Create an enrollment ticket for the Vercel function | ||
|
||
```sh | ||
# Generate an inlet ticket for the Vercel function. | ||
ockam project ticket --usage-count 100 --expires-in 10h --attribute snowflake-api-service-inlet > vercel-inlet.ticket | ||
``` | ||
|
||
### Setup a Vercel project and add the inlet ticket as a secret | ||
|
||
- Select `Project Settings`, `Environments`, `Production | ||
- Click `Add Environment Variable` | ||
- Select `Sensitive` | ||
- Add `OCKAM_SNOWFLAKE_INLET_ENROLLMENT_TICKET` as the name | ||
- Add the value of `vercel-inlet.ticket` as the value | ||
- Click `Add` | ||
|
||
### Deploy the Vercel function | ||
|
||
- Setup vercel cli and select the project | ||
- Deploy the function inside the `api` directory | ||
|
||
```sh | ||
vercel --prod | ||
``` | ||
|
||
### Test the Vercel function | ||
|
||
- Use the `/api` endpoint to get the products | ||
- Use the `/api/update` endpoint to update the products | ||
|
||
|
||
## Setup a Next.js frontend deployed to Vercel to access vercel function | ||
|
||
- Optionally you can setup a Next.js frontend to access the vercel function via it's public URLs. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,182 @@ | ||
""" | ||
Simple API handler for accessing Snowflake private API endpoints via Ockam portal | ||
""" | ||
|
||
import json | ||
import os | ||
import subprocess | ||
import threading | ||
import time | ||
from http.server import BaseHTTPRequestHandler | ||
import requests | ||
|
||
# Constants | ||
OCKAM_BINARY_PATH = '/var/task/data/linux-x86_64/ockam' if os.environ.get('VERCEL_ENV') == 'production' else 'ockam' | ||
MAX_RETRIES = 60 | ||
RETRY_DELAY = 1 # seconds | ||
LOCAL_API_ENDPOINT = "http://127.0.0.1:8081" | ||
REQUEST_TIMEOUT = 3 # seconds | ||
|
||
# Track if Ockam node creation has been attempted | ||
# Only one Ockam node should run per serverless function runtime - this flag prevents multiple node creation attempts. | ||
NODE_INITIALIZED = False | ||
|
||
def is_production() -> bool: | ||
"""Check if the environment is production.""" | ||
return os.environ.get('VERCEL_ENV') == 'production' | ||
|
||
def get_ockam_version() -> str: | ||
""" | ||
Get the installed Ockam version. | ||
Returns version string or error message. | ||
""" | ||
try: | ||
ockam_path = OCKAM_BINARY_PATH | ||
if is_production() and not os.path.exists(ockam_path): | ||
return f"Error: Ockam binary not found at {os.path.abspath(ockam_path)}" | ||
|
||
result = subprocess.run([ockam_path, '--version'], capture_output=True, text=True) | ||
|
||
return result.stdout.strip() if result.returncode == 0 else f"Error running ockam: {result.stderr}" | ||
except Exception as e: | ||
return f"Error: {str(e)} (CWD: {os.getcwd()})" | ||
|
||
class handler(BaseHTTPRequestHandler): | ||
"""Handler for Snowflake API requests via Ockam secure channel.""" | ||
|
||
def create_ockam_node(self, enrollment_ticket: str) -> bool: | ||
""" | ||
Initialize Ockam node with the provided enrollment ticket. | ||
Returns True if successful or already attempted, False otherwise. | ||
""" | ||
if not is_production(): | ||
return False | ||
|
||
global NODE_INITIALIZED | ||
request_id = self.headers.get('x-vercel-id', os.urandom(8).hex()) | ||
|
||
print(f"[{request_id}] Starting with NODE_INITIALIZED = {NODE_INITIALIZED}") | ||
|
||
if not NODE_INITIALIZED: | ||
NODE_INITIALIZED = True | ||
|
||
def run_ockam_node(): | ||
try: | ||
config = '''{ | ||
"tcp-inlet": { | ||
"from": "127.0.0.1:8081", | ||
"via": "snowflake-api-service-relay", | ||
"allow": "snowflake-api-service-outlet" | ||
} | ||
}''' | ||
|
||
print(f"[{request_id}] Starting ockam node with config: {config}", flush=True) | ||
|
||
subprocess.Popen([ | ||
OCKAM_BINARY_PATH, | ||
'node', | ||
'create', | ||
'--configuration', | ||
config, | ||
'--enrollment-ticket', | ||
enrollment_ticket.strip() | ||
], env={ | ||
**os.environ, | ||
'OCKAM_HOME': '/tmp', | ||
'OCKAM_OPENTELEMETRY_EXPORT': 'false', | ||
'OCKAM_DISABLE_UPGRADE_CHECK': 'true' | ||
}) | ||
|
||
except Exception as e: | ||
print(f"[{request_id}] Error in node creation: {str(e)}", flush=True) | ||
|
||
thread = threading.Thread(target=run_ockam_node) | ||
thread.daemon = True | ||
print(f"[{request_id}] Starting ockam node thread", flush=True) | ||
thread.start() | ||
|
||
return True | ||
|
||
def handle_select(self, request_id: str) -> dict: | ||
"""Handle GET requests for product data.""" | ||
for retry in range(MAX_RETRIES): | ||
try: | ||
response = requests.get( | ||
f"{LOCAL_API_ENDPOINT}/connector/products", | ||
timeout=REQUEST_TIMEOUT | ||
) | ||
response.raise_for_status() | ||
print(f"[{request_id}] Connected successfully after {retry} retries") | ||
return {"status": "success", "data": response.json()} | ||
|
||
except Exception as e: | ||
print(f"[{request_id}] Attempt {retry + 1} failed: {str(e)}") | ||
if retry < MAX_RETRIES - 1: | ||
time.sleep(RETRY_DELAY) | ||
else: | ||
return {"status": "error", "message": f"Connection failed: {str(e)}"} | ||
|
||
def handle_update(self, request_id: str) -> dict: | ||
"""Handle POST requests for product updates.""" | ||
for retry in range(MAX_RETRIES): | ||
try: | ||
response = requests.post( | ||
f"{LOCAL_API_ENDPOINT}/connector/products/update", | ||
timeout=REQUEST_TIMEOUT | ||
) | ||
response.raise_for_status() | ||
print(f"[{request_id}] Update successful after {retry} retries") | ||
|
||
updated_values = self.handle_select(request_id) | ||
return { | ||
"status": "success", | ||
"update_result": response.json(), | ||
"current_values": updated_values["data"] | ||
} | ||
|
||
except Exception as e: | ||
print(f"[{request_id}] Attempt {retry + 1} failed: {str(e)}") | ||
if retry < MAX_RETRIES - 1: | ||
time.sleep(RETRY_DELAY) | ||
else: | ||
return {"status": "error", "message": f"Update failed: {str(e)}"} | ||
|
||
def do_GET(self): | ||
"""Handle incoming GET requests.""" | ||
try: | ||
# Verify Ockam installation | ||
version_result = get_ockam_version() | ||
if 'Error' in version_result: | ||
self._send_error(500, version_result) | ||
return | ||
|
||
# Verify enrollment ticket | ||
enrollment_ticket = os.environ.get('OCKAM_SNOWFLAKE_INLET_ENROLLMENT_TICKET') | ||
if not enrollment_ticket: | ||
self._send_error(500, 'OCKAM_SNOWFLAKE_INLET_ENROLLMENT_TICKET not configured') | ||
return | ||
|
||
# Initialize Ockam node | ||
if not self.create_ockam_node(enrollment_ticket): | ||
self._send_error(500, 'Failed to initialize Ockam node') | ||
return | ||
|
||
# Handle the request | ||
request_id = self.headers.get('x-vercel-id', os.urandom(8).hex()) | ||
result = self.handle_update(request_id) if self.path == '/api/update' else self.handle_select(request_id) | ||
|
||
self._send_response(200, result) | ||
|
||
except Exception as e: | ||
self._send_error(500, str(e)) | ||
|
||
def _send_response(self, status: int, data: dict): | ||
"""Helper method to send JSON responses.""" | ||
self.send_response(status) | ||
self.send_header('Content-Type', 'application/json') | ||
self.end_headers() | ||
self.wfile.write(json.dumps(data).encode()) | ||
|
||
def _send_error(self, status: int, message: str): | ||
"""Helper method to send error responses.""" | ||
self._send_response(status, {'error': message}) |
Empty file.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,3 @@ | ||
Flask==3.0.3 | ||
psycopg2-binary==2.9.9 | ||
requests==2.31.0 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,19 @@ | ||
{ | ||
"redirects": [ | ||
{ | ||
"source": "/", | ||
"destination": "/api" | ||
} | ||
], | ||
"rewrites": [ | ||
{ | ||
"source": "/api/update", | ||
"destination": "/api" | ||
} | ||
], | ||
"functions": { | ||
"api/**/*": { | ||
"maxDuration": 300 | ||
} | ||
} | ||
} |