Translation AI by Sync Labs

Translation AI is an open-source tool for translating spoken language in video content with accurate lip synchronization. The purpose of this repo is to help developers quickly integrate multilingual support into video-based apps. This project provides essential APIs and documentation to facilitate the development of applications requiring video translation with lip-sync capabilities.

How it works

• Sync Labs for perfectly synchronized lip movements
• Gladia for transcribing and diarization
• Open AI for translating
• Eleven Labs for voice cloning and speech synthesis
• Next.js for web app
• Vercel for deployment
• Supabase for db, auth and storage
• Stripe for billing

Live demo

Getting set up

To successfully deploy the template, execute the following steps in order:

1. Initiate Deployment

Clicking the Deploy button will open up a browser tab with instructions on generating a new GitHub repository from this template and creating a corrseponding project in Supabase for auth and storage. Below are the steps you'll walk through:

1. Create Git Repository

   • Under Git Scope select the account you'd like to deploy the repo to.
   • Under Repository Name give your new repo a name.

2. Add Integrations

   The Supabase Vercel Deploy Integration handles configuration of environment variables and initiates the SQL migrations to construct your database schema. You can view the resulting tables in your project's Table editor.

   • Click Add in the Supabase integration and fill out the form that pops up.

3. Configure Project

   Enter the environment variables required for this project. Below are links to documentation on how to obtain an API key for each of the required services to ensure full functionality.

   • Sync Labs API Key
   • Open AI
   • Gladia
   • Eleven Labs
   • Stripe

2. Configure Auth

You'll need to configure your Supabase project's site URL and Vercel's NEXT_PUBLIC_SITE_URL environment variable to secure and streamline authentication.

1. In your Supabase project, navigate to Authentication > URL configuration and enter your production URL (for example, https://your-deployment-url.vercel.app) as the site URL.

2. In Vercel, under your project's deployment settings, create a new Production environment variable called NEXT_PUBLIC_SITE_URL and set it to your production URL. Make sure you uncheck the options for preview and development environments to maintain the correct operation for preview branches and local development.

3. Configure Storage

After that, you'll need to create a new storage bucket within your Supabase project to store video and audio files.

1. In your Supabase project, navigate to Storage and click the New bucket button.

2. Enter translation as the name for your new bucket and toggle Public bucket to on.

3. Click Save

4. Still in the Storage section of your Supabase project, click Policies and add the below policy:

• Within the section called translation (the name of your bucket) click New policy then For full costumization and fill it in as shown below:

  

• Within the section called Other policies under storage.objects add the following three policies:

  

  

  

4. Configure Stripe

Next, you'll need to configure Stripe in order to handle test payments.

Initial setup

1. Create a Stripe account if you don't have one yet
2. Ensure you are in "Test Mode" by toggling the Test Mode switch at the top of the dashboard to the ON position.

Create a webhook

1. In Stripe's Developers section, navigate to the Webhooks tab and click Add endpoint
2. Under Endpoint URL enter your deployment URL with /api/webhooks appended to the end (e.g., https://your-deployment-url.vercel.app/api/webhooks).
3. Under Select events to listen to choose Select events, toggle on Select all events and then click Add events
4. Scroll to the bottom of the form and select Add endpoint
5. In the page that appears, under Signing secret click reveal and copy the secret.
6. Add this Signing secret as the STRIPE_WEBHOOK_SECRET environment variable in your Vercel settings.

Redeploy with Updated Environment Variables

Redeploy your application in Vercel for the new environment variables to activate. Go to your Vercel Dashboard, find the Deployments section, and choose Redeploy from the overflow menu. Ensure "Use existing Build Cache" is not selected.

Create product(s)

With the Stripe webhook setup, you can start creating your products in the Stripe Dashboard. Stripe Checkout supports billing a set amount at regular intervals. It does not support complex pricing structures like tiered pricing or per-seat billing.

Configure the customer portal

• Set up branding in the Branding settings
• Set up the customer portal in the Customer Portal settings

Running locally

1. Clone your GitHub repository to your local machine

2. Link your project using the Vercel CLI

In a terminal navigate to your project's root directory and enter the following command.

vercel login
vercel link

3. Set up local environment variables

Setting up the env vars locally

In a terminal navigate to your project's root directory and enter the following command.

vercel env pull .env.local

This command will create a new .env.local file in your project folder.

4. Test webhooks using the Stripe CLI

• Install the Stripe CLI
• Link your Stripe account.

In a seperate terminal, run the below command to initiate local webhook forwarding

stripe listen --forward-to=localhost:3000/api/webhooks

Running this Stripe command will print a webhook secret (whsec_***), to the console. Set STRIPE_WEBHOOK_SECRET to this value in your .env.local file.

5. Install dependencies and run the project

In a separate terminal, navigate to your project's root directory and run the following commands to install dependencies and start the dev server

yarn
yarn dev

6. You've done it!

Open your web browser and visit http://localhost:3000 to view your application.

How to Contribute

We embrace the collaborative spirit of the open-source community.

To suggest enhancements or submit changes:

1. Fork the repository.
2. Create a new branch from dev for your changes.
3. Commit your updates and push to your fork.
4. Submit a pull request back to the dev branch of the original repository.

Your contributions are highly appreciated!

Resources

• Sync Labs Slack Community
• Sync Labs Docs

License

Translation AI is released under the MIT License.