Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

README updates #149

Merged
merged 1 commit into from
Oct 5, 2023
Merged

README updates #149

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
101 changes: 60 additions & 41 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,33 +8,40 @@ Serval is a REST API for natural language processing services.

## Development

### CSharpier

All C# code should be formatted using [CSharpier](https://csharpier.com/). The best way to enable support for CSharpier is to install the appropriate [IDE extension](https://csharpier.com/docs/Editors) and configure it to format on save.

### Coding conventions
### Development in Docker Compose

[Here](https://learn.microsoft.com/en-us/dotnet/csharp/fundamentals/coding-style/identifier-names) is a good overview of naming conventions. [Here](https://learn.microsoft.com/en-us/dotnet/csharp/fundamentals/coding-style/coding-conventions) is a good overview of coding conventions. If you want to get in to even more detail, check out the [Framework design guidelines](https://learn.microsoft.com/en-us/dotnet/standard/design-guidelines/).
This is the simplest way to develop.
First, git clone the repo:
```
git clone https://github.com/sillsdev/serval.git
```
Then build Serval:
```
dotnet build serval
```
Clone the [Machine repo](https://github.com/sillsdev/machine) into an adjacent folder to Serval:
```
git clone https://github.com/sillsdev/machine.git
```
Build the Machine repo:
```
dotnet build machine
```
Make sure that the environment variable ASPNETCORE_ENVIRONMENT is set to "Development" by running `export ASPNETCORE_ENVIRONMENT=Development` or adding it to your `.bashrc`.
In the Serval root, run docker compose up
```
cd serval && docker compose up
```
If using vscode, launch "DockerComb" to debug Serval and the for-testing-only Echo Engine.

### Development locally
Alternatively, you can develop without containerizing Serval.

- Install MongoDB 6.0 as a replca set and MongoDBCompass and run it on localhost:27017
- set the following environment variables:
- ASPNETCORE_ENVIRONMENT=Development
- Open "Serval.sln" and debug the ApiServer
- Now, you are running the complete environment where everything is being debugged and the mongodb is exposed.
Install MongoDB 6.0 as a replica set run it on localhost:27017. (You can run `docker compose -f "docker-compose.mongo.yaml" up` from the root of the serval repo to do so).

#### local dev in ubuntu
* Make sure you install dotnet from the microsoft apr-get repo, otherwise debugging doesn't work! Also you will need to "searchMicrosoftSymbolServer" for the symbols.
* To have a local MongoDB, use the docker-compose script to create a replica set of one instance.
Make sure that the environment variable ASPNETCORE_ENVIRONMENT is set to "Development" by running `export ASPNETCORE_ENVIRONMENT=Development` or adding it to your `.bashrc`.

### Development in Docker Compose

* Build this repository with dotnet build
* Download this repository and place the https://github.com/sillsdev/machine repo in ../machine, relative to this repo.
* Build the machine repo with dotnet build in the root of that repo
* In the serval root, run docker-compose up
* To debug serval and echo together, launch "ServalComb" in VSCode
Open "Serval.sln" and debug the ApiServer.

## Deployment on kubernetes
There are 3 different environments that Serval is deployed to:
Expand All @@ -52,7 +59,7 @@ There are 3 different environments that Serval is deployed to:
- `helm install serval deploy/serval -n nlp -f deploy/qa-int-values.yaml`

### To update the cluster
- To upgrade serval:
- To upgrade Serval:
- For QA internal Run:
- `kubectl config use-context dallas-rke`
- `helm upgrade serval deploy/serval -n nlp -f deploy/qa-int-values.yaml`
Expand All @@ -63,38 +70,50 @@ There are 3 different environments that Serval is deployed to:
- `kubectl config use-context aws-rke`
- `helm upgrade serval deploy/serval -n serval -f deploy/values.yaml`

## Debugging
### To access Serval API

* Internal QA:
* Use the VPN
* In `C:\Windows\System32\drivers\etc\hosts`, enter in a line for `10.3.0.119 serval-api.org`
* go to `https://machine-api.org/swagger` and accept the security warning
* External QA:
* go to `https://qa.serval-api.org/swagger` and accept the security warning

### Pod logs:

### To view pod logs:
- Run: `kubectl get pods` to get the currently running pods
- Run: `kubectl logs <pod name>`
- Run: `kubectl describe pod <pod name>` to check a stalled pod stuck in containercreating

### API E2E Testing
- Get auth0 token using curl:
- Get Client ID and Client Secret from auth0.com
- Login, go to Applications-> Applications -> "Machine API (Test Application)", or similar
- Copy `Client ID` into Environment variable `SERVAL_CLIENT_ID`
- Copy `Client Secret` into Environment variable `SERVAL_CLIENT_SECRET`
- Copy the auth0 url into Environment variable `SERVAL_AUTH_URL` (i.e. `SERVAL_AUTH_URL=https://sil-appbuilder.auth0.com`)
- Set `SERVAL_HOST_URL` to the api's URL (i.e. `SERVAL_HOST_URL=http://localhost`)
- Run tests from `Serval.E2ETests`
- The token will automatically be retrieved from Auth0 when you run the tests
### Running the API E2E Tests
In order to run the E2E tests, you will need to have the appropriate credentials
- Get Client ID and Client Secret from auth0.com
- Login, go to Applications-> Applications -> "Machine API (Test Application)" or similar
- Copy `Client ID` into Environment variable `SERVAL_CLIENT_ID`
- Copy `Client Secret` into Environment variable `SERVAL_CLIENT_SECRET`
- Copy the auth0 url into Environment variable `SERVAL_AUTH_URL` (e.g. `SERVAL_AUTH_URL=https://sil-appbuilder.auth0.com`)
- Set `SERVAL_HOST_URL` to the api's URL (e.g. `SERVAL_HOST_URL=http://localhost`)
Now, when you run the tests from `Serval.E2ETests`, the token will automatically be retrieved from Auth0.

### Debugging the S3 bucket

To view files stored in the bucket, run
```
aws s3 ls s3://aqua-ml-data/<deployment environment>
```
### Mongo debugging

* First, get the mongo pod name: `kubectl get pods -n serval`
* Then forward to a local port, such as 28015: `kubectl port-forward <pod name> 28015:27017 -n serval`
* Download [MongoDB Compass](https://www.mongodb.com/try/download/compass).
* Then, open MongoDB Compass and connect to `mongodb://localhost:28015/?directConnection=true`


### Access S3 bucket
## Development Notes
### CSharpier

- to view files, run `aws s3 ls s3://aqua-ml-data/<deployment environment>`
All C# code should be formatted using [CSharpier](https://csharpier.com/). The best way to enable support for CSharpier is to install the appropriate [IDE extension](https://csharpier.com/docs/Editors) and configure it to format on save.

### Mongo investigation
### Coding conventions

[Here](https://learn.microsoft.com/en-us/dotnet/csharp/fundamentals/coding-style/identifier-names) is a good overview of naming conventions. [Here](https://learn.microsoft.com/en-us/dotnet/csharp/fundamentals/coding-style/coding-conventions) is a good overview of coding conventions. If you want to get in to even more detail, check out the [Framework design guidelines](https://learn.microsoft.com/en-us/dotnet/standard/design-guidelines/).

* First, get the pod name: `kubectl get pods -n serval`
* Then forward to local port, such as 28015: `kubectl port-forward <pod name> 28015:27017 -n serval`
* Then open MongoDB Compass and connect to `mongodb://localhost:28015/?directConnection=true`