Skip to content

Commit

Permalink
Reorganization of README and minor changes to formatting for ease of …
Browse files Browse the repository at this point in the history
…use and consistency

After helping Matthew get Serval running, I've realized that although the README has a lot of information, it's not organized as helpfully as it could be.
  • Loading branch information
Enkidu93 committed Oct 4, 2023
1 parent e8f0826 commit 91b609e
Showing 1 changed file with 60 additions and 41 deletions.
101 changes: 60 additions & 41 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,33 +8,40 @@ Serval is a REST API for natural language processing services.

## Development

### CSharpier

All C# code should be formatted using [CSharpier](https://csharpier.com/). The best way to enable support for CSharpier is to install the appropriate [IDE extension](https://csharpier.com/docs/Editors) and configure it to format on save.

### Coding conventions
### Development in Docker Compose

[Here](https://learn.microsoft.com/en-us/dotnet/csharp/fundamentals/coding-style/identifier-names) is a good overview of naming conventions. [Here](https://learn.microsoft.com/en-us/dotnet/csharp/fundamentals/coding-style/coding-conventions) is a good overview of coding conventions. If you want to get in to even more detail, check out the [Framework design guidelines](https://learn.microsoft.com/en-us/dotnet/standard/design-guidelines/).
This is the simplest way to develop.
First, git clone the repo:
```
git clone https://github.com/sillsdev/serval.git
```
Then build Serval:
```
dotnet build serval
```
Clone the [Machine repo](https://github.com/sillsdev/machine) into an adjacent folder to Serval:
```
git clone https://github.com/sillsdev/machine.git
```
Build the Machine repo:
```
dotnet build machine
```
Make sure that the environment variable ASPNETCORE_ENVIRONMENT is set to "Development" by running `export ASPNETCORE_ENVIRONMENT=Development` or adding it to your `.bashrc`.
In the Serval root, run docker compose up
```
cd serval && docker compose up
```
If using vscode, launch "DockerComb" to debug Serval and the for-testing-only Echo Engine.

### Development locally
Alternatively, you can develop without containerizing Serval.

- Install MongoDB 6.0 as a replca set and MongoDBCompass and run it on localhost:27017
- set the following environment variables:
- ASPNETCORE_ENVIRONMENT=Development
- Open "Serval.sln" and debug the ApiServer
- Now, you are running the complete environment where everything is being debugged and the mongodb is exposed.
Install MongoDB 6.0 as a replica set run it on localhost:27017. (You can run `docker compose -f "docker-compose.mongo.yaml" up` from the root of the serval repo to do so).

#### local dev in ubuntu
* Make sure you install dotnet from the microsoft apr-get repo, otherwise debugging doesn't work! Also you will need to "searchMicrosoftSymbolServer" for the symbols.
* To have a local MongoDB, use the docker-compose script to create a replica set of one instance.
Make sure that the environment variable ASPNETCORE_ENVIRONMENT is set to "Development" by running `export ASPNETCORE_ENVIRONMENT=Development` or adding it to your `.bashrc`.

### Development in Docker Compose

* Build this repository with dotnet build
* Download this repository and place the https://github.com/sillsdev/machine repo in ../machine, relative to this repo.
* Build the machine repo with dotnet build in the root of that repo
* In the serval root, run docker-compose up
* To debug serval and echo together, launch "ServalComb" in VSCode
Open "Serval.sln" and debug the ApiServer.

## Deployment on kubernetes
There are 3 different environments that Serval is deployed to:
Expand All @@ -52,7 +59,7 @@ There are 3 different environments that Serval is deployed to:
- `helm install serval deploy/serval -n nlp -f deploy/qa-int-values.yaml`

### To update the cluster
- To upgrade serval:
- To upgrade Serval:
- For QA internal Run:
- `kubectl config use-context dallas-rke`
- `helm upgrade serval deploy/serval -n nlp -f deploy/qa-int-values.yaml`
Expand All @@ -63,38 +70,50 @@ There are 3 different environments that Serval is deployed to:
- `kubectl config use-context aws-rke`
- `helm upgrade serval deploy/serval -n serval -f deploy/values.yaml`

## Debugging
### To access Serval API

* Internal QA:
* Use the VPN
* In `C:\Windows\System32\drivers\etc\hosts`, enter in a line for `10.3.0.119 serval-api.org`
* go to `https://machine-api.org/swagger` and accept the security warning
* External QA:
* go to `https://qa.serval-api.org/swagger` and accept the security warning

### Pod logs:

### To view pod logs:
- Run: `kubectl get pods` to get the currently running pods
- Run: `kubectl logs <pod name>`
- Run: `kubectl describe pod <pod name>` to check a stalled pod stuck in containercreating

### API E2E Testing
- Get auth0 token using curl:
- Get Client ID and Client Secret from auth0.com
- Login, go to Applications-> Applications -> "Machine API (Test Application)", or similar
- Copy `Client ID` into Environment variable `SERVAL_CLIENT_ID`
- Copy `Client Secret` into Environment variable `SERVAL_CLIENT_SECRET`
- Copy the auth0 url into Environment variable `SERVAL_AUTH_URL` (i.e. `SERVAL_AUTH_URL=https://sil-appbuilder.auth0.com`)
- Set `SERVAL_HOST_URL` to the api's URL (i.e. `SERVAL_HOST_URL=http://localhost`)
- Run tests from `Serval.E2ETests`
- The token will automatically be retrieved from Auth0 when you run the tests
### Running the API E2E Tests
In order to run the E2E tests, you will need to have the appropriate credentials
- Get Client ID and Client Secret from auth0.com
- Login, go to Applications-> Applications -> "Machine API (Test Application)" or similar
- Copy `Client ID` into Environment variable `SERVAL_CLIENT_ID`
- Copy `Client Secret` into Environment variable `SERVAL_CLIENT_SECRET`
- Copy the auth0 url into Environment variable `SERVAL_AUTH_URL` (e.g. `SERVAL_AUTH_URL=https://sil-appbuilder.auth0.com`)
- Set `SERVAL_HOST_URL` to the api's URL (e.g. `SERVAL_HOST_URL=http://localhost`)
Now, when you run the tests from `Serval.E2ETests`, the token will automatically be retrieved from Auth0.

### Debugging the S3 bucket

To view files stored in the bucket, run
```
aws s3 ls s3://aqua-ml-data/<deployment environment>
```
### Mongo debugging

* First, get the mongo pod name: `kubectl get pods -n serval`
* Then forward to a local port, such as 28015: `kubectl port-forward <pod name> 28015:27017 -n serval`
* Download [MongoDB Compass](https://www.mongodb.com/try/download/compass).
* Then, open MongoDB Compass and connect to `mongodb://localhost:28015/?directConnection=true`


### Access S3 bucket
## Development Notes
### CSharpier

- to view files, run `aws s3 ls s3://aqua-ml-data/<deployment environment>`
All C# code should be formatted using [CSharpier](https://csharpier.com/). The best way to enable support for CSharpier is to install the appropriate [IDE extension](https://csharpier.com/docs/Editors) and configure it to format on save.

### Mongo investigation
### Coding conventions

[Here](https://learn.microsoft.com/en-us/dotnet/csharp/fundamentals/coding-style/identifier-names) is a good overview of naming conventions. [Here](https://learn.microsoft.com/en-us/dotnet/csharp/fundamentals/coding-style/coding-conventions) is a good overview of coding conventions. If you want to get in to even more detail, check out the [Framework design guidelines](https://learn.microsoft.com/en-us/dotnet/standard/design-guidelines/).

* First, get the pod name: `kubectl get pods -n serval`
* Then forward to local port, such as 28015: `kubectl port-forward <pod name> 28015:27017 -n serval`
* Then open MongoDB Compass and connect to `mongodb://localhost:28015/?directConnection=true`

0 comments on commit 91b609e

Please sign in to comment.