Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

✨ Add preliminary python support #348

Merged
merged 11 commits into from
Oct 10, 2023
Merged

✨ Add preliminary python support #348

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
f6e0d36
Updated protocol to more recent version
JonahSussman Sep 8, 2023
6584079
Update types for robustness
JonahSussman Sep 14, 2023
8448e49
Merge branch 'konveyor:main' into update-protocol
JonahSussman Sep 14, 2023
6982513
Resolve merge conflicts
JonahSussman Sep 19, 2023
6681eeb
Merge branch 'main' into update-protocol
JonahSussman Sep 22, 2023
f55d500
Added python support via generic-external-provider
JonahSussman Sep 28, 2023
44af4d8
Merge branch 'main' into python-exploration
JonahSussman Sep 28, 2023
d48a9d5
Fixed Python venv example and demo-testing.yml
JonahSussman Sep 28, 2023
eb032bc
Fixed issue where nonconforming notification responses gummed up the …
JonahSussman Oct 6, 2023
024f778
Fixed requested changes
JonahSussman Oct 6, 2023
1ddb892
Fixed nits and added more robust jsonrpc2 debugger
JonahSussman Oct 9, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 10 additions & 2 deletions .github/workflows/demo-testing.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,15 +14,23 @@ jobs:
- name: build demo image
run: podman build -f demo.Dockerfile -t localhost/testing:latest

- name: install yq for testing
run: go install github.com/mikefarah/yq/v4@latest

- name: run demo image and ensure violations output unchanged
run: |
podman run -v $(pwd)/demo-output.yaml:/analyzer-lsp/output.yaml:Z localhost/testing:latest
git diff --exit-code demo-output.yaml
diff \
<(sort <(sed 's/incidents\.[0-9]\+/incidents\.x/g' <(yq -P 'sort_keys(..)' -o=props <(git show HEAD:demo-output.yaml)))) \
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what sorcery is this?

Copy link
Contributor Author

@JonahSussman JonahSussman Sep 29, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since the incidents in a violation can be returned in any order, git diff kept failing. i.e. git diff would compare

sample-rule:
  incidents:
  - A
  - B
sample-rule:
  incidents:
  - B
  - A

and fail. This is because diff only compares the textual differences between the files. It has no knowledge of the YAML structure. To fix this, I installed yq which allows us to examine the actual structure of the input YAML. My whole goal was to still use diff to compare the two files, but make the position of each incident in the array irrelevant.

Moving from right to left, I first get the different versions of demo-output.yaml. I then pipe that into yq, with the -o=props flag . -P 'sort_keys(..)' is actually unnecessary due to what I do later. If we piped the YAML files I showed into that, it would return:

sample-rule.incidents.0 = A
sample-rule.incidents.1 = B
sample-rule.incidents.0 = B
sample-rule.incidents.1 = A

I then pipe that output into sed, replacing every incidents.<number> with incidents.x to erase which index each element is in. Finally, I sort the result and diff the two streams. Both files would return

sample-rule.incidents.x = A
sample-rule.incidents.x = B

Which is the same thing. No error, ta-da!

There might be a better way than cobbling together some shell commands for this. I wouldn't even be surprised if there was a way you could do all of this in yq! At least one of the advantages of having it installed is that we can investigate the actual structure of the YAML now in our tests.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@JonahSussman lovely, I think this is a much better version of what we are doing in Kantra now, would love to copy it there

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would consider it a great honor, copy away! I would remove the -P 'sort_keys(..)' part as I'm almost positive it's redundant.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we sort these in the engine then?

<(sort <(sed 's/incidents\.[0-9]\+/incidents\.x/g' <(yq -P 'sort_keys(..)' -o=props <(cat demo-output.yaml))))

- name: run demo image and ensure dependency output unchanged
run: |
podman run --entrypoint /usr/bin/konveyor-analyzer-dep -v $(pwd)/demo-dep-output.yaml:/analyzer-lsp/demo-dep-output.yaml:Z localhost/testing:latest --output-file=demo-dep-output.yaml
git diff --exit-code demo-dep-output.yaml
podman run -v $(pwd)/demo-output.yaml:/analyzer-lsp/output.yaml:Z localhost/testing:latest
diff \
<(sort <(sed 's/incidents\.[0-9]\+/incidents\.x/g' <(yq -P 'sort_keys(..)' -o=props <(git show HEAD:demo-dep-output.yaml)))) \
<(sort <(sed 's/incidents\.[0-9]\+/incidents\.x/g' <(yq -P 'sort_keys(..)' -o=props <(cat demo-dep-output.yaml))))

- name: save image
run: |
Expand Down
5 changes: 5 additions & 0 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,8 @@ examples/java/target
examples/java/.settings
examples/java/.classpath
examples/java/.project
examples/python/.venv
examples/python/__pycache__
org.eclipse.core*
org.eclipse.osgi*
org.eclipse.equinox*
Expand All @@ -14,3 +16,6 @@ konveyor-analyzer
konveyor-analyzer-dep
go.work
go.work.sum

external-providers/generic-external-provider/generic-external-provider
external-providers/golang-dependency-provider/golang-dependency-provider
3 changes: 3 additions & 0 deletions Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,9 @@ FROM jaegertracing/all-in-one:latest AS jaeger-builder
# The unofficial base image w/ jdtls and gopls installed
FROM quay.io/konveyor/jdtls-server-base

RUN python3 -m ensurepip --upgrade
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note to selves, we need to eventually find a way to have these broken up and use some shared filesystem while running. Probably something for Kantra to figure out. Do we have an issue for this?

RUN python3 -m pip install python-lsp-server

COPY --from=jaeger-builder /go/bin/all-in-one-linux /usr/bin/

COPY --from=builder /analyzer-lsp/konveyor-analyzer /usr/bin/konveyor-analyzer
Expand Down
10 changes: 5 additions & 5 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,17 +10,17 @@ If you would like to run a quick demo we have a Dockerfile that has all the depe

To run this demo build the containers:

```
$ podman build -f Dockerfile -t quay.io/konveyor/analyzer-lsp
$ podman build -f demo.Dockerfile -t test-analyzer-engine
```sh
podman build -f Dockerfile -t quay.io/konveyor/analyzer-lsp
podman build -f demo.Dockerfile -t test-analyzer-engine
```

This will build the engine, and include the current set of rules and examples in the container to be used.

To run the rules (rule-example.yaml) against the examples, and save the output to the `demo-output.yaml` file:

```
$ podman run -v $(pwd)/demo-output.yaml:/analyzer-lsp/output.yaml:Z test-analyzer-engine
```sh
podman run -v $(pwd)/demo-output.yaml:/analyzer-lsp/output.yaml:Z test-analyzer-engine
```

## Running from source code
Expand Down
45 changes: 42 additions & 3 deletions demo-output.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -137,6 +137,8 @@
- testing
- test
incidents:
- uri: file:///analyzer-lsp/examples/golang/dummy/test_functions.go
message: all go files
- uri: file:///analyzer-lsp/examples/golang/main.go
message: all go files
links:
Expand All @@ -158,8 +160,8 @@
category: potential
incidents:
- uri: file:///analyzer-lsp/examples/golang/main.go
message: golang apiextensions/v1/customresourcedefinitions found file:///analyzer-lsp/examples/golang/main.go:9
lineNumber: 9
message: golang apiextensions/v1/customresourcedefinitions found file:///analyzer-lsp/examples/golang/main.go:10
lineNumber: 10
variables:
file: file:///analyzer-lsp/examples/golang/main.go
golang-gomod-dependencies:
Expand Down Expand Up @@ -217,7 +219,7 @@
name: main
- uri: file:///analyzer-lsp/examples/golang/main.go
message: apiextensions/v1beta1/customresourcedefinitions is deprecated, apiextensions/v1/customresourcedefinitions should be used instead
lineNumber: 9
lineNumber: 10
variables:
file: file:///analyzer-lsp/examples/golang/main.go
lang-ref-003:
Expand Down Expand Up @@ -262,6 +264,43 @@
variables:
tags:
- Golang
python-sample-rule-001:
description: ""
category: potential
incidents:
- uri: file:///analyzer-lsp/examples/python/file_a.py
message: python sample rule 001
lineNumber: 2
variables:
file: file:///analyzer-lsp/examples/python/file_a.py
- uri: file:///analyzer-lsp/examples/python/file_b.py
message: python sample rule 001
lineNumber: 0
variables:
file: file:///analyzer-lsp/examples/python/file_b.py
python-sample-rule-002:
description: ""
category: potential
incidents:
- uri: file:///analyzer-lsp/examples/python/file_a.py
message: python sample rule 002
lineNumber: 5
variables:
file: file:///analyzer-lsp/examples/python/file_a.py
- uri: file:///analyzer-lsp/examples/python/file_b.py
message: python sample rule 002
lineNumber: 7
variables:
file: file:///analyzer-lsp/examples/python/file_b.py
python-sample-rule-003:
description: ""
category: potential
incidents:
- uri: file:///analyzer-lsp/examples/python/main.py
message: python sample rule 003
lineNumber: 27
variables:
file: file:///analyzer-lsp/examples/python/main.py
singleton-sessionbean-00001:
description: ""
category: potential
Expand Down
3 changes: 3 additions & 0 deletions demo.Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,9 @@ WORKDIR /analyzer-lsp
COPY rule-example.yaml /analyzer-lsp/rule-example.yaml
COPY examples /analyzer-lsp/examples

RUN python3 -m venv /analyzer-lsp/examples/python/.venv
RUN yes | python3 -m pip install -r /analyzer-lsp/examples/python/requirements.txt

EXPOSE 16686

ENTRYPOINT ["sh", "-c", "all-in-one-linux &> /dev/null & sleep 5 && konveyor-analyzer --enable-jaeger && curl -o traces.json http://localhost:16686/api/traces?service=analyzer-lsp"]
Loading
Loading