Replace Paver quality and js_test commands #35159

salman2013 · 2024-07-23T08:55:57Z

Paver is deprecated, and we aim to fully remove it soon. You can find more details here:

Paver deprecation issue
Paver removal PR

A few Paver commands remain in the edx-platform, and to proceed with its removal, we've replaced the following Paver commands with equivalent Make commands:

Replaced paver run_pep8 with pycodestyle
Replaced paver run_eslint
Replaced paver run_stylelint
Replaced paver run_xsslint
Replaced paver run_pii_check
Replaced paver check_keywords
Replaced paver test_js* and paver diff_coverage

How to Test This PR:

In the edx-platform directory, you can use the following terminal commands to check for linting issues in Python and JavaScript files. Make sure your virtual environment is activated before running these commands.

make pycodestyle
make eslint
make stylelint
make xsslint
make pi_check
make check_keyword
make test-js
make test-coverage
make quality-test

For more details, please refer to the ticket: Replace paver quality and js_test commands

salman2013 · 2024-07-23T11:30:30Z

@feanil @kdmccormick
Should we remove the paver tests as well like https://github.com/openedx/edx-platform/tree/master/pavelib/paver_tests?

kdmccormick · 2024-07-23T14:10:18Z

@salman2013 as you remove Paver commands, please remove the tests that correspond to those commands.

kdmccormick · 2024-07-24T14:09:15Z

Thanks for the PR @salman2013, but this only goes part of the way. The goal of #35159 is not only to get rid of the paver ... commands. The goal is get rid of all the Python code that runs our quality checks. In the end, we should be able to delete the whole pavelib directory.

Keep in mind that there are several things that pavelib code did that are now completely unnecessary, for example:

junit.xml -- I don't know if we need to write this file any more. If nothing is obviously using it, it can be deleted.
violation counts -- There was a time when all checks had "violation limits", so we would need to keep track of the violation count, and then fail only if the count exceeded the limit. For many checks, the violation limit is now zero. If the violation limit for a check is zero, then that check can be simplified, since any violation is enough to fail it.

I recommend starting with one simple check, such as pycodestyle, and see if you can get that running without any Python scripts involved.

salman2013 · 2024-08-05T11:46:36Z

@kdmccormick Could you please look into my comment #34845 (comment)

kdmccormick

Just "Requesting Changes" so this doesn't show up in my queue. The direction is looking great-- just re-request my review whenever you're ready Salman.

salman2013 · 2024-08-21T05:50:35Z

@kdmccormick This PR is ready for another pass. Please take a look thanks.

kdmccormick

Fantastic work so far. Really happy to see these running without Paver.

scripts/generic-ci-tests.sh

pavelib/prereqs.py

pavelib/utils/envs.py

pavelib/paver_tests/test_assets.py

scripts/quality_test/js_test.py

scripts/quality_test/suites/suite.py

scripts/quality_test/utils/envs.py

stylelint.config.js

Makefile

salman2013 · 2024-08-27T07:18:43Z

@kdmccormick I believe this PR is ready for another pass. Please take a look thanks.

kdmccormick

Most of this feedback is straightforward, except the very last item regarding subprocess.run. I'm happy to go over that in our sync tomorrow morning if my comment isn't clear.

Makefile

pavelib/utils/envs.py

Makefile

scripts/quality_test/quality_test.py

salman2013 · 2024-09-12T10:47:04Z

@kdmccormick Thanks for reviewing the PR, I have fixed all your comments, and the PR is ready for another pass.

kdmccormick · 2024-10-24T16:52:14Z

Taking another look...

kdmccormick

Here's a partial review-- I'll have some more comments tomorrow, and I still need to run each check locally. But generally this is looking good and it's exciting to see how much you were able to delete.

Makefile

kdmccormick · 2024-10-25T18:53:16Z

I've created a series of PRs based on this branch to make sure that each check fails when a violation is added:

pycodestyle
- Validate pycodestyle kdmccormick/edx-platform#21
pii_check
- Validate pii_check kdmccormick/edx-platform#18
check_keywords
- Validate check_keywords kdmccormick/edx-platform#19
test-js
- Validate test-js kdmccormick/edx-platform#20
coverage-js
- Validate coverage-js kdmccormick/edx-platform#22
eslint
- Validate eslint kdmccormick/edx-platform#24
stylelint
- Validate stylelint kdmccormick/edx-platform#23
xsslint
- Validate xsslint kdmccormick/edx-platform#25

kdmccormick · 2024-10-28T14:37:33Z

@salman2013 I'm currently validating each individual check with this spreadsheet: https://docs.google.com/spreadsheets/d/1KHl4bD9z1OodgLsJLUkobL8R0itMikEOSbHmkIFsMok/edit?gid=0#gid=0

kdmccormick

pycodestyle and test-js are failing correctly 😄 Good job on those ✔️
pii_check has problems on this PR, but it also has problems on master-- see my comment below.
xsslint on this PR is failing to catch violations, although it works on master.
check_keywords on this PR is properly finding violations, but it does not print the message to CI output.
I'm still trying out coverage-js, eslint, stylelint-- I'll let you know how testing those goes.

scripts/quality_test.py

kdmccormick

As I've highlighted, existing issues with Paver have made it so that these checks are not working on master or your branch, so please just remove them entirely:

coverage-js 🪓
stylelint 🪓

Furthermore, the following checks do not rely on any complicated output parsing, so instead of being part of a Python script, please pull the underlying shell commands directly into their respective Makefile target:

check_keywords
pii_check

and, for the same reason, pull this into its own npm run command in package.json:

test-js

which will allow you to delete scripts/js_test.py.

That leaves just two commands in scripts/quality_test.py...

run_xsslint: This is a python function that uses a shell command to call a python script. That's bonkers (not your fault, it was that way before). The implementation in your PR, though, currently silently passes when there are xsslint violations. Instead of fixing it, let's simplify further. Please delete this function. Move the logic which compares violations against xsslint_thresholds.json logic into scripts/xsslint/xsslint/main.py. Finally, create a Makefile target which invokes scripts/xsslint/xss_linter.py directly.
run_eslint: This command currently works, both in the Paver implementation, and in your implementation. However, it is still more complicated than it needs to be. Please pull it into a new Python script called scripts/eslint.py. Keep it as simple as possible. Minimal variables, minimal functions, no constants. Do not generate any report files; just capture the output directly and parse the last line. Please also be sure to print() both the shell command and the command's output so that developers can understand the check when they look at the build log.

kdmccormick · 2024-10-29T18:40:46Z

scripts/js_test.py

+#     ("compare-branch=", "b", "Branch to compare against, defaults to origin/master"),
+# ], share_with=['coverage'])
+
+def diff_coverage():


Unfortunately, I have found that the JS coverage check does not actually validate anything, both on master and on this branch.

Here is a PR that removes a significant number of JS tests. You would expect that this would cause JS coverage to fail. However, both with the Paver checks and with your new Paver-free checks, it seems that coverage check is not actually doing anything. The operative log lines seems to be:

------------- Diff Coverage Diff: origin/master..HEAD, staged and unstaged changes ------------- No lines with coverage information in this diff. -------------

In my opinion, it is not worthwhile trying to turn JS coverage check back on. All the JS in edx-platform is legacy code. It is either deprecated or soon-to-be-deprecated. Please just remove all the JS coverage checks from this PR.

scripts/js_test.py

kdmccormick · 2024-10-29T18:57:38Z

scripts/quality_test.py

+    command = [
+        'node', 'node_modules/stylelint',
+        '*scss_files',
+        '--custom-formatter', 'stylelint-formatter-pretty/index.js'
+    ]


This command was silently passing because:

node_modules/stylelint is a directory, not a binary. The binary is at node_modules/.bin/stylelint. Running node on a directory does nothing.

*scss_files matches nothing. The correct pattern is **/*.scss

Regardless, though, the Paver version of this command doesn't work on master either. It has been silently failing with /bin/sh: 1: stylelint: not found for potentially several years now. There are now 24k+ sass style violations on master. Fixing all those violations would not be a good use of our time, especially since we intend to delete all the Sass code in edx-platform.

So, you can just remove all the stylelint checks.

kdmccormick · 2024-10-29T18:58:48Z

scripts/quality_test.py

-        ),
-        ignore_error=True
-    )
+    violations_limit = 4950


This threshold is way too high! I know it comes from the master branch. I don't know why it was set so high in the first place.

There are only 1213 eslint violations on master currently. Can you lower the threshold down to that?

pavelib/quality_test.py

scripts/generic-ci-tests.sh

Makefile

stylelint.config.js

scripts/quality_test/quality_test.py

scripts/js_test.py

scripts/eslint.py

kdmccormick · 2024-11-14T14:48:16Z

scripts/eslint.py

+        num_violations = int(re.search(regex, last_line).group(0)) if last_line else 0
+    # An AttributeError will occur if the regex finds no matches.
+    except (AttributeError, ValueError):
+        print(f"FAILURE: Number of eslint violations could not be found in '{last_line}'")


If num_violations cannot be parsed, then that should cause the quality to check to immediately fail with this error message.

Otherwise, we would fail with a a NameError on line 53, which would be confusing to developers.

@salman2013 , simply printing an error message is this case is not sufficient. We need to be sure that the process fails (exits non-zero status code) so that the PR check fails. You can do so by adding raise to this block.

scripts/eslint.py

.pii_annotations.yml

Makefile

pavelib/paver_tests/test_stylelint.py

Makefile

kdmccormick · 2024-12-03T16:37:12Z

While testing this PR, I found an existing logging issue with our JS CI, and I made a fix that is ready for review: #35954

Once that merges, could you rebase on top of it (or merge it in) so that we have better JS test logging to work with?

salman2013 · 2024-12-04T11:51:11Z

@kdmccormick I have rebased this branch with #35954

salman2013 · 2024-12-04T18:45:13Z

@kdmccormick I have fixed the comments. Could you take another look thanks.

kdmccormick

Also see https://github.com/openedx/edx-platform/pull/35159/files#r1870245413

kdmccormick · 2024-12-04T20:39:45Z

scripts/eslint.py

+def fail_quality(name, message):
+    """
+    Fail the specified quality check.
+    """
+    print(name)
+    print(message)
+    sys.exit()


As written, this displays the word "FAILURE" but it not actually cause the quality check to fail. That is problem... you can check the CI logs on this PR, and see how we are violating the threshold but still getting a green build:

(separate comment: looks like the violations limit needs to be 1303 rather than 1213)

Are you familiar with the idea of exit codes? It is a universal convention where each process in a system returns an integer upon completion. Zero indicates success, nonzero indicates failure. That is how GitHub Actions and many other systems determine whether a process succeeded or not. When you raise a Python exception, that automatically causes the process to return a nonzero exit code.

Here, sys.exit() returns exit code 0, indicating success. Obviously, that is not what we want :) You want to raise exit code 1, indicating "general error" to the surrounding process, which will cause the PR check to fail as desired.

kdmccormick · 2024-12-05T13:18:24Z

scripts/eslint.py

+        "--ext", ".js",
+        "--ext", ".jsx",
+        "--format=compact",
+        "."


@salman2013

Suggested change

"."

"lms",

"cms",

"common",

"openedx",

"xmodule",

I haven't tested it yet, but perhaps specifying each root path would lower the violation count. Specifying "." (current directory) might be including node_modules accidentally.

@kdmccormick I believe your thoughts are right, making these changes the error count reduced to 1285 from 1303.

kdmccormick · 2024-12-06T15:19:04Z

scripts/eslint.py

+    result = subprocess.run(
+        command,
+        text=True,
+        check=False,


Suggested change

check=False,

check=True,

Using check=True ensures that the script will fail if subprocess.run fails. Using check=False hides the errors-- in other words, check=False means "fail silently".

Failing silently is usually bad. In CI scripting, failing silently is very bad, because it means that PRs can pass tests even when they shouldn't.

@salman2013 , every time you call subprocess.run, including here, please please use check=True unless you are actively handling the error case yourself.

@kdmccormick
You are right, actually, i intentionally used check=False, Reason is that check-True command is failing with status code 1. I tried to figure out the reason, there could be the following three possibility

Command Syntax

Eslint Configuration issue

Files path issue

The above three issues were not the reasons i investigated. The result.stderr is showing nothing but the result.stdout is printing almost 1285 errors from different files. Which i figured out the command is sending status code 1 as a lot of errors in different files.

As we raise exceptions in case of violation count comparison, I thought it safe to use check=False.

Ahh I see @salman2013 . You are right, it makes sense to check=False here. Just please then inspect the actual integer value of the exit code. According to the eslint docs, exit code 1 indicates that linting happened successfully but there were violations, whereas exit code 2 indicates that an unexpected error occurred (in which case we'd want to fail CI).

scripts/eslint.py

scripts/xsslint/xsslint/main.py

kdmccormick

I used these PRs to confirm that each kind of build failure is caught by the check as we'd expect. Great work!

Just one comment about raising the pii_check threshold, and there is my comment from earlier about xsslint exit codes 1 vs 2. With those comments resolved, feel free to merge 🚀

Nitpick: Can you label this commit as build: rather than chore:? "chore" is meant to indicate minor, repetitive tasks like routine requirement upgrades. This PR is an overhaul of the quality and JS build suite, much more ambitious than a chore 😄

kdmccormick · 2024-12-10T15:40:44Z

.pii_annotations.yml

@@ -1,7 +1,7 @@
 source_path: ./
 report_path: pii_report
 safelist_path: .annotation_safe_list.yml
-coverage_target: 94.5


PII annotation coverage has improved even further since my last review! Can you bump this up to 85.3?

kdmccormick · 2024-12-10T16:18:51Z

scripts/eslint.py

@@ -0,0 +1,73 @@
+"""  # pylint: disable=django-not-configured
+Check code quality using pycodestyle, pylint, and diff_quality.


Please update this comment before merging.

openedx-webhooks added the open-source-contribution PR author is not from Axim or 2U label Jul 23, 2024

This comment was marked as outdated.

Sign in to view

salman2013 requested a review from kdmccormick July 23, 2024 11:20

salman2013 marked this pull request as ready for review July 23, 2024 11:21

salman2013 requested a review from feanil July 23, 2024 11:31

salman2013 closed this Jul 31, 2024

salman2013 reopened this Jul 31, 2024

kdmccormick requested changes Aug 9, 2024

View reviewed changes

salman2013 closed this Aug 14, 2024

salman2013 reopened this Aug 14, 2024

salman2013 closed this Aug 15, 2024

salman2013 reopened this Aug 15, 2024

kdmccormick marked this pull request as draft August 20, 2024 18:57

salman2013 requested a review from kdmccormick August 21, 2024 05:49

salman2013 marked this pull request as ready for review August 21, 2024 05:50

kdmccormick requested changes Aug 21, 2024

View reviewed changes

salman2013 changed the title ~~Replace paver quality commands~~ Replace Paver quality and js_test commands Aug 27, 2024

salman2013 requested a review from kdmccormick August 27, 2024 07:18

salman2013 mentioned this pull request Aug 27, 2024

Replace Paver quality and js_test commands #34845

Open

kdmccormick requested changes Sep 4, 2024

View reviewed changes

salman2013 requested a review from kdmccormick September 12, 2024 10:47

kdmccormick reviewed Oct 24, 2024

View reviewed changes

Makefile Outdated Show resolved Hide resolved

Makefile Outdated Show resolved Hide resolved

kdmccormick mentioned this pull request Oct 28, 2024

Side-by-side testing of Paver checks and New checks kdmccormick/edx-platform#26

Closed

kdmccormick requested changes Oct 28, 2024

View reviewed changes

scripts/quality_test.py Outdated Show resolved Hide resolved

kdmccormick requested changes Oct 29, 2024

View reviewed changes

salman2013 commented Nov 4, 2024

View reviewed changes

kdmccormick mentioned this pull request Nov 5, 2024

Re-enable pii_check linting #35775

Open

kdmccormick mentioned this pull request Nov 12, 2024

Paver Removal 2/3 #34830

Open

salman2013 requested a review from kdmccormick November 13, 2024 17:38

kdmccormick requested changes Nov 21, 2024

View reviewed changes

salman2013 requested a review from a team as a code owner December 2, 2024 16:45

kdmccormick requested changes Dec 2, 2024

View reviewed changes

Makefile Outdated Show resolved Hide resolved

kdmccormick mentioned this pull request Dec 3, 2024

fix: downgrade karma-spec-reporter to fix JS test logging #35954

Merged

kdmccormick mentioned this pull request Dec 3, 2024

Temporarily disable broken Webpack JS test suites #35955

Merged

kdmccormick mentioned this pull request Dec 4, 2024

Fix and re-enable tests for legacy Webpack-built JS #35956

Open

salman2013 requested a review from kdmccormick December 4, 2024 18:44

kdmccormick mentioned this pull request Dec 4, 2024

Script to print xsslint violations is missing #35965

Open

kdmccormick requested changes Dec 4, 2024

View reviewed changes

kdmccormick reviewed Dec 5, 2024

View reviewed changes

salman2013 requested a review from kdmccormick December 6, 2024 10:45

kdmccormick requested changes Dec 6, 2024

View reviewed changes

salman2013 force-pushed the salman/remove-paver-commands branch from a61d9a7 to ca2712a Compare December 10, 2024 14:39

salman2013 requested a review from kdmccormick December 10, 2024 14:40

kdmccormick approved these changes Dec 10, 2024

View reviewed changes

kdmccormick reviewed Dec 10, 2024

View reviewed changes

build: replace paver quality tests and js commands

f85d452

salman2013 force-pushed the salman/remove-paver-commands branch from 4788953 to f85d452 Compare December 11, 2024 06:03

-        "."
+        "lms",
+        "cms",
+        "common",
+        "openedx",
+        "xmodule",

		@@ -0,0 +1,73 @@
		""" # pylint: disable=django-not-configured
		Check code quality using pycodestyle, pylint, and diff_quality.

Replace Paver quality and js_test commands #35159

Are you sure you want to change the base?

Replace Paver quality and js_test commands #35159

Conversation

salman2013 commented Jul 23, 2024 • edited Loading

This comment was marked as outdated.

salman2013 commented Jul 23, 2024 • edited Loading

kdmccormick commented Jul 23, 2024

kdmccormick commented Jul 24, 2024

salman2013 commented Aug 5, 2024

kdmccormick left a comment

Choose a reason for hiding this comment

salman2013 commented Aug 21, 2024

kdmccormick left a comment

Choose a reason for hiding this comment

salman2013 commented Aug 27, 2024

kdmccormick left a comment

Choose a reason for hiding this comment

salman2013 commented Sep 12, 2024

kdmccormick commented Oct 24, 2024

kdmccormick left a comment

Choose a reason for hiding this comment

kdmccormick commented Oct 25, 2024 • edited Loading

kdmccormick commented Oct 28, 2024

kdmccormick left a comment

Choose a reason for hiding this comment

kdmccormick left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kdmccormick commented Dec 3, 2024 • edited Loading

salman2013 commented Dec 4, 2024

salman2013 commented Dec 4, 2024

kdmccormick left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kdmccormick Dec 10, 2024 • edited Loading

Choose a reason for hiding this comment

kdmccormick left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kdmccormick Dec 10, 2024 • edited Loading

Choose a reason for hiding this comment

salman2013 commented Jul 23, 2024 •

edited

Loading

salman2013 commented Jul 23, 2024 •

edited

Loading

kdmccormick commented Oct 25, 2024 •

edited

Loading

kdmccormick left a comment •

edited

Loading

kdmccormick commented Dec 3, 2024 •

edited

Loading

kdmccormick Dec 10, 2024 •

edited

Loading

kdmccormick left a comment •

edited

Loading

kdmccormick Dec 10, 2024 •

edited

Loading