Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enhance find_json garbage filtering #688

Open
wants to merge 2 commits into
base: openSUSE/release/3006.0
Choose a base branch
from

Conversation

m-czernek
Copy link
Contributor

@m-czernek m-czernek commented Oct 31, 2024

What does this PR do?

In this PR, we enhance the find_json function to work with valid JSONs that contain garbage right after the JSON end. This is a problem when the text contains JSON and, for example, deprecation messages, such as:

{
    "local": {
        "saltutil_|-sync_states_|-sync_states_|-sync_states": {
            "name": "sync_states",
            "changes": {},
            "result": true,
            "comment": "No updates to sync",
            "__sls__": "util.syncstates",
            "__run_num__": 0,
            "start_time": "14:35:37.150819",
            "duration": 1256.999,
            "__id__": "sync_states"
        },
...JSON continues...
}/usr/lib/python3.6/site-packages/salt/states/x509.py:214: DeprecationWarning: The x509 modules are deprecated. Please migrate to the replacement modules (x509_v2). They are the default from Salt 3008 (Argon) onwards.
  "The x509 modules are deprecated. Please migrate to the replacement "

What issues does this PR fix or reference?

Related to: #25514
Upstream PR: saltstack/salt#67023

Merge requirements satisfied?

[NOTICE] Bug fixes or features added to Salt require tests.

Commits signed with GPG?

Yes/No

Please review Salt's Contributing Guide for best practices.

See GitHub's page on GPG signing for more information about signing commits with GPG.

@m-czernek m-czernek force-pushed the find-json-enhancement branch from dd79fb7 to 992906a Compare October 31, 2024 08:59
@vzhestkov vzhestkov force-pushed the openSUSE/release/3006.0 branch from 71a8c26 to eb6c67e Compare November 26, 2024 09:23
@@ -39,6 +39,7 @@ def find_json(raw):
# Search for possible starts end ends of the json fragments
for ind, _ in enumerate(lines):
line = lines[ind].lstrip()
line = line[0] if line else line
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm bit confused with this part, maybe there are no such examples, but for me it looks like it can work with only pretty formated json. It's confusing because looks like it just takes only first symbol from the string with ignoring the rest part completely.

Copy link
Contributor Author

@m-czernek m-czernek Nov 27, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It does work with pretty formatted JSONs only indeed. I don't think json.find_json supports unformatted JSONS - except for the best-effort search at the end, which just filters the text before it and then tries to load whatever is left.

Our implementation that you authored definitely doesn't support unformatted JSONs:

    for ind, _ in enumerate(lines):
        line = lines[ind].lstrip()
        line = line[0] if line else line
        if line == "{" or line == "[": # <-- this will never work for unformatted JSON
            starts.append((ind, line))
        if line == "}" or line == "]": # <-- this will never work for unformatted JSON
            ends.append((ind, line))

In other words, if I'm not mistaken, we didn't support unformatted JSONs before, and we still don't, right?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants