Code Coverage Reports in Deploy Commands Produce Unreliable Covered Lines

[mcarvin8]

Summary

This has already been reported as bugs such as forcedotcom/cli#1568 but this has been closed due to "working as expected".

Please look into this soon for a new CLI Build.

When running sf project deploy start/sf project deploy validate commands with the code coverage flags (like --coverage-formatters json), the uncovered lines reported back are correct. The covered lines are incorrect for various reasons:

• covered lines will sometimes be out-of-range lines (if there is a 98 line Apex Class, the coverage reports will report line 100 as covered)
• covered lines are inconsistent and report back more lines than there are in the file

Please update the Salesforce CLI to reliably report covered lines in the code coverage files so they can be exported to tools like SonarQube for code coverage tracking. In the current state, the uncovered lines can be exported to SonarQube correctly, but you need to have covered lines reported to SonarQube to calculate the coverage percentage. At this point, coverage in SonarQube will always be at 0% or the coverage % will be unreliable due to the covered lines being incorrectly reported.

I'm unclear if the CLI's apex commands produce the same outputs as the deploy commands in terms of code coverage. But we need reliable deploy coverage output so we can report back accurate code coverage during deployment and validation.

@xL3o - Please add anything I missed here based on our testing in my plugin.

See additional notes on my plugin here - mcarvin8/apex-code-coverage-transformer#2

Steps To Reproduce

1. Deploy a package with apex classes or triggers with the code coverage formatted as a JSON or JSON summary. sf project deploy start -x manifest/package.xml --coverage-formatters json
2. Confirm the covered and uncovered lines in the coverage JSON file created by the CLI

Expected result

Uncovered and covered lines are reported back correctly. The covered lines should not have out-of-range line numbers.

Actual result

Uncovered lines are correct. Covered Lines are incorrectly reported back.

System Information

Tested on Windows 11/Powershell and Ubuntu/Bash (CI/CD).

{
  "architecture": "win32-x64",
  "cliVersion": "@salesforce/cli/2.30.8",
  "nodeVersion": "node-v20.11.1",
  "osVersion": "Windows_NT 10.0.22621",
  "rootPath": "C:\\Users\\matthew.carvin\\AppData\\Local\\sf\\client\\2.30.8-d9ea14b",
  "shell": "cmd.exe",
  "pluginVersions": [
    "@oclif/plugin-autocomplete 3.0.9 (core)",
    "@oclif/plugin-commands 3.1.4 (core)",
    "@oclif/plugin-help 6.0.13 (core)",
    "@oclif/plugin-not-found 3.0.11 (core)",
    "@oclif/plugin-plugins 4.2.5 (core)",
    "@oclif/plugin-search 1.0.16 (core)",
    "@oclif/plugin-update 4.1.13 (core)",
    "@oclif/plugin-version 2.0.12 (core)",
    "@oclif/plugin-warn-if-update-available 3.0.12 (core)",
    "@oclif/plugin-which 3.1.0 (core)",
    "@salesforce/cli 2.30.8 (core)",
    "apex 3.0.25 (core)",
    "auth 3.3.13 (core)",
    "data 3.1.1 (core)",
    "deploy-retrieve 3.2.16 (core)",
    "dev 2.1.13 (user)",
    "info 3.0.27 (core)",
    "limits 3.1.10 (core)",
    "marketplace 1.0.25 (core)",
    "org 3.3.14 (core)",
    "packaging 2.1.10 (core)",
    "schema 3.1.3 (core)",
    "settings 2.0.27 (core)",
    "sobject 1.1.13 (core)",
    "source 3.1.15 (core)",
    "telemetry 3.1.13 (core)",
    "templates 56.0.16 (core)",
    "trust 3.3.10 (core)",
    "user 3.2.11 (core)",
    "sfdx-decomposer 3.1.0 (user)",
    "sfdx-git-delta 5.34.0 (user)"
  ]
}

Additional information

[github-actions]

Thank you for filing this issue. We appreciate your feedback and will review the issue as soon as possible. Remember, however, that GitHub isn't a mechanism for receiving support under any agreement or SLA. If you require immediate assistance, contact Salesforce Customer Support.

[shetzel]

There is nothing new to report on the code coverage front from that last issue you mentioned. The CLI team is dependent on some other internal teams and their priorities for any work to begin on this. If the code coverage data in the API response is not accurate then there's nothing we can do except report what's given to us. So from a CLI perspective, the code is making the correct calls and reporting the response as provided (i.e., working as designed), and to change that we need other teams to prioritize that work. It's on their roadmaps but I don't know when it will be delivered.

[mcarvin8]

Should we keep this issue open @shetzel or just close it since this seems out of the CLI team's responsibility? Seems like once the API team updates the API response to provide accurate code coverage, it will work fine on any version of the CLI.

[shetzel]

In my opinion we should close it. Unless there is a breaking change involved with responses there is nothing the CLI would have to do to provide better, more accurate code coverage when those fixes are made to the server side APIs and libraries. You and any others are welcome to post here asking for updates on this. I'm hopeful that the work will at least be prioritized this year and we can relay that information when we have it to keep everyone up to date.

...it will work fine on any version of the CLI.

If the fixes are only serverside, then yes. If there is more involved (e.g., updates to the apex-node library) then the CLI could need to bump version dependencies.

[jfaderanga]

it is a blocker for pipeline with Quality Gates. I've been struggling with this issue for few months now. One thing I noticed is the code should be modularized and clean for the issue not to happened. There are times that the error still shows up even the code is modularized and well written which becomes a blocker on development as the code can't be enhanced anymore.

Please re-open the issue. @mcarvin8 @shetzel

[mcarvin8]

@jfaderanga - Per @shetzel , this issue needs to be fixed by the team that manages the API first. I agree with you that this is a blocker for anyone who is trying to implement Quality Gates that enforce code coverage.

@shetzel - Is there a different way to contact the right team that owns updates to the API (outside of a general Salesforce support case)?

[shetzel]

I agree with both of you that it's an issue, just not an issue that anyone on the CLI team can do anything about. This repo is about issues with the CLI. If it's open, it implies we can take some action on it but in this case we can't.

Is there a different way to contact the right team that owns updates to the API (outside of a general Salesforce support case)?

The best ways to influence the right team are opening a support case, and voting up the idea on the IdeaExchange.

[mcarvin8]

it is a blocker for pipeline with Quality Gates. I've been struggling with this issue for few months now. One thing I noticed is the code should be modularized and clean for the issue not to happened. There are times that the error still shows up even the code is modularized and well written which becomes a blocker on development as the code can't be enhanced anymore.

Please re-open the issue. @mcarvin8 @shetzel

mcarvin8/apex-code-coverage-transformer#3

@jfaderanga - I was working on this with my plug-in that converts the coverage JSON to an XML SonarQube will accept.

This will convert covered lines and work around this known issue by reassigning out of bound lines with unused lines. The coverage % will vary but it should still pass if you're hitting coverage. I could release this but I'm skeptical since the covered lines won't always report back the same way (comments and new lines can change it). But then again, however the API calculates deploy coverage is incorrect so this plugin would just convert that JSON entry to a XML entry.

Like I've seen it report back 86% coverage and then 88% coverage on other runs for example. So it should pass in most cases as long as your not near the 75% minimum requirement but this updates isn't ideal until the API is fixed to report back the right covered lines.

If you want, you can try my beta plugin with these updates - sf plugins install apex-code-coverage-transformer@beta

That creates a coverage.xml you can pass to SQ.

It's not ideal until the API is fixed but this will get coverage % in SonarQube relative to what the deploy command currently splits out.

[jfaderanga]

@mcarvin8 I'm using the direct XML (cobertura) output from the deploy command, it has coverage rate from the class node. I upload this to Azure DevOps and Sonar Cloud, from what I noticed, the SC only reads the coverage rate from the class node and it just uses the lines for the UI line highlighting. I haven't seen the JSON output but I expect it should be the same, does your plugin also update the coverage rate, it shouldn't be ay, just the lines?

I've been doing heavy manipulation already to fix the XML to correct the file path (no-map/classname issue), adding more may affect the pipeline performance, but I'll give it a go as I don't have any workaround at the moment, thanks for the suggestion.

[mcarvin8]

@jfaderanga - My plugin essentially just converts the JSON output from the deploy command into a XML file containing the uncovered lines/covered lines.

The latest build of my plugin just prints uncovered lines in the XML since that's accurate.
The beta build of my plugin prints covered and uncovered lines with some extra tinkering done on covered lines due to this bug.

All builds will take care of replacing no-map in the file name based on the --dx-directory flag (the default path being force-app/main/default). So my plugin would save you some steps that you're doing to fix the file-paths.

We noticed with SonarQube that just providing uncovered lines will not update the Coverage % (with the latest plugin build).

After doing the updates in the beta build with adding covered lines (automatically adjusting line numbers with this bug if the line number provided is out-of-range), we did confirm SonarQube calculates the coverage % automatically. It's just not as ideal as having the API return the correct # of covered lines back to the CLI so that my plugin can just convert the JSON output exactly as provided.

[xL3o]

The best ways to influence the right team are opening a support case, and voting up the idea on the IdeaExchange.

@shetzel is IdeaExchange really the best way forward in our situation? I mean this code coverage functionality for validation deployments is already part of the end product, rather than being a new "Idea" that the POs would need to prioritize. Therefore, there should be a more streamlined approach how to highlight this issue/bug directly to the correct team for them to start bug fixing.

[shetzel]

@xL3o - I meant that more as a general statement of how to get issues in front of non-CLI teams. For this issue, a support case is best. However, I'll open this back up and transfer it to the apex-node library. That team is a better choice to own this issue.

[jfaderanga]

@jfaderanga - My plugin essentially just converts the JSON output from the deploy command into a XML file containing the uncovered lines/covered lines.

The latest build of my plugin just prints uncovered lines in the XML since that's accurate. The beta build of my plugin prints covered and uncovered lines with some extra tinkering done on covered lines due to this bug.

All builds will take care of replacing no-map in the file name based on the --dx-directory flag (the default path being force-app/main/default). So my plugin would save you some steps that you're doing to fix the file-paths.

We noticed with SonarQube that just providing uncovered lines will not update the Coverage % (with the latest plugin build).

After doing the updates in the beta build with adding covered lines (automatically adjusting line numbers with this bug if the line number provided is out-of-range), we did confirm SonarQube calculates the coverage % automatically. It's just not as ideal as having the API return the correct # of covered lines back to the CLI so that my plugin can just convert the JSON output exactly as provided.

The no-map replacement feature, does it support multiple packageDirectories or just defaults it to force-app?

[mcarvin8]

Currently it just supports one directory via the --dx-directory flag. That flag defaults to force-app/main/default if you don't provide the flag a different path.

I can look into supporting multiple package directories if you want (probably something like scanning for the sfdx-project.json file and reading all directories listed).

[mcarvin8]

@jfaderanga - I just released a new version of the plugin which supports multiple package directories as listed in your project's sfdx-project.json file. If you run this plugin in the root directory of your project which has the JSON file, you don't need to supply the flag for the sfdx-project.json file path.

https://github.com/mcarvin8/apex-code-coverage-transformer/releases/tag/v1.5.0

This version (which is the latest/stable tag) only will add uncovered lines in the final XML file since that's what's always accurate.

I will need to do some additional work to back-fit these changes into a pre-release/beta build that adds covered lines.

Of course, if this issue is escalated and fixed soon, I don't need to do extra manipulation to correct covered lines as printed by the deploy command.

[jfaderanga]

@mcarvin8 that's awesome mate, thanks for the quick turn around time for this, i'll try this today..

[mcarvin8]

@jfaderanga - Awesome!

I just released a new beta pre-release build which will add covered lines to the final XML file.

Big catch like I've stated previously is that the Salesforce CLI will produce out-of-range covered lines (line 100 in a 98 line file might show as covered). To correct this, I added code that counts the total number of lines in the file and re-assigns out-of-range lines to an unused line number. It's not ideal (since @xL3o has demonstrated that the Salesforce CLI will produce extra lines in the coverage report as well - for example: Salesforce CLI will report 118 lines total covered & uncovered in a file that is only 100 lines). But this will ensure SonarQube always calculates Code Coverage percentage.

If you want to try the beta build with covered lines, run sf plugins install [email protected] and give that a shot.

Otherwise, the latest build (1.5.0) is the safe route since that just prints uncovered lines which is always correct. SonarQube just won't calculate code coverage % if you only provide uncovered lines, which @xL3o and I have confirmed with previous builds.

So just to sum up:

1. Install sf plugins install [email protected] to produce a coverage XML with uncovered lines for multiple package directories.
   1. You will see uncovered lines in SonarQube when viewing the file, but you will not get code coverage percentage
2. Install sf plugins install [email protected] to produce a coverage XML with covered and uncovered lines for multiple package directories, keeping in mind that Salesforce produces unreliable covered lines.
   1. You will get code coverage % in SonarQube, it just won't be 100% accurate due to how the API produces covered lines

[jfaderanga]

@mcarvin8 I have tried the beta version 1.6.0-beta.1 and it works as described.

From what I noticed with the API output, the coverage percentage is correct. If you divide the number of uncovered lines by the number of covered lines, it gives you the coverage percentage which is the same with what is showing in Salesforce.

The plugin beta version 1.6.0-beta.1 converts the JSON (api output) to XML with the exact number of covered and uncovered lines as what's in the JSON (assigning out-of-range lines to unused lines starting from line 1). This is great as this still gives the same coverage percentage assuming the number out of range lines won't go beyond the number of unused lines, which I think impossible to happen.

Also the plugin correctly maps the file path, kudos for this mate!

I would say your plugin mate is an acceptable long-term workaround for this issue as I bet it will take a long time for the issue to be generally fixed. I'm happy to implement this in our CI, thanks a lot @mcarvin8 and @xL3o and @shetzel ! Let me know when you release the stable version of the 1.6.0 beta.

[mcarvin8]

@jfaderanga - Thank you for testing this out and letting me know!

I have released the 1.6.0 tag which adds support for covered lines by re-numbering out-of-range lines.

Once this issue is resolved by Salesforce, I'll remove that re-number function.

sf plugins install [email protected]

[jfaderanga]

@mcarvin8 I started looking into implementing this and made few more tests. The plugin seems to be having an issue trying to find a matching file if the file name contains underscores.

The sample below is from the JSON API output.

    "no-map/ABC_API_CLASSNAME": {
        "fnMap": {},
        "branchMap": {},
        "path": "no-map/ABC_API_CLASSNAME",
        "f": {},
        "b": {},
        "s": {
        ....

The plugin returns the following error even though the file is indeed within one of the package directory.
Error (1): The file name ABC_API_CLASSNAME was not found in any package directory.

[mcarvin8]

@jfaderanga Can you make an issue here on my repo, please? - https://github.com/mcarvin8/apex-code-coverage-transformer/issues

I just tested it out by renaming the Test Apex Class I use in my Unit Test to ABC_API_CLASSNAME.cls and I wasn't able to replicate this error in Windows. I also wasn't able to replicate this failure in my pipeline, which is Ubuntu. mcarvin8/apex-code-coverage-transformer@01012a7

What's the full path to the file in your package directory? In the code, I'm assuming the classes, triggers, and flows are in these sub-folder names (classes, flows, triggers). This works even if you use the default structure force-app/main/default or just 1 folder packaged. If this isn't the right assumption to make, I'll need to update the code differently.

I've only worked with repos following the default force-app/main/default structure so I'm unclear on how you can structure different package directories.

  let relativeClassPath = await findSubFolder(dxDirectory, 'classes');
  let relativeTriggerPath = await findSubFolder(dxDirectory, 'triggers');
  let relativeFlowPath = await findSubFolder(dxDirectory, 'flows');

EDIT: I have released a new version - sf plugins install [email protected] - which will recursively search each package directory to find a file match, without hard-coding any sub-folder names like classes, flows, or triggers.
I tested this in my team's repo (which is a pretty huge code base) and this still ran pretty quickly with the below JSON:

{
 "no-map/ACCOUNT_PROFILE": {
   "fnMap": {},
   "branchMap": {},
   "path": "no-map/ACCOUNT_PROFILE",

Please try 1.6.1 with the class you mentioned above and see if that resolves your issue.

[jfaderanga]

@mcarvin8 yeah looks like the 1.6.1 fixes it. I forgot to mentioned, the other directory in my project, the directory name has space in it. It's a messed up structure as it came from a unlock package.

Folder Name
  ﹂main
    ﹂ default
      ﹂classes
  ﹂Folder Name
     ﹂main
       ﹂ default
         ﹂ labels

The 1.6.1 seems fixed it, thanks heaps mate!

[mcarvin8]

@peternhale - Could you provide any updates to the task of fixing the API to correctly return Code Coverage covered lines during deploy commands? The covered lines returned by the API for deployment commands are inaccurate in their current state, which makes it unreliable to track coverage percentages in tools like SonarQube.

I recall your name being brought up in a code coverage discussion with Allan Oricil, so feel free to include anyone else here if you are the not right person to contact for this.

Please ignore the back-and-forth with JaysonF and I in previous comments on this thread regarding my Salesforce plugin which is designed to convert the code coverage JSON output to an XML for SQ.

[peternhale]

Hi @mcarvin8 Sure, the root issue of the inaccuracy of the code coverage for tests run during a deploy is in the data returned from the deploy. To get to the code coverage we drill down on the returned data starting with DeployResult -> DeployDetails[] -> RunTestResult -> CodeCoverageResult[]

You will find field locationsNotCovered on CodeCoverageResult, but no field for locationsCovered, which is where we find ourselves. The implementation tried to "fill in the blanks", but, as you know, does not do a good job inferring which are lines of code and which are not.

The CodeCoverageResult object needs to be changed to include covered locations. The object is owned by another team here at Salesforce. We have requested requested change, but the work has yet to be prioritized.

I have tried to find an alternative source to fetch the coverage for the test run, but that did not yield anything to use.

[mcarvin8]

Thanks @peternhale. Is there any POC you could provide on the team who owns the CodeCoverageResult object that I could tag to escalate this issue? Or should I raise a Salesforce support case to see if this can get prioritized for release this year?

[peternhale]

@mcarvin8 the only recourse would be to open a support case.

[marcoalmodova]

I just discovered this problem yesterday and found this thread. @mcarvin8 do you already have a case?
I think this should be recognised by Salesforce as a known issue and prioritised.

I can help with a scenario for them to analyse if needed.

[mcarvin8]

@marcoalmodova - Here's the case i just submitted. I can add you as a collaborator if you provide me an email to use. A scenario to have them analyze would be great.

https://help.salesforce.com/s/case-view?caseId=500Hx00000VnAu1IAF

[jfaderanga]

@mcarvin8 hey mate, can you add me too as collaborator? thanks heaps mate..

[marcoalmodova]

@mcarvin8 thanks! My email: [email protected]

I can create an org with a basic apex class + test method, but I think I will need to give them a user so they can authentic locally to run the command, right?

[git2gus]

This issue has been linked to a new work item: W-16286380

[xL3o]

Is there a way to track the linked work item (W-16286380) status somewhere (publicly) or is it private. If so, could someone please share a link?

[bdJohnson72]

I am trying to implement a pipleine with coverage data as well. I see that when I run with cobertura no results are returned from the CLI.

<?xml version="1.0" ?>
<!DOCTYPE coverage SYSTEM "http://cobertura.sourceforge.net/xml/coverage-04.dtd">
<coverage lines-valid="0" lines-covered="0" line-rate="NaN" branches-valid="0" branches-covered="0" branch-rate="NaN" timestamp="1722359815806" complexity="0" version="0.1">
  <sources>
    <source>.</source>
  </sources>
  <packages>
  </packages>
</coverage>

[littyphilip90]

@mcarvin8 can you help me figure what this error means? What am I missing here?

Error (2): Parsing --sfdx-configuration
        No file found at deploy

I'm using [email protected] version and the commands I'm running are as follows

sf project deploy validate -o test -w 20 --test-level RunLocalTests --source-dir abc-application --verbose --coverage-formatters json --results-dir coverage
sf apex-code-coverage transformer transform -j "coverage/coverage/coverage.json" -x "coverage/coverage/coverage.xml" -c "deploy"

Is this because of source directory not being force-app. It's with a different name. I tried --dx-directory abc-application as well, but got same error.

[littyphilip90]

I figured it out, the right command to be run is
sf apex-code-coverage transformer transform -j "coverage/coverage/coverage.json" -x "coverage/coverage/coverage.xml" -c sfdx-project.json

But it still gives me error,
Error (1): The file name no-map\AssignedResourceUpdateBatchClass was not found in any package directory.

Any help further is appreciated, thank you!

[xL3o]

@littyphilip90 please use the latest plugin version.
Also I would like to keep that issue clean and targeted at the bug of Salesforce API responding false coverage data. For any additional questions please post here: https://github.com/mcarvin8/apex-code-coverage-transformer

[mcarvin8]

Concur with @xL3o , @littyphilip90 please use the latest plugin version and submit any issues you may have with the plugin to https://github.com/mcarvin8/apex-code-coverage-transformer. Let's keep this specific issue clean so that Salesforce can let us know when the API is corrected to report back correct deploy coverage data.

Your specific issue with my plugin sounds fixed by the Windows-specific path fix I included with 1.7.1 - mcarvin8/apex-code-coverage-transformer@v1.7.0...v1.7.1

Please use the latest plugin though. I removed the need to provide the -c sfdx-project.json argument. The latest plugin will find the JSON automatically as long as you run the command in your git repo. I added this during 1.6.7.

Starting with the 1.7.X builds, I have replaced the -c flag though with the specific Salesforce CLI command you are running (deploy or test) since both Salesforce CLI commands produce different code coverage outputs (formatting/etc.). Please refer to my plugin's readme for more information on the latest plugin version - https://github.com/mcarvin8/apex-code-coverage-transformer?tab=readme-ov-file#command

[mcarvin8]

Is there a way to track the linked work item (W-16286380) status somewhere (publicly) or is it private. If so, could someone please share a link?

@peternhale - Could you potentially answer @xL3o 's question above, please?

[peternhale]

@mcarvin8 @xL3o That reference is to a private issue tracking system. The root cause lies with a another team and until they make the covered lines available in the CodeCoverageResult, my team cannot reliably fix this issue.

[svc-idee-bot]

We have determined that the issue you reported exists in code owned by another team that uses only the official support channels, and thus we are closing this issue. To ensure that your issue is addressed, open an official Salesforce customer support ticket with a link to this issue. We encourage anyone experiencing this issue to do the same to increase the priority.

[diyer]

@mcarvin8 @xL3o Please feel free to file a Salesforce customer support ticket as mentioned in the comment above. That is the only way this issue will get prioritized and potentially fixed as this is not owned by our team.

[mcarvin8]

Sounds like this may have been fixed via forcedotcom/cli#3030

Can someone from Salesforce confirm?

[daphne-sfdc]

Hi @mcarvin8, The issue is on the server side and hasn't been resolved yet by the Apex Team. The CLI Team produced a workaround to try to produce better code coverage results, but the coverage percentage will not be 100% accurate until the issue on the server side is fixed.

[mcarvin8]

Is there any way to escalate this issue to the Apex Team without going through the customer support ticket process?

I do not have the time to go through the ticketing process of having a service rep engage me, having to get on a call with them to go over details, potentially trying to reproduce it, etc. for a bug which affects all users that is well known and has been open for years.

I'm not sure if there's a simple way to create a playground environment to reproduce this issue as I've seen it randomly and not sure if there's a sure fire way to reproduce this with test apex code.

But I do not think a bug fix should take dozens of developers raising their own service rep tickets and going through the chains for this ticket to be prioritized.

I echo this comment from this thread made earlier. I'd rather open an IdeaExchange or something that everyone here can post their comments on rather than having everyone open their own support cases, but there has to be a more efficient way to get bug fixes on existing code prioritized. I have no issue with how CLI bugs are resolved by creating tickets on GitHub for them. I understand GitHub isn't apart of the SLA, but that is a much more efficient process for engaging end-users and getting bug fixes than going through support reps.

I'm not sure who moved my ticket to this Repo anyway, but is there any GitHub repos that the Apex Team monitors? https://github.com/forcedotcom/salesforcedx-apex/issues perhaps?

[daphne-sfdc]

Hi @mcarvin8, Our product manager let us know that Apex Team prioritizes issues based on the number of customer reports. To escalate this ticket and increase the report count, please file a support case to make Apex Team aware of the impact of this bug. We apologize for the inconvenience and appreciate your patience.