Skip to content

Commit

Permalink
Add --test-plan-config flag & deprecate --test-run (#36)
Browse files Browse the repository at this point in the history 
Change Description: The --test-run flag named itself off of the internal implementation of xcresult's ActionTestPlanRunSummary which is used to represent the test plan configuration runs. However, this name is not that transparent to users of xcparse unless theyre familiar with the xcresult internals. Due to this, we should have the flag use the name users who only see the XCResult from within Xcode would expect & there Xcode always refers to it as "Configurations" of the test plan so "--test-plan-config" seems most appropriate to convey that we'll split on each test plan configuration you have in the xcresult.

Test Plan/Testing Performed: Ran the new flag name and was I got the correct output. Ran with the deprecated flag & ensured I got a warning message, but still got exactly what I'd expect.
  • Loading branch information
@abotkin-cpi
abotkin-cpi authored Nov 22, 2019
1 parent e970f79 commit c9c4cfc
Show file tree
Hide file tree
Showing 5 changed files with 40 additions and 27 deletions.
34 changes: 17 additions & 17 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ Below are a few examples of common commands. For further assistance, use the --h
### Screenshots

```
xcparse screenshots --os --model --test-run /path/to/Test.xcresult /path/to/outputDirectory
xcparse screenshots --os --model --test-plan-config /path/to/Test.xcresult /path/to/outputDirectory
```

This will cause screenshots to be exported like so:
Expand All @@ -35,21 +35,21 @@ Options can be added & remove to change the folder structure used for export. U

Options available include:

| Option | Description |
|------------------|-----------------------------------------|
| ```--model``` | Divide by test target model |
| ```--os``` | Divide by test target operating system |
| ```--test-run``` | Divide by test run configuration |
| ```--test``` | Divide by test |
| Option | Description |
|--------------------------|-----------------------------------------|
| ```--model``` | Divide by test target model |
| ```--os``` | Divide by test target operating system |
| ```--test-plan-config``` | Divide by test run configuration |
| ```--test``` | Divide by test |

See ```xcparse screenshots --help``` for a full-listing

#### Test Status

The ```--test-status``` option can allow for whitelisting only screenshots from tests that have a status that matches at least one of the provided status strings

| Examples | Description |
|------------------|-----------------------------------------|
| Examples | Description |
|-------------------------------------|--------------------------------|
| ```--test-status Success``` | Passing tests only |
| ```--test-status Failure``` | Failing tests only |
| ```--test-status Success Failure``` | Passing or failing tests only |
Expand All @@ -61,8 +61,8 @@ Test status strings can be found by using verbose mode with the screenshots sub-

The ```--activity-type``` option allows for whitelisting screenshots whose activity type matches at least one of the provided activity type strings.

| Examples | Description |
|------------------|-----------------------------------------|
| Examples | Description |
|-----------------------------------------------------------------------------|-------------------------------|
| ```--activity-type com.apple.dt.xctest.activity-type.testAssertionFailure```| Test failure screenshots only |
| ```--activity-type attachmentContainer userCreated``` | User created screenshots only |

Expand All @@ -76,13 +76,13 @@ Therefore, these two are option calls are equivalent:

Activity types can be found in verbose mode. Below are a listing of common ones:

| Activity Type | Description |
|------------------|-----------------------------------------|
| com.apple.dt.xctest.activity-type.attachmentContainer| Placeholder activity that contains an attachment, may contain user created screenshot |
| com.apple.dt.xctest.activity-type.deletedAttachment | Deleted attachment placeholder activity |
| com.apple.dt.xctest.activity-type.internal | Internal test step, may have automatic screenshot to show test progression |
| Activity Type | Description |
|--------------------------------------------------------|-----------------------------------------|
| com.apple.dt.xctest.activity-type.attachmentContainer | Placeholder activity that contains an attachment, may contain user created screenshot |
| com.apple.dt.xctest.activity-type.deletedAttachment | Deleted attachment placeholder activity |
| com.apple.dt.xctest.activity-type.internal | Internal test step, may have automatic screenshot to show test progression |
| com.apple.dt.xctest.activity-type.testAssertionFailure | Step where the test failed in an assertion, may have failure screenshot |
| com.apple.dt.xctest.activity-type.userCreated | User created screenshot/attachment |
| com.apple.dt.xctest.activity-type.userCreated | User created screenshot/attachment |

### Attachments

Expand Down
12 changes: 9 additions & 3 deletions Sources/xcparse/AttachmentsCommand.swift
View file
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,8 @@ struct AttachmentsCommand: Command {

var divideByModel: OptionArgument<Bool>
var divideByOS: OptionArgument<Bool>
var divideByTestPlanRun: OptionArgument<Bool>
var divideByTestRun: OptionArgument<Bool>
var divideByTestPlanConfig: OptionArgument<Bool>
var divideByTest: OptionArgument<Bool>

var utiWhitelist: OptionArgument<[String]>
Expand All @@ -37,7 +38,8 @@ struct AttachmentsCommand: Command {

divideByModel = subparser.add(option: "--model", shortName: nil, kind: Bool.self, usage: "Divide attachments by model")
divideByOS = subparser.add(option: "--os", shortName: nil, kind: Bool.self, usage: "Divide attachments by OS")
divideByTestPlanRun = subparser.add(option: "--test-run", shortName: nil, kind: Bool.self, usage: "Divide attachments by test plan configuration")
divideByTestRun = subparser.add(option: "--test-run", shortName: nil, kind: Bool.self, usage: "Deprecated. Use --test-plan-config")
divideByTestPlanConfig = subparser.add(option: "--test-plan-config", shortName: nil, kind: Bool.self, usage: "Divide attachments by test plan configuration")
divideByTest = subparser.add(option: "--test", shortName: nil, kind: Bool.self, usage: "Divide attachments by test")

utiWhitelist = subparser.add(option: "--uti", shortName: nil, kind: [String].self, strategy: .upToNextOption,
Expand Down Expand Up @@ -68,11 +70,15 @@ struct AttachmentsCommand: Command {
let xcpParser = XCPParser()
xcpParser.console.verbose = verbose

if let _ = arguments.get(self.divideByTestRun) {
xcpParser.console.writeMessage("\nThe \"--test-run\" flag is deprecated & will be removed in a future release. Please replace with \"--test-plan-config\"\n")
}

// Let's set up our export options
var options = AttachmentExportOptions(addTestScreenshotsDirectory: false,
divideByTargetModel: arguments.get(self.divideByModel) ?? false,
divideByTargetOS: arguments.get(self.divideByOS) ?? false,
divideByTestRun: arguments.get(self.divideByTestPlanRun) ?? false,
divideByTestPlanConfig: arguments.get(self.divideByTestPlanConfig) ?? (arguments.get(self.divideByTestRun) ?? false),
divideByTest: arguments.get(self.divideByTest) ?? false)
if let allowedUTIsToExport = arguments.get(self.utiWhitelist) {
options.attachmentFilter = {
Expand Down
2 changes: 1 addition & 1 deletion Sources/xcparse/CommandRegistry.swift
View file
Original file line number Diff line number Diff line change
Expand Up @@ -67,7 +67,7 @@ struct CommandRegistry {
let options = AttachmentExportOptions(addTestScreenshotsDirectory: true,
divideByTargetModel: false,
divideByTargetOS: false,
divideByTestRun: false,
divideByTestPlanConfig: false,
xcresulttoolCompatability: xcresulttoolCompatability,
attachmentFilter: {
return UTTypeConformsTo($0.uniformTypeIdentifier as CFString, "public.image" as CFString)
Expand Down
13 changes: 10 additions & 3 deletions Sources/xcparse/ScreenshotsCommand.swift
View file
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,8 @@ struct ScreenshotsCommand: Command {
var addTestScreenshotDirectory: OptionArgument<Bool>
var divideByModel: OptionArgument<Bool>
var divideByOS: OptionArgument<Bool>
var divideByTestPlanRun: OptionArgument<Bool>
var divideByTestRun: OptionArgument<Bool>
var divideByTestPlanConfig: OptionArgument<Bool>
var divideByTest: OptionArgument<Bool>

var testStatusWhitelist: OptionArgument<[String]>
Expand All @@ -39,7 +40,8 @@ struct ScreenshotsCommand: Command {
addTestScreenshotDirectory = subparser.add(option: "--legacy", shortName: nil, kind: Bool.self, usage: "Create \"testScreenshots\" directory in outputDirectory & put screenshots in there")
divideByModel = subparser.add(option: "--model", shortName: nil, kind: Bool.self, usage: "Divide screenshots by model")
divideByOS = subparser.add(option: "--os", shortName: nil, kind: Bool.self, usage: "Divide screenshots by OS")
divideByTestPlanRun = subparser.add(option: "--test-run", shortName: nil, kind: Bool.self, usage: "Divide screenshots by test plan configuration")
divideByTestRun = subparser.add(option: "--test-run", shortName: nil, kind: Bool.self, usage: "Deprecated. Use --test-plan-config")
divideByTestPlanConfig = subparser.add(option: "--test-plan-config", shortName: nil, kind: Bool.self, usage: "Divide attachments by test plan configuration")
divideByTest = subparser.add(option: "--test", shortName: nil, kind: Bool.self, usage: "Divide screenshots by test")

testStatusWhitelist = subparser.add(option: "--test-status", shortName: nil, kind: [String].self, strategy: .upToNextOption,
Expand Down Expand Up @@ -70,10 +72,15 @@ struct ScreenshotsCommand: Command {
let xcpParser = XCPParser()
xcpParser.console.verbose = verbose

if let _ = arguments.get(self.divideByTestRun) {
xcpParser.console.writeMessage("\nThe \"--test-run\" flag is deprecated & will be removed in a future release. Please replace with \"--test-plan-config\"\n")
}

// Let's set up our export options
var options = AttachmentExportOptions(addTestScreenshotsDirectory: arguments.get(self.addTestScreenshotDirectory) ?? false,
divideByTargetModel: arguments.get(self.divideByModel) ?? false,
divideByTargetOS: arguments.get(self.divideByOS) ?? false,
divideByTestRun: arguments.get(self.divideByTestPlanRun) ?? false,
divideByTestPlanConfig: arguments.get(self.divideByTestPlanConfig) ?? (arguments.get(self.divideByTestRun) ?? false),
divideByTest: arguments.get(self.divideByTest) ?? false,
attachmentFilter: {
return UTTypeConformsTo($0.uniformTypeIdentifier as CFString, "public.image" as CFString)
Expand Down
6 changes: 3 additions & 3 deletions Sources/xcparse/XCPParser.swift
View file
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ import Foundation
import SPMUtility
import XCParseCore

let xcparseCurrentVersion = Version(2, 0, 0)
let xcparseCurrentVersion = Version(2, 1, 0)

extension Foundation.URL {
func fileExistsAsDirectory() -> Bool {
Expand Down Expand Up @@ -71,7 +71,7 @@ struct AttachmentExportOptions {
var addTestScreenshotsDirectory: Bool = false
var divideByTargetModel: Bool = false
var divideByTargetOS: Bool = false
var divideByTestRun: Bool = false
var divideByTestPlanConfig: Bool = false
var divideByTest: Bool = false

var xcresulttoolCompatability = XCResultToolCompatability()
Expand Down Expand Up @@ -129,7 +129,7 @@ struct AttachmentExportOptions {
return baseURL
}

if self.divideByTestRun {
if self.divideByTestPlanConfig {
if self.xcresulttoolCompatability.supportsUnicodeExportPaths != true {
let asciiTestPlanRunName = testPlanRunName.lossyASCIIString() ?? testPlanRunName
return baseURL.appendingPathComponent(asciiTestPlanRunName, isDirectory: true)
Expand Down

0 comments on commit c9c4cfc

Please sign in to comment.