New Collabra paper draft

[rempsyc]

New Collabra paper draft.

Real time PDF: https://github.com/easystats/performance/blob/collabra_paper/papers/Collabra/paper.pdf

Edit: reminder to use [skip ci] in this PR

Follow-up on #544

[rempsyc]

Should we streamline the review process for Collabra?

Streamlined review?
Authors whose articles have been rejected within the previous 365 days from other journals for reasons that are not due to lack of scientific, methodological, or ethical rigor are welcome to submit prior reviews and decision letters along with their submission, and request a streamlined review in their cover letter.

[rempsyc]

@bwiernik, you mentioned the paper would need some polishing for Collabra. Any specific suggestions on the current version?

I did add the height-weight code demo as discussed earlier.

@DominiqueMakowski mentioned maybe arguing more strongly in favour of our multiple methods approach. But I I am not really able to do that. So Dom would you like to make an attempt? If not no worry, we can leave as is.

Also, there is no rticles template for Collabra, so I didn't know what template to use for the paper but they use double blind review so I used the preprint archive thing for now because I didn't want to have to change all the meta-info parameters to classic rmarkdown manually again.

[rempsyc]

Also, in the paper, we cite easystats as follows:

Lüdecke, Daniel, Mattan S. Ben-Shachar, Indrajeet Patil, Brenton M. Wiernik, Etienne Bacher, Rémi Thériault, and Dominique Makowski. 2022. “Easystats: Framework for Easy Statistical Modeling, Visualization, and Reporting.” CRAN. https://easystats.github.io/easystats/.

However, that does not correspond to the order of authors on the package website or using report::cite_easystats().

Lüdecke, D., Makowski, D., Ben-Shachar, M. S., Patil, I., Wiernik, B. M., Bacher, Etienne, & Thériault, R. (2023). easystats: Streamline model interpretation, visualization, and reporting (0.6.0) [R package]. https://easystats.github.io/easystats/ (Original work published 2019)

So @IndrajeetPatil which citation/author order is correct!?

[IndrajeetPatil]

Also, in the paper, we cite easystats as follows:

Lüdecke, Daniel, Mattan S. Ben-Shachar, Indrajeet Patil, Brenton M. Wiernik, Etienne Bacher, Rémi Thériault, and Dominique Makowski. 2022. “Easystats: Framework for Easy Statistical Modeling, Visualization, and Reporting.” CRAN. https://easystats.github.io/easystats/.

However, that does not correspond to the order of authors on the package website or using report::cite_easystats().

Lüdecke, D., Makowski, D., Ben-Shachar, M. S., Patil, I., Wiernik, B. M., Bacher, Etienne, & Thériault, R. (2023). easystats: Streamline model interpretation, visualization, and reporting (0.6.0) [R package]. https://easystats.github.io/easystats/ (Original work published 2019)

So @IndrajeetPatil which citation/author order is correct!?

Why are we citing the easystats meta-paper?! I don't think we should cite it. We haven't done so for any of the other publications. We can just mention it, and even link to the GitHub organization, but I don't think we need to cite it here. That said, we can cite the relevant JOSS papers. E.g., in ggstatsplot paper, this is what I do:

So @IndrajeetPatil which citation/author order is correct!?

At some point, we will need to decide on the order, but I feel it's a bit early for that.

[rempsyc]

I believe @DominiqueMakowski added the reference to easystats. I don't think it's necessarily a bad idea even if it's early and that things could change. I like for instance referring to the website.

I mean that's the outcome of using citation("easystats") so people are probably already citing that. If that's a problem maybe we need to change it?

I had not realized that the two titles were different though. As you say, the first one is probably the meta-paper, whereas in the second one, we're simply citing the website/package, so I think that's fine?

[rempsyc]

Anybody else would like to make a last reread of the paper before I submit to Collabra @easystats/core-team? I would like to submit next weekend if nobody requests changes. Thanks.

[DominiqueMakowski]

It would be nice to have a discussion or some thoughts about what to do after outlier detection: re-run analyses? Show results with and without outliers? Describe outlying sample? I don't think there is one single best approach but we could at least mention some caveats and important things.

One thing we might highlight is the need for reporting the characteristics of the outliers (how many were removed, the percentage, and eventually the features). For instance, we could add an argument in report_participants() and report_sample(), outliers that takes a vector of outliers / the output of check_outliers, and that would add a description of the outliers.

In report_participants, could look like... "Description of whole sample. Out of this sample, X (3.54%) of participants were flagged as outliers (demographic summary), leaving X participants in the final set (new demographics without)."

in report_sample(): @strengejacke ideas how to present it?

[rempsyc]

It would be nice to have a discussion or some thoughts about what to do after outlier detection: re-run analyses? Show results with and without outliers?

We do have already a section called "Handling Outliers". And then we go about describing the different types of outliers, and what to do for each type of outlier, and whether to keep, exclude, or winsorize depending on the case. Does that correspond to what you wanted?

One thing we might highlight is the need for reporting the characteristics of the outliers (how many were removed, the percentage, and eventually the features)

We already kind of do that in the transparency section no? (I added a mention of the percentage though, thanks)

 we could add an argument in report_participants() and report_sample(), outliers that takes a vector of outliers / the output of check_outliers, and that would add a description of the outliers.

It's true that we don't mention describing the characteristics of the outliers; perhaps I could add a paragraph on that. But do we really need a new function, when we can simply subset on the data directly with the outlier object or as you say a vector of outliers directly, if you have it?

report_participants(data[which(outliers),])

And I wonder if that assumes homogeneity of the outliers, which they should not be for random outliers. Even if they are, there would be no way of knowing unless you also report measures of homogeneity. But with so few observations as 1 to maybe 5 outliers, how meaningful would these data be? Especially if they are e.g., error outliers that are excluded, then any sample description would be meaningless. Maybe only for interesting outliers to try to figure out a pattern? Please convince me why you think this is important :)

[rempsyc]

Actually we don't discuss benefits and limitations of the consensus-based approach right now. @DominiqueMakowski actually what are the method's benefits and limitations exactly? Since I believe you came up with the method, if you give me some pointers I may be able to flesh it out in the paper.

[rempsyc]

Alright, so I have finished updating the cover letter and the manuscript, and writing the response to reviewers.

@IndrajeetPatil @strengejacke @DominiqueMakowski @mattansb that's the last opportunity to review those documents before I resubmit. Also two small things for the paper:

1. On p. 6 of the PDF, I added a summary decision table. Thoughts? Is it OK?
2. On p. 7, I wanted to draw a nice scatter plot to show the model expectation without adding too many lines of code. But I was not able to do this using see, ggplot2, or base R, so I just rempsyc::nice_scatter. But I don't know how I feel about using rempsyc here. Is it OK?

[rempsyc]

I'll create a new PR for JOSE. Should we merge this then?

[mattansb]

Merge? Or delete?

[rempsyc]

Hum, delete probably, yeah. I was thinking in case we want to keep archives or files and stuff but it was bugging the JOSE submission system anyway so I deleted it as well in the other PR.