[REVIEW]: PINE: An open source collaborative text annotation tool

[whedon]

Submitting author: @brantapl (Brant Chee)
Repository: https://github.com/jhuapl/PINE
Version: v1.0
Editor: @majensen
Reviewer: @bill-baumgartner, @bensonml
Archive: Pending

⚠️ JOSS reduced service mode ⚠️

Due to the challenges of the COVID-19 pandemic, JOSS is currently operating in a "reduced service mode". You can read more about what that means in our blog post.

Status

Status badge code:

HTML: <a href="https://joss.theoj.org/papers/09332b443f4b44060b2f2bb45f7c2485"><img src="https://joss.theoj.org/papers/09332b443f4b44060b2f2bb45f7c2485/status.svg"></a>
Markdown: [![status](https://joss.theoj.org/papers/09332b443f4b44060b2f2bb45f7c2485/status.svg)](https://joss.theoj.org/papers/09332b443f4b44060b2f2bb45f7c2485)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) by leaving comments in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@bill-baumgartner & @bensonml, please carry out your review in this issue by updating the checklist below. If you cannot edit the checklist please:

1. Make sure you're logged in to your GitHub account
2. Be sure to accept the invite at this URL: https://github.com/openjournals/joss-reviews/invitations

The reviewer guidelines are available here: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html. Any questions/concerns please let @majensen know.

✨ Please start on your review when you are able, and be sure to complete your review in the next six weeks, at the very latest ✨

Review checklist for @bill-baumgartner

Conflict of interest

• I confirm that I have read the JOSS conflict of interest (COI) policy and that: I have no COIs with reviewing this work or that any perceived COIs have been waived by JOSS for the purpose of this review.

Code of Conduct

• I confirm that I read and will adhere to the JOSS code of conduct.

General checks

• Repository: Is the source code for this software available at the repository url?
• License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
• Contribution and authorship: Has the submitting author (@brantapl) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?
• Substantial scholarly effort: Does this submission meet the scope eligibility described in the JOSS guidelines

Functionality

• Installation: Does installation proceed as outlined in the documentation?
• Functionality: Have the functional claims of the software been confirmed?
• Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

• A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
• Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
• Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
• Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?
• Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

• Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?
• A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• State of the field: Do the authors describe how this software compares to other commonly-used packages?
• Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?
• References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?

Review checklist for @bensonml

Conflict of interest

• I confirm that I have read the JOSS conflict of interest (COI) policy and that: I have no COIs with reviewing this work or that any perceived COIs have been waived by JOSS for the purpose of this review.

Code of Conduct

• I confirm that I read and will adhere to the JOSS code of conduct.

General checks

• Repository: Is the source code for this software available at the repository url?
• License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
• Contribution and authorship: Has the submitting author (@brantapl) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?
• Substantial scholarly effort: Does this submission meet the scope eligibility described in the JOSS guidelines

Functionality

• Installation: Does installation proceed as outlined in the documentation?
• Functionality: Have the functional claims of the software been confirmed?
• Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

• A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
• Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
• Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
• Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?
• Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

• Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?
• A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• State of the field: Do the authors describe how this software compares to other commonly-used packages?
• Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?
• References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?

[whedon]

Hello human, I'm @whedon, a robot that can help you with some common editorial tasks. @bill-baumgartner, @bensonml it looks like you're currently assigned to review this paper 🎉.

⚠️ JOSS reduced service mode ⚠️

Due to the challenges of the COVID-19 pandemic, JOSS is currently operating in a "reduced service mode". You can read more about what that means in our blog post.

⭐ Important ⭐

If you haven't already, you should seriously consider unsubscribing from GitHub notifications for this (https://github.com/openjournals/joss-reviews) repository. As a reviewer, you're probably currently watching this repository which means for GitHub's default behaviour you will receive notifications (emails) for all reviews 😿

To fix this do the following two things:

1. Set yourself as 'Not watching' https://github.com/openjournals/joss-reviews:

2. You may also like to change your default settings for this watching repositories in your GitHub profile here: https://github.com/settings/notifications

For a list of things I can do to help you, just type:

@whedon commands

For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:

@whedon generate pdf

[whedon]

PDF failed to compile for issue #2726 with the following error:

Can't find any papers to compile :-(

[majensen]

@whedon generate pdf from branch JOSS

[whedon]

Attempting PDF compilation from custom branch JOSS. Reticulating splines etc...

[whedon]

👉📄 Download article proof 📄 View article proof on GitHub 📄 👈

[majensen]

@bill-baumgartner @bensonml - your TL;DR:

• Your review checklists are at the top of the issue
• PDF of draft paper is here
• The software is in this repo
• Make comments for improvements in this issue. It is a conversation with the author; goal is to improve the package in "real time"
• For more technical issues, make an issue directly in the repo.
• Have fun
• Ping me with any questions
  Thank you for your service!

[brantapl]

Thanks everyone! I had trouble nominating anyone who could install this.

[bensonml]

Looks like something went awry - figure 1 in the pre-print is missing

[majensen]

@openjournals/dev - pdf generated on a custom branch is missing the rendering of figure "architecture.png". That file appears to be present in the directory containing paper.md in the branch. Can you advise? Thanks

[arfon]

@majensen - the syntax is wrong. This should fix things: JHUAPL/PINE#8

[majensen]

@brantapl can you make the fix mentioned in JHUAPL/PINE#8 ? Thanks!

[bensonml]

Problem installing PINE with Docker, opened issue: JHUAPL/PINE#10 - attn:@brantapl

[majensen]

@brantapl can you review the above issues JHUAPL/PINE#8 JHUAPL/PINE#10 ? These appear to be blockers for the reviews

I know this is a difficult season for many of us, so just let me know what your timeframe is.

[majensen]

@bill-baumgartner have you made any headway on this review? Let me know if you are having any issues-- thank you!

[bwkchee]

@ALL

@brantapl can you review the above issues JHUAPL/PINE#8 JHUAPL/PINE#10 ? These appear to be blockers for the reviews

I know this is a difficult season for many of us, so just let me know what your timeframe is.

Looking into this now. It will probably be a couple days since people are out today (for the election). Thank you!

[bill-baumgartner]

@bill-baumgartner have you made any headway on this review? Let me know if you are having any issues-- thank you!

@majensen I have made some progress, but am having install issues as noted in Issue #14.

One question for you: what is the preferred method for making comments on the manuscript? Should we simply add comments on this review page?

[majensen]

@bill-baumgartner thanks for the update - @brantapl can you or @bwkchee look into the issue noted above?

Feel free to make any paper comments right here. If there are technical issues that need a detailed discussion, you can create an issue in the repo and refer to that here in general comments.
thanks

[bill-baumgartner]

Here are my initial comments on the paper:

• PMAP is used in the first sentence but is not defined until later in the document.
• Although NER and document classification are mentioned in the paper, the types of annotation supported by PINE are not explicitly enumerated. A sentence or two in the introduction listing the supported types of annotation would be helpful to the reader.
• Inter-annotator agreement is mentioned however the specific metric used is not detailed in the manuscript as far as I could tell.
• You may want to include the Inception annotation platform in the Related work section -- I believe it has an active learning component so it seems relevant.
• A spell checker is needed for some typos throughout the manuscript.
• The description of the Pipeline wrapper on page 3 might be considered API documentation, and therefore may belong in the software documentation instead of the paper according to the review criteria.
• The manuscript would benefit from one or more figures displaying the annotation UI in my opinion.
• References that are just URLs (e.g. Gunicorn, Eve) could be replaced by footnotes if that is permitted.
• The Dublin Core reference has numbers where the author is typically listed.

Hope those are helpful. Please let me know if something is unclear.

[bensonml]

Comments on Paper:

I concur with the above comments about the paper.

Here are my own comments:

1. Grammar: add "the"

   1. We have developed an extensible framework to support annotation of plain text
   2. We have developed an extensible framework to support the annotation of plain text

2. References: Citation Syntax: maybe use a reference or inline url instead of a footnote for PMAP? Similar journals such as PLOS recommends either for referencing a website:

   1. reference style: PMAP: The Johns Hopkins Precision Medicine Analytics Platform. 2019. Precision Medicine Portal | Johns Hopkins Medicine. [online] Available at: https://pm.jh.edu [Accessed 19 November 2020].
   2. inline style: "...on the Johns Hopkins University Precision Medicine Analytics Platform (PMAP) (https://pm.jh.edu)"

3. Grammar: better wording ?
   from: "Docker Compose and Kubernetes allow for horizontal scaling of various components if necessary such as machine learning containers, annotation user interface containers, etc. "
   to: "Docker Compose and Kubernetes allow for horizontal scaling of various components if necessary such as machine learning containers, annotation user interface containers, etc. "

4. References: Citation Syntax: the Dublin Core Metadata Elements Set reference needs to include the initialism "ISO" instead of just the number "15836". Examples of reference syntax for the Dublin Core set in literature:

   1. Dublin Core (ISO 15836) http://dublincore.org/
      1. as used in https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3224535/#B61
   2. The Dublin Core Metadata Initiative (DCMI) Element Set (ISO 15836:2009)
      1. as used in https://pubmed.ncbi.nlm.nih.gov/22075810/
   3. ISO Standard 15836:2009 of February 2009 [ISO15836]
      1. as described on https://www.dublincore.org/specifications/dublin-core/dces/

5. Grammar:
   from : PINE is a web based application that
   to: PINE is a web**-**based application that

6. Grammar:
   from: changes are tracked and store. This
   to: changes are tracked and stored. This

7. References: Citation Syntax for: Gunicorn Python, W. Don't think references are really needed for frameworks (e.g. EVE) or technologies like Mongo, Redis, etc. I looked at several other papers published in JOSS, and they do not have such references. Concur that maybe they could be either footnotes, or simply inline citations (just put the url in the text). Besides, the reference for Gunicorn just doesn't make sense: "Python, W."

8. Glitch: The figure ?? depicts thearchitecture of PINE.

   1. This was supposedly fixed as part of Fixing image syntax JHUAPL/PINE#8 , but I'm not able to see it. I don't know if it is possible to re-build/render the pdf, but this needs to be verified before being published.

9. Software Documentation including API: the discussion about need to implement the Pipeline class appears to fall under the material that should not be included in the paper, as described under the review criteria in the box (link to review criteria). It is recommended to rewrite this section. I believe it is appropriate to point out this is the place for extending/adding pipelines, but doesn't necessarily need to list all of the functions in this class. Furthermore, I could not find this information is not found anywhere in the PINE github repo. (in other words: this needs to be part of documentation in the PINE github repo)

   1. "Note the paper should not include software documentation such as API (Application Programming Interface) functionality, as this should be outlined in the software documentation."

10. References: Citation Syntax and grammar: The Acronyms section

    1. Precision Medicine Analytics Platoform --> Precision Medicine Analytics Platform
    2. Use semicolons to separate acronyms. It currently comes across as a single run on sentence:
       1. PMAP - Precision Medicine Analytics Platoform NLP - Natural Language processing PINE -PMAP Interface for NLP Experimentation
       2. PMAP - Precision Medicine Analytics Platform; NLP - Natural Language processing; PINE -PMAP Interface for NLP Experimentation;

11. Grammar:
    from: The wraper is similar
    to: The wrapper is similar

12. Grammar:
    from: Turnbull, J. (2014). The docker book: Containerization is the new virtualization.
    to: Turnbull, J. (2014). The Docker Book: Containerization is the new virtualization.

---

Functionality

Installation: Does installation proceed as outlined in the documentation?

1. "Docker Environments" installation/running ok: ✅
2. "Development Environment" installation/running ⚠️
   1. I needed to first run pip install -U setuptools in order to get the backend to work
   2. just ./setup_dev_stack.sh did NOT work, nor going into backend/ and starting there (following README.md instructions).
   3. I was testing on a clean ubuntu 20.04 install running python 3.6; It would stop with essentially the same error (see below) for two packages:

• overrides==2.8.0
• python-json-logger==0.1.11

but the error would be resolved if I ran pip install -U setuptools before pipenv install --dev

key to figuring this out was https://stackoverflow.com/questions/36296134/attributeerror-install-layout-when-attempting-to-install-a-package-in-a-virtual

error snippet:

      File "/home/mark/.pyenv/versions/3.6.0/lib/python3.6/distutils/cmd.py", line 103, in __getattr__
        raise AttributeError(attr)
    AttributeError: install_layout

To clarify the above comment - the install above was on a clean install of ubuntu 20.04. I was able to run the backend build (pipenv install --dev) on macOS 10.15.7 (Catalina) without any problems.

[bensonml]

PINE is clearly an Impressive piece of work, and it is obvious they put a lot of effort into this project.

[majensen]

@brantapl - when you have a chance, please go over the comments from @bill-baumgartner and @bensonml above. I think we are closing in, but it looks like there is some work to do.

Once these have been addresses, reviewers please have a look at the result and then go over the items that are currently unchecked in your review forms above.
Thanks for all the work guys-

[majensen]

@brantapl @bwkchee - So it's been about a month since my last ping. Would it be better to pause this review until you have more time to work on the paper?
@bill-baumgartner any new progress?
thanks

[bill-baumgartner]

@majensen - no new progress on my end. I have been able to install the software, but have not been able to evaluate its use. I believe I am currently blocked by this issue.

[majensen]

@brantapl @bwkchee can I hear from you about any progress on this work, in particular JHUAPL/PINE#17 which is blocking review?
Thanks

[majensen]

@bill-baumgartner @bensonml I am having trouble getting a response from @brantapl on external channels. I will put this review in the "paused" state. Will ping everyone if I can rev it up again.

[majensen]

@jhuapl-lglenden - Can you comment on the status of this paper in JOSS, submitted by your colleague @bwkchee? Thanks for any help you can provide. Feel free to contact me at maj -dot- fortinbras -at- gmail -dot- com.

[danielskatz]

If we don't get feedback about the process of addressing the reviewer comments on the submission in another 3 weeks, we will reject it

[majensen]

@danielskatz it has been 4 weeks since the in-danger-of-rejection label was added. See #2726 (comment).

[danielskatz]

@majensen - thanks for the reminder

[danielskatz]

@brantapl - given your lack of responsiveness, this submission will now be rejected. If you fix the issues, you can resubmit it

[danielskatz]

@whedon reject

[whedon]

Paper rejected.