diff --git a/Dockerfile b/Dockerfile index 52cab60..64c8e17 100644 --- a/Dockerfile +++ b/Dockerfile @@ -1,4 +1,4 @@ -FROM taccwma/tacc-docs:v0.6.0 +FROM taccwma/tacc-docs:v0.8.0 # To archive TACC content, before replacing it RUN mv /docs /docs-from-tacc diff --git a/user-guide/docs/analysis/hvsrweb.md b/user-guide/docs/analysis/hvsrweb.md index cfd4e13..ee53af5 100644 --- a/user-guide/docs/analysis/hvsrweb.md +++ b/user-guide/docs/analysis/hvsrweb.md @@ -2,31 +2,31 @@ The horizontal-to-vertical spectral ratio (HVSR) method has become an increasingly popular tool for developing a quick and reliable estimate of a site’s fundamental natural frequency (f0). HVSRweb is an open-source, web-based application for performing HVSR calculations in a convenient, reliable, and statistically-consistent manner. HVSRweb allows the user to upload three-component ambient noise records and perform the HVSR calculation in the cloud, with no installation required. The HVSR calculation can be performed using a single combined horizontal component (e.g., geometric-mean, squared-average) or multiple rotated horizontal components to investigate azimuthal variability. HVSRweb can reject spurious time windows using an automated frequency-domain window-rejection algorithm, removing the need for subjective and time-consuming user interaction. It also facilitates the use of lognormal statistics for f0, allowing for a consistent statistical framework capable of quantifying measurement uncertainty in terms of either frequency or its reciprocal, period. In addition, HVSRweb presents the opportunity to rapidly incorporate new developments in processing and quantification of uncertainty as they emerge, and encourages standardization of HVSR processing across users while avoiding the challenges associated with traditional approaches that require the user to perform regular updates to keep pace with new developments. -### [Step 1](#step1) { #step1 } +### Step 1 { #step1 } ![](./imgs/HVSR-1.png) -### [Step 2](#step2) { #step2 } +### Step 2 { #step2 } ![](./imgs/HVSR-2.png) -### [Step 3](#step3) { #step3 } +### Step 3 { #step3 } ![](./imgs/HVSR-3.png) -### [Step 4](#step4) { #step4 } +### Step 4 { #step4 } ![](./imgs/HVSR-4.png) -### [Step 5](#step5) { #step5 } +### Step 5 { #step5 } ![](./imgs/HVSR-5.png) -### [Step 6](#step6) { #step6 } +### Step 6 { #step6 } ![](./imgs/HVSR-6.png) -### [Step 7](#step7) { #step7 } +### Step 7 { #step7 } ![](./imgs/HVSR-7.png) diff --git a/user-guide/docs/analysis/matlab.md b/user-guide/docs/analysis/matlab.md index 3d7867d..c0c093b 100644 --- a/user-guide/docs/analysis/matlab.md +++ b/user-guide/docs/analysis/matlab.md @@ -23,7 +23,7 @@ We provide two MATLAB applications in the Workspace: NOTE: Because MATLAB is an interactive program that you access in a GUI, you will not have the same responsiveness as running on your local laptop. We provide MATLAB as a convenience to interacting with your data in the Data Depot without having to download it to your local system, and also the version running on HPC resources is great for large datasets or heavy workloads that are too much for your laptop. -### [How to Start a MATLAB Interactive Session in the Workspace](#start) { #start } +### How to Start a MATLAB Interactive Session in the Workspace { #start } MATLAB that runs on VM @@ -37,7 +37,7 @@ MATLAB that runs on VM
  • Click Run to start your interactive session.
    • -### [How to Submit a MATLAB Batch job in the Workspace](#submit) { #submit } +### How to Submit a MATLAB Batch job in the Workspace { #submit } MATLAB Batch that runs on HPC diff --git a/user-guide/docs/analysis/overview.md b/user-guide/docs/analysis/overview.md index 8c412fc..4d7a086 100644 --- a/user-guide/docs/analysis/overview.md +++ b/user-guide/docs/analysis/overview.md @@ -1,14 +1,14 @@ ## Analysis Applications -### [Requesting New Applications](#newapps) { #newapps } +### Requesting New Applications { #newapps } DesignSafe regularly adds new software applications in support of natural hazards engineering research. You may contact DesignSafe by submitting a help ticket if you would like to request the addition of a software application to the Workspace. -### [Getting Your Own HPC Application](#software) { #software } +### Getting Your Own HPC Application { #software } For those researchers with larger computational needs on the order of tens of thousands, or even millions of core-hours, or if you have a software application that we don't support in the web portal, you may request your own allocation of computing time on TACC's HPC systems. Your files can still be stored in the Data Depot, allowing you to share your research results with your team members, as well as curate and publish your findings. -### [Commercial/Licensed Applications](#commercial) { #commercial } +### Commercial/Licensed Applications { #commercial } The DesignSafe infrastructure includes support for commercial/licensed software. Wile in some cases licenses can be provided by the DesignSafe project itself, not all vendors will make licenses available for larger open communities at reasonable cost. You may contact DesignSafe by submitting a help ticket if you have questions regarding a commercial software application. diff --git a/user-guide/docs/analysis/swbatch.md b/user-guide/docs/analysis/swbatch.md index 6d4c31c..0c8b670 100644 --- a/user-guide/docs/analysis/swbatch.md +++ b/user-guide/docs/analysis/swbatch.md @@ -14,7 +14,7 @@ Vantassel, J.P., Cox, B.R., (2020). SWinvert: A workflow for performing rigorous Note: For software, version specific citations should be preferred to general concept citations, such as that listed above. To generate a version specific citation for SWbatch, please use the citation tool for that specific version on the SWbatch archive. -### [Getting Started](#start) { #start } +### Getting Started { #start } There are two ways of using SWbatch: @@ -23,7 +23,7 @@ There are two ways of using SWbatch:
  • Or directly through the DesignSafe-CI Research Workbench.
    • -#### [Instructions for using the Jupyter Workflow](#start-jupyter) { #start-jupyter } +#### Instructions for using the Jupyter Workflow { #start-jupyter }
    1. Visit the SWprepost GitHub and follow the Getting Started instructions. The advanced example walks you through using the SWinvert surface wave inversion Jupyter workflow. (30 minutes)
      2. @@ -32,7 +32,7 @@ There are two ways of using SWbatch:
    2. Enjoy!
    -#### [Instructions for using the DesignSafe-CI Research Workbench](#start-ds) { #start-ds } +#### Instructions for using the DesignSafe-CI Research Workbench { #start-ds }
    1. Visit the SWprepost GitHub and follow the Getting Started instructions. This will introduce you to SWprepost and the SWinvert workflow, which is required before proceeding to step 2 in these instructions. (30 minutes)
      2. diff --git a/user-guide/docs/curating/bestpractices.md b/user-guide/docs/curating/bestpractices.md index ac4f6fd..3fc13c8 100644 --- a/user-guide/docs/curating/bestpractices.md +++ b/user-guide/docs/curating/bestpractices.md @@ -1,10 +1,10 @@ ### Data Collections Development -#### [Accepted Data ](#accepteddata) { #accepteddata } +#### Accepted Data { #accepteddata } The DDR accepts engineering datasets generated through simulation, hybrid simulation, experimental, and field research methods regarding the impacts of wind, earthquake, and storm surge hazards, as well as debris management, fire, and blast explosions. We also accept social and behavioral sciences (SBE) data encompassing the study of the human dimensions of hazards and disasters. As the field and the expertise of the community evolves we have expanded our focus to include datasets related to COVID-19. Data reports, publications of Jupyter notebooks, code, scripts, lectures, and learning materials are also accepted. -#### [Accepted and Recommended File Formats ](#acceptedfileformats) { #acceptedfileformats } +#### Accepted and Recommended File Formats { #acceptedfileformats } Due to the diversity of data and instruments used by our community, there are no current restrictions on the file formats users can upload to the DDR. However, for long-term preservation and interoperability purposes, we recommend and promote storing and publishing data in open formats, and we follow the Library of Congress Recommended Formats. @@ -24,7 +24,7 @@ Below is an adaptation of the list of recommended formats for data and documenta * CODE: (tcl files, py files) Jupyter Notebook * Seismology: SEED -#### [Data Size](#datasize) { #datasize } +#### Data Size { #datasize } Currently we do not pose restrictions on the volume of data users upload to and publish in the DDR. This is meant to accommodate the vast amount of data researchers in the natural hazards community can generate, especially during the course of large-scale research projects. @@ -40,7 +40,7 @@ Extensive support for data curation can be found in the here. -#### [Selecting a Project Type](#selectingprojecttype) { #selectingprojecttype } +#### Selecting a Project Type { #selectingprojecttype } Depending on the research method pursued, users may curate and publish data as "Experimental", "Simulation", "Hybrid Simulation," or "Field Research" project type. The Field Research project type accommodates "Interdisciplinary Datasets" involving engineering and/or social science collections. @@ -73,7 +73,7 @@ Based on data models designed by experts in the field, t Users should select the project type that best fits their research method and dataset. If the data does not fit any of the above project types, they can select project type" Other." In project type "Other" users can curate and publish standalone reports, learning materials, white papers, conference proceedings, tools, scripts, or data that does not fit with the research models mentioned above. -#### [Working in My Project](#working) { #working } +#### Working in My Project { #working } Once the project type is selected, the interactive interface in My Project will guide users through the curation and publication steps through detailed onboarding instructions. @@ -83,33 +83,33 @@ Because My Project is a shared space, it is recommended that teams select a data After data is published users can still work on My Project for progressive publishing of new experiments, missions or simulations within the project, to version and/or to edit or amend the existing publication. See amends and versions in this document. -#### [General Research Data Best Practices](#bestpractices) { #bestpractices } +#### General Research Data Best Practices { #bestpractices } Below we include general research data best practices but we strongly recommend to review the available Data Curation Primers for more specific directions on how to document and organize specific research data types. -##### [Proprietary Formats](#bestpractices-formats) { #bestpractices-formats } +##### Proprietary Formats { #bestpractices-formats } Excel and Matlab are two proprietary file formats highly used in this community. Instead of Excel spreadsheet files, it is best to publish data as simple csv so it can be used by different software. However, we understand that in some cases (e.g. Matlab, Excel) conversion may distort the data structures. Always retain an original copy of any structured data before attempting conversions, and then check between the two for fidelity. In addition, in the DDR it is possible to upload and publish both the proprietary and the converted version, especially if you consider that publishing with a proprietary format is convenient for data reuse. -##### [Compressed Data](#bestpractices-compresseddata) { #bestpractices-compresseddata } +##### Compressed Data { #bestpractices-compresseddata } Users that upload data as a zip file should unzip before curating and publishing, as zip files prevent others from directly viewing and understanding the published data. If uploading compressed files to "My Data" , it is possible to unzip it using the extraction utility available in the workspace before copying data to My Project for curation and publication. -##### [Simulation Data](#bestpractices-simulationdata) { #bestpractices-simulationdata } +##### Simulation Data { #bestpractices-simulationdata } In the Data Depot's Published directory there is a [best practices document](https://doi.org/10.17603/ds2-wsqp-fw44){:target="_blank"} for publishing simulation datasets in DesignSafe. The topics addressed reflect the numerical modeling community needs and recommendations, and are informed by the experience of the working group members and the larger DesignSafe expert cohort while conducting and curating simulations in the context of natural hazards research. These best practices focus on attaining published datasets with precise descriptions of the simulations’ designs, references to or access to the software involved, and complete publication of inputs and if possible all outputs. Tying these pieces together requires documentation to understand the research motivation, origin, processing, and functions of the simulation dataset in line with FAIR principles. These best practices can also be used by simulation researchers in any domain to curate and publish simulation data in any repository. -##### [Geospatial Data](#bestpractices-geospatial) { #bestpractices-geospatial } +##### Geospatial Data { #bestpractices-geospatial } We encourage the use of open Geospatial data formats. Within DS Tools and Applications we provide two open source software for users to share and analyze geospatial data. QGIS can handle most open format datasets and HazMapper, is capable of visualizing geo-tagged photos and GeoJSON files. To access these software users should get an account in DesignSafe. Understanding that ArcGIS software is widespread in this community in the DDR it is possible to upload both proprietary and recommended geospatial data formats. When publishing feature and raster files it is important to make sure that all of the relevant files for reuse such as the projection file and header file are included in the publication for future re-use. For example, for shapefiles it is important to publish all .shp (the file that contains the geometry for all features), .shx (the file that indexes the geometry) and .dbf (the file that stores feature attributes in a tabular format) files. -##### [Point Cloud Data](#bestpractices-pointcloud) { #bestpractices-pointcloud } +##### Point Cloud Data { #bestpractices-pointcloud } It is highly recommended to avoid publishing proprietary point cloud data extensions. Instead, users should consider publishing post-processed and open format extension data such as las or laz files. In addition, point cloud data publications may be very large. In DS, we have Potree available for users to view point cloud datasets. Through the Potree Convertor application, non-proprietary point cloud files can be converted to a potree readable format to be visualized in DesignSafe. -##### [Jupyter Notebooks](#jupyter) { #jupyter } +##### Jupyter Notebooks { #jupyter } More and more researchers are publishing projects that contain Jupyter Notebooks as part of their data. They can be used to provide sample queries on the published data as well as providing digital data reports. As you plan for publishing a Jupyter Notebook, please consider the following issues: @@ -117,7 +117,7 @@ More and more researchers are publishing projects that contain Jupyter Notebooks 1. The published area is a read-only space. In the published section, users can run notebooks, but the notebook is not allowed to write any file to this location. If the notebook needs to write a file, you as the author of the notebook should make sure the notebook is robust to write the file in each user directory. Here is an example of a published notebook that writes files to user directories. Furthermore, since the published space is read-only, if a user wants to revise, enhance or edit the published notebook they will have to copy the notebook to their mydata and continue working on the copied version of the notebook located in their mydata. To ensure that users understand these limitations, we require a readme file be published within the project that explains how future users can run and take advantage of the Jupyter Notebook. 1. Jupyter Notebooks rely on packages that are used to develop them (e.g., numpy, geopandas, ipywidgets, CartoPy, Scikit-Learn). For preservation purposes, it is important to publish a requirement file including a list of all packages and their versions along with the notebook as a metadata file. -#### [Data Organization and Description](#organization) { #organization } +#### Data Organization and Description { #organization } In My Projects, users may upload files and or create folders to keep their files organized; the latter is common when projects have numerous files. However, browsing through an extensive folder hierarchy on the web may be slower on your local computer, so users should try to use the smallest number of nested folders necessary and if possible, none at all, to improve all users’ experience. @@ -155,7 +155,7 @@ The following are good examples of data organization and description of differen * Contribution to Cold Formed Steel Seismic Design within CFS-NHERI Project. * Global Academic Hazards and Disaster research Centers Data -#### [Project Documentation](#documentation) { #documentation } +#### Project Documentation { #documentation } NH datasets can be very large and complex, so we require that users submit a data report or a readme file to publish along with their data to express information that will facilitate understanding and reuse of your project. This documentation may include the structure of the data, a data dictionary, information of where everything is, explanation of the file naming convention used, and the methodology used to check the quality of the data. The data report in this published dataset is an excellent example of documentation. @@ -163,7 +163,7 @@ To provide connections to different types of information about the published dat When applicable, we ask users to include information about their research funding in the Awards Info fields. -#### [Data Quality Control](#quality) { #quality } +#### Data Quality Control { #quality } Each data publication is unique; it reflects and provides evidence of the research work of individuals and teams. Due to the specificity, complexity, and scope of the research involved in each publication, the DDR cannot complete quality checks of the contents of the data published by users. It is the user's responsibility to publish data that is up to the best standards of their profession, and our commitment is to help them achieve these standards. In the DDR, data and metadata quality policies as well as the curation and publication interactive functions are geared towards ensuring excellence in data publications. In addition, below we include general data content quality recommendations: @@ -175,7 +175,7 @@ Researchers in NH generate enormous amounts of images. While we are not posing r It is possible to publish raw and curated data. Raw data is that which comes directly from the recording instruments (camera, apps, sensors, scanners, etc). When raw data is corrected, calibrated, reviewed, edited or post-processed in any way for publication, it is considered curated. Some researchers want to publish their raw data as well as their curated data. For users who seek to publish both, consider why it is necessary to publish both sets and how another researcher would use them. Always clarify whether your data is raw or curated in the description or in a data report/readme file including the method used to post-process it. -#### [Managing Protected Data in the DDR](#protectedddr) { #protectedddr } +#### Managing Protected Data in the DDR { #protectedddr } Users that plan to work with human subjects should have their IRB approval in place prior to storing, curating, and publishing data in the DDR. We recommend following the recommendations included in the CONVERGE series of check sheets that outline how researchers should manage/approach the lifecycle data that contain personal and sensitive information; these check sheets have also been published in the DDR. @@ -185,7 +185,7 @@ DesignSafe My Data and My Projects are secure spaces to store raw protected data Projects that do not include the study of human subjects and are not under IRB purview may still contain items with Personally Identifiable Information (PII). For example, researchers conducting field observations may capture human subjects in their documentation including work crews, passersby, or people affected by the disaster. If camera instruments capture people that are in the observed areas incidentally, we recommend that their faces and any Personally Identifiable Information should be anonymized/blurred before publishing. In the case of images of team members, make sure they are comfortable with making their images public. Do not include roofing/remodeling records containing any form of PII. When those are public records, researchers should point to the site from which they are obtained using the Referenced Data and or Related Work fields. In short, users should follow all other protected data policies and best practices outlined further in this document. -#### [Metadata Requirements](#metadatareqs) { #metadatareqs } +#### Metadata Requirements { #metadatareqs } Metadata is information that describes the data in the form of schemas. Metadata schemas provide a structured way for users to share information about data with other platforms and individuals. Because there is no standard schema to describe natural hazards engineering research data, the DDR developed data models containing elements and controlled terms for categorizing and describing NH data. The terms have been identified by experts in the NH community and are continuously expanded, updated, and corrected as we gather feedback and observe how researchers use them in their publications. @@ -437,7 +437,7 @@ Due to variations in research methods, users may not need to use all the metadat ### Data Publication -#### [Protected Data](#protecteddata) { #protecteddata } +#### Protected Data { #protecteddata } Protected data in the Data Depot Repository (DDR) are generally (but not always) included within interdisciplinary and social science research projects that study human subjects, which always need to have approval from an Institutional Review Board (IRB). We developed a data model and onboarding instructions in coordination with our CONVERGE partners to manage this type of data within our curation and publication pipelines. Additionally, CONVERGE has a series of check sheets that outline how researchers should manage data that could contain sensitive information; these check sheets have also been published in the DDR. @@ -457,7 +457,7 @@ To publish protected data researchers should pursue the following steps and requ 1. The responsibility of maintaining and managing a restricted dataset for the long term lies on the authors, and they can use TACC's Protected Data Services if they see fit. 1. Please contact DDR through a help ticket or join curation office hours prior to preparing this type of publication. -##### [De-identification Resources](#protecteddata-deidentification) { #protecteddata-deidentification } +##### De-identification Resources { #protecteddata-deidentification } The NISTIR 8053 publication De-Identification of Personal Information provides all the definitions and approaches to reduce privacy risk and enable research. @@ -465,7 +465,7 @@ Another Applications to Assist in De-identification of Human Subjects Research Data. -#### [Reusing Data Resources in your Publication](#reusingdata) { #reusingdata } +#### Reusing Data Resources in your Publication { #reusingdata } Researchers frequently use data, code, papers or reports from other sources in their experiments, simulations and field research projects as input files, to integrate with data they create, or as references, and they want to republish them. It is a good practice to make sure that this data can be reused appropriately and republished as well as give credit to the data creators. Citing reused sources is also important to provide context and provenance to the project. In the DDR you can republish or reference reused data following the next premises: @@ -487,14 +487,14 @@ Researchers frequently use data, code, papers or reports from other sources in t
    -#### [Timely Data Publication ](#timelypublication) { #timelypublication } +#### Timely Data Publication { #timelypublication } Although no firm timeline requirements are specified for data publishing, researchers are expected to publish in a timely manner. Recommended timelines for publishing different types of research data (i.e., Experimental, Simulation, and Reconnaissance) are listed in Table 1. Guidelines specific to RAPID reconnaissance data can be found at rapid.designsafe-ci.org/media/filer_public/b3/82/b38231fb-21c9-41f8-b658-f516dfee87c8/rapid-designsafe_curation_guidelines_v3.pdf
    -##### [Table 1. Recommended Publishing Timeline for Different Data Types](#table1) { #table1 } +##### Table 1. Recommended Publishing Timeline for Different Data Types { #table1 } @@ -541,7 +541,7 @@ Guidelines specific to RAPID reconnaissance data can be found at
    -##### [SOFTWARE](#licensing-software) { #licensing-software } +##### SOFTWARE { #licensing-software } If you are publishing community software, scripts, libraries, applications, etc, choose the following: @@ -676,13 +676,13 @@ Please read the L -#### [Subsequent Publishing](#publishing) { #publishing } +#### Subsequent Publishing { #publishing } With the exception of Project Type Other, which is a one time publication, in the DDR it is possible to publish datasets or works subsequently. A project can be conceived as an umbrella where reports or learning materials, code, and datasets from distinct experiments, simulations, hybrid simulations or field research missions that happen at different time periods, involve participation of distinct authors, or need to be released more promptly, can be published at different times. Each new product will have its own citation and DOI, and users may select a different license if that is appropriate for the material, (e.g. a user publishing a data report will use a Creative Commons license, and an Open Data Commons license to publish the data). The subsequent publication will be linked to the umbrella project via the citation, and to the other published products in the project through metadata. After a first publication, users can upload more data and create a new experiment/simulation/hybrid simulation or mission and proceed to curate it. Users should be aware that momentarily they cannot publish the new product following the publication pipeline. After curation and before advancing through the Publish My Project button, they should write a help ticket or attend curation office hours so that the DDR team can assist and publish the new product. -#### [Amends and Version Control ](#amends) { #amends } +#### Amends and Version Control { #amends } Once a dataset is published users can do two things to improve and or / continue their data publication: amends and version control. Amends involve correcting certain metadata fields that do not incur major changes to the existing published record, and version control includes changes to the data. Once a dataset is published, however, we do not allow title or author changes. If those changes need to be made due to omission or mistake, users have to submit a Help ticket and discuss the change with the data curator. If applicable, changes will be done by the curation team. @@ -713,7 +713,7 @@ When implementing amend and version take the following into consideration: Important: Any changes to the project’s metadata will also be updated (this update is limited to the same fields allowed in the Amend section), so there is no need to amend a newly versioned project unless you have made a mistake in the latest version. -#### [Leave Data Feedback](#feedback) { #feedback } +#### Leave Data Feedback { #feedback } We welcome feedback from users about the published datasets. For this, users can click on the "Leave Feedback" button at the top of the data presentation on the data publication landing pages. We suggest that feedback is written in a positive, constructive language. The following are examples of feedback questions and concerns: @@ -730,7 +730,7 @@ We welcome feedback from users about the published datasets. For this, users can
  • Congratulations.
    • -#### [Marketing Datasets](#marketing) { #marketing } +#### Marketing Datasets { #marketing } Datasets take a lot of work to produce; they are important research products. By creating a complete, organized, and clearly described publication in DDR, users are inviting others to reuse and cite their data. Researchers using published data from DDR must cite it using the DOI, which relies on the     DataCite schema for accurate citation. For convenience, users can retrieve a formatted citation from the published data landing page. It is recommended to insert the citations in the reference section of the paper to facilitate citation tracking and count. @@ -738,7 +738,7 @@ When using social media or any presentation platform to communicate research, it --- -### [Data Preservation](#datapreservation) { #datapreservation } +### Data Preservation { #datapreservation } In the Data Depot Repository (DDR) data preservation is achieved through the combined efforts of the NH community that submits data and metadata following policies and best practices, and the DDR's administrative responsibilities and technical capabilities. The following data preservation best practices ensure preservation of the data from the moment in which researchers plan their data projects and for the long term after the data is published. diff --git a/user-guide/docs/curating/faq.md b/user-guide/docs/curating/faq.md index 2dc1c39..ba6c454 100644 --- a/user-guide/docs/curating/faq.md +++ b/user-guide/docs/curating/faq.md @@ -1,4 +1,4 @@ -### [Selecting Files & Data](#selecting) { #selecting } +### Selecting Files & Data { #selecting } **Q: What are the best file formats for data publications?** **A**: For long-term preservation purposes it is best to publish data in interoperable and open formats. For example, instead of Excel spreadsheet files -which are proprietary- it is best to convert them to CSV for publication. And, instead of Matlab files -also proprietary- it is best to publish data as simple txt (ascii) so it can be used by many different software. However, be aware that conversion may distort the data structure, so retain an original copy of any structured data (e.g. Matlab, Excel files) before attempting conversions and then check between the two for fidelity. In addition, you may publish both the proprietary and the open format, and/or consult the Data Curation Primers to find out how to better curate research data. @@ -24,7 +24,7 @@ **Q: What should I consider before publishing Jupyter Notebooks?** **A**: Please refer to our Jupyter User Guide document to find information on how to publish a Jupyter Notebook. -### [Organizing & Describing Your Dataset](#organizing) { #organizing } +### Organizing & Describing Your Dataset { #organizing } **Q: How should I organize the data files to be published in a project?** **A**: For each type of project publication, the best way to organize your data is to map them to the organizational schema provided by the data models available for each research type (simulation, experimental, hybrid simulation and field research). These models were designed by experts and represent the main data and documentation components required for others to understand and reuse your dataset. @@ -47,7 +47,7 @@ **Q: I have another published work that is related to the project I am now planning to publish. How can I relate them?** **A**: On the project landing page under Edit Project there is a "Related Work" field where you have the option to include one or more associated projects and publications to your current project. Here, you can provide the title as well as a link to that project or publication. This link can be a DOI or a URL for any content found inside or outside of DesignSafe. For DOIs, please make sure you are adding the entire DOI address starting with "http" to correctly link the webpage to the related project. -### [Publishing](#publishing) { #publishing } +### Publishing { #publishing } **Q: Which license is appropriate for my publication?** **A**: Licenses indicate the conditions in which you, as a data creator, want the data to be used by others. Due to the variety of resources published in DesignSafe, we provide four different types of open licenses. These cover datasets, software, materials with intellectual property rights, and the different ways in which you want your work to be attributed. You can find relevant information under licensing here: . @@ -72,7 +72,7 @@
    Rathje, E., Dawson, C. Padgett, J.E., Pinelli, J.-P., Stanzione, D., Adair, A., Arduino, P., Brandenberg, S.J., Cockerill, T., Dey, C., Esteva, M., Haan, Jr., F.L., Hanlon, M., Kareem, A., Lowes, L., Mock, S., and Mosqueda, G. 2017. "DesignSafe: A New Cyberinfrastructure for Natural Hazards Engineering," ASCE Natural Hazards Review, doi:10.1061/(ASCE)NH.1527-6996.0000246.
    -### [Data Reuse](#datareuse) { #datareuse } +### Data Reuse { #datareuse } **Q: How can I contribute to the reuse of data?** **A**: Datasets take a lot of work to produce; they are important research products. By creating a complete, clean, and clearly described data publication you are already inviting others to use and cite your data. Always cite your datasets and those of others that you have reused in the reference section of your papers using the citation language and DOI provided on DesignSafe, and encourage your peers and students to do the same. If you use social media or a presentation platform to talk about your research, always include the proper citation and DOI on your materials (ppts, abstracts, emails, etc.). Note that a researcher does not actually need to reuse a dataset to cite it, but rather may cite a dataset to reference something of note in the dataset (e.g., how it was collected, its uniqueness, etc.). This is similar to the process of citing other papers. diff --git a/user-guide/docs/curating/guides.md b/user-guide/docs/curating/guides.md index fdbb981..d3e6d2b 100644 --- a/user-guide/docs/curating/guides.md +++ b/user-guide/docs/curating/guides.md @@ -6,7 +6,7 @@ Below are step-by-step guides on how to create projects in the Data Depot, and c ### [Experimental](#experimental) -#### [1. Add a Project](#step1) { #experimental-step1 } +#### 1. Add a Project { #experimental-step1 } You can start a project at the very beginning of its lifespan, upload and curate data incrementally, then publish sets of data at your convenience. @@ -24,7 +24,7 @@ You can edit all these fields later if you make any mistakes. Once finished, click + Add Project and you will be taken to your new project in the My Projects tab. -#### [2. Add an Experiment](#step2) { #step2 } +#### 2. Add an Experiment { #step2 } To begin curation and add an experiment, click on the Curation Directory and select Experimental as your Project Type. @@ -44,7 +44,7 @@ Click + Add Experiment when you are done and i You can edit an experiment from the inventory. -#### [3. Add Categories](#step3) { #step3 } +#### 3. Add Categories { #step3 } Click Add Categories to begin. @@ -64,7 +64,7 @@ Categories group files together based on a shared purpose in an experiment. Be s After filling out the fields, click + Add Category and it will appear below in your inventory. If you make any mistakes, expand the category and click Edit. -#### [4. Relate Data](#step4) { #step4 } +#### 4. Relate Data { #step4 } Click Relate Data to begin. @@ -74,7 +74,7 @@ Relating Data allows you to relate categories to each other and to an experiment When published, this diagram will help others understand the structure of your experiment at a glance. -#### [5. Assign Categories to Files](#step5) { #step5 } +#### 5. Assign Categories to Files { #step5 } ![](./imgs/guide-experimental-5a.png) @@ -88,7 +88,7 @@ Click Remove if you make any mistakes. ![](./imgs/guide-experimental-5b.png) -#### [6. Tag Files](#step6) { #step6 } +#### 6. Tag Files { #step6 } ![](./imgs/guide-experimental-6.png) @@ -100,7 +100,7 @@ These tags are optional, but recommended. If you do not see a file tag that fits, you can select Other and write in your own. -#### [7. Publication Preview](#step7) { #step7 } +#### 7. Publication Preview { #step7 } ![](./imgs/guide-experimental-7a.png) @@ -110,7 +110,7 @@ All of the curation work is done in the Curation Directory, while the Publicatio Look through the Publication Preview early and often to catch any mistakes. If you are working collaboratively with others, this is a good way to proofread changes they make. -#### [8. Prepare to Publish](#step8) { #step8 } +#### 8. Prepare to Publish { #step8 } When you are satisfied with how your work is curated and wish to publish it, select Prepare to Publish in the Publication Preview. @@ -131,11 +131,11 @@ Finally, click Request DOI & Publish and agree to the agree --- -### [Simulation](#simulation) { #simulation } +### Simulation { #simulation } Read the [Simulation Data Best Practices Guide](../../curating#bestpractices-simulationdata) prior to initiating your project. -#### [1. Add a Project](#simulation-step1) { #simulation-step1 } +#### 1. Add a Project { #simulation-step1 } You can start a project at the very beginning of its lifespan, upload and curate data incrementally, then publish sets of data at your convenience. @@ -153,7 +153,7 @@ You can edit all these fields later if you make any mistakes. Once finished, click + Add Project and you will be taken to your new project in the My Projects tab. -#### [2. Add a Simulation](#simulation-step2) { #simulation-step2 } +#### 2. Add a Simulation { #simulation-step2 } To begin curation and add a simulation, click on the Curation Directory and select Simulation as your Project Type. @@ -173,7 +173,7 @@ Click + Add Simulation when you are done and i You can edit a simulation from the inventory. -#### [3. Add Categories](#simulation-step3) { #simulation-step3 } +#### 3. Add Categories { #simulation-step3 } Click Add Categories to begin. @@ -193,7 +193,7 @@ Categories group files together based on a shared purpose in a simulation. Be su After filling out the fields, click + Add Category and it will appear below in your inventory. If you make any mistakes, expand the category and click Edit. -#### [4. Relate Data](#simulation-step4) { #simulation-step4 } +#### 4. Relate Data { #simulation-step4 } Click Relate Data to begin. @@ -204,7 +204,7 @@ Relating Data allows you to relate categories to each other and to an simulation When published, this diagram will help others understand the structure of your simulation at a glance. -#### [5. Assign Categories to Files](#simulation-step5) { #simulation-step5 } +#### 5. Assign Categories to Files { #simulation-step5 } As you create categories, they will appear in a dropdown by each file. This allows you to group files in each category. Click Save to confirm the category. @@ -216,7 +216,7 @@ A file can belong to one or more categories. Click Remove if you make any mistakes. -#### [6. Tag Files](#simulation-step6) { #simulation-step6 } +#### 6. Tag Files { #simulation-step6 } After putting files in categories, dropdowns will appear to allow you to tag specific files. @@ -228,7 +228,7 @@ These tags are optional, but recommended. If you do not see a file tag that fits, you can select Other and write in your own. -#### [7. Publication Preview](#simulation-step7) { #simulation-step7 } +#### 7. Publication Preview { #simulation-step7 } All of the curation work is done in the Curation Directory, while the Publication Preview lets you examine the layout of your publication to give you a peace of mind about how your work will appear to other researchers once published. @@ -238,7 +238,7 @@ Look through the Publication Preview early and often to catch any mistakes. If y ![](./imgs/guide-simulation-7b.png) -#### [8. Prepare to Publish](#simulation-step8) { #simulation-step8 } +#### 8. Prepare to Publish { #simulation-step8 } ![](./imgs/guide-simulation-8a.png) @@ -261,17 +261,17 @@ Finally, click Request DOI & Publish and agree to the agree --- -### [Hybrid Simulation](#hybrid) { #hybrid } +### Hybrid Simulation { #hybrid } Hybrid Simulation User Guide in progress. --- -### [Field Research](#fieldresearch) { #fieldresearch } +### Field Research { #fieldresearch } Field Research User Guide is in progress. -#### [1. Add a Project](#fieldresearch-step1) { #fieldresearch-step1 } +#### 1. Add a Project { #fieldresearch-step1 } ![](./imgs/guide-allguides-1a.jpg) @@ -289,7 +289,7 @@ You can edit all these fields later. Once finished, click + Add Project and you will be taken to your new project in the My Projects tab. -#### [2. Add a Mission](#fieldresearch-step2) { #fieldresearch-step2 } +#### 2. Add a Mission { #fieldresearch-step2 } To begin curation, click on the Curation Directory and select Field Research as your Project Type. @@ -317,7 +317,7 @@ You can edit a mission from the mission inventory. -#### [3. Add Collections](#fieldresearch-step3) { #fieldresearch-step3 } +#### 3. Add Collections { #fieldresearch-step3 } Click Add Collections to begin. @@ -342,7 +342,7 @@ After filling out the fields, click + Add Collection and it wil ![](./imgs/guide-fieldresearch-3.png) -#### [4. Relate Data](#fieldresearch-step4) { #fieldresearch-step4 } +#### 4. Relate Data { #fieldresearch-step4 } Click Relate Data to begin. @@ -354,7 +354,7 @@ When published, this diagram will help others understand the structure of your F ![](./imgs/guide-fieldresearch-4.png) -#### [5. Assign Collections to Files](#fieldresearch-step5) { #fieldresearch-step5 } +#### 5. Assign Collections to Files { #fieldresearch-step5 } As you create collections, they will appear in a dropdown next to each file. This allows you select collections for any file in your project and group them under each collection. Click Save to confirm the collection. @@ -371,7 +371,7 @@ Click Remove if you make any mistakes. ![](./imgs/guide-fieldresearch-5b.png) -#### [6. Tag Files](#fieldresearch-step6) { #fieldresearch-step6 } +#### 6. Tag Files { #fieldresearch-step6 } After putting files in collections, dropdowns will appear to allow you to tag/describe unique files. @@ -385,7 +385,7 @@ If you do not see a file tag that fits, you can select Other an ![](./imgs/guide-fieldresearch-6.png) -#### [7. Publication Preview](#fieldresearch-step7) { #fieldresearch-step7 } +#### 7. Publication Preview { #fieldresearch-step7 } All of the curation work is done in the Curation Directory. The Publication Preview lets you examine the layout of your publication so you can visualize/verify how your work will appear once published. @@ -398,7 +398,7 @@ Look through the Publication Preview early and often to make changes or catch mi ![](./imgs/guide-fieldresearch-7b.png) -#### [8. Prepare to Publish](#fieldresearch-step8) { #fieldresearch-step8 } +#### 8. Prepare to Publish { #fieldresearch-step8 } When you are satisfied with how your work is curated and wish to publish it, select Prepare to Publish in the Publication Preview. @@ -423,7 +423,7 @@ Finally, click Request DOI & Publish and agree to the agree ### [Other](#other) -#### [1. Add a Project](#other-step1) { #other-step1 } +#### 1. Add a Project { #other-step1 } To add a new project, click + Add, then select New Project. @@ -439,7 +439,7 @@ You can edit all these fields later if you make any mistakes. Once finished, click + Add Project and you will be taken to your new project in the My Projects tab. -#### [2. Begin Curation](#other-step2) { #other-step2 } +#### 2. Begin Curation { #other-step2 } To begin curating and tagging your files, click on the Curation Directory and select Other as your Project Type. @@ -449,7 +449,7 @@ Fill out additional required fields in the Edit Project window, ![](./imgs/guide-other-step2b.png) -#### [3. Tag Files](#other-step3) { #other-step3 } +#### 3. Tag Files { #other-step3 } Dropdowns will appear by each file to allow you to tag specific files. @@ -459,7 +459,7 @@ These tags are optional, but recommended. The help other users understand your d If you do not see a file tag that fits, you can select Other and write in your own. -#### [4. Prepare to Publish](#other-step4) { #other-step4 } +#### 4. Prepare to Publish { #other-step4 } When you are satisfied with your work and wish to publish it and recieve a DOI, click Prepare to Publish in the Publication Preview. diff --git a/user-guide/docs/curating/metrics.md b/user-guide/docs/curating/metrics.md index edc124b..df25a12 100644 --- a/user-guide/docs/curating/metrics.md +++ b/user-guide/docs/curating/metrics.md @@ -1,4 +1,4 @@ -### [Data Metrics](#data) { #data } +### Data Metrics { #data } Data metrics are research impact indicators complementary to other forms of evaluation such as number of paper citations, allowing researchers to assess the repercussions and influence of their work. @@ -24,7 +24,7 @@ Data Metrics is a work in progress and we add measurements on an ongoing basis. Below are descriptions of each type of metric and what is counted at the project and at the data publication levels. -### [Project Metrics](#project) { #project } +### Project Metrics { #project } File Preview: Examining data in any data publication within a project such as clicking on a file name brings up a modal window that allows previewing the files. However, not all data types can be previewed. Among those that can are: text, spreadsheets, graphics and code files. (example extensions: .txt, .doc, .docx, .csv, .xlsx, .pdf, .jpg, .m, .ipynb). Those that can't include binary executables, MATLAB containers, compressed files, and video (eg. .bin, .mat, .zip, .tar, mp4, .mov). Only those files that can be previewed are counted. Users will get a count of all the files that have been previewed in the entire project. @@ -36,7 +36,7 @@ Below are descriptions of each type of metric and what is counted at the project Project Downloads: Total downloads of a compressed entire project and its metadata to a user's machine. -### [Data Publication Metrics](#publication) { #publication } +### Data Publication Metrics { #publication } File Preview: Examining data from an individual data publication such as clicking on a file name brings up a modal window that allows previewing files. Those file previews are counted. However, not all document types can be previewed. Among those that can are: text, spreadsheets, graphics and code files. (example extensions: .txt, .doc, .docx, .csv, .xlsx, .pdf, .jpg, .m, .ipynb). Those that can't include binary executables, MATLAB containers, compressed files, and video (eg. .bin, .mat, .zip, .tar, mp4, .mov). Only those files that can be previewed are counted. Users will get a count of all the files that have been previewed in the data publication. diff --git a/user-guide/docs/curating/officehours.md b/user-guide/docs/curating/officehours.md index e4c768d..6428c4c 100644 --- a/user-guide/docs/curating/officehours.md +++ b/user-guide/docs/curating/officehours.md @@ -3,7 +3,7 @@ Virtual Office hours is the newest way to connect with experts in the NHERI network. Using the links below, schedule a time to meet with facility personnel to answer your questions. If you don’t see the facility you are interested in, email us at eco-feedback@designsafe-ci.org. --- -#### [DesignSafe Data Depot/Curation](#depot) { #depot } +#### DesignSafe Data Depot/Curation { #depot } ![](./imgs/maria-esteva.jpg){: width="40%" } @@ -14,7 +14,7 @@ DesignSafe’s Data Curator, Dr. Maria Esteva, holds virtual office hours every Passcode: 595633 --- -#### [DesignSafe Workspace](#workspace) { #workspace } +#### DesignSafe Workspace { #workspace } ![](./imgs/wenyang-zhang.png){: width="40%" } @@ -25,7 +25,7 @@ Dr. Wenyang Zhang holds virtual office hours every otherPasscode: 776504 --- -#### [Florida International University, Wall of Wind](#wallofwind) { #wallofwind } +#### Florida International University, Wall of Wind { #wallofwind } ![](./imgs/wow-team.jpg__800x534_q85_subsampling-2.jpg){: width="40%" } @@ -35,7 +35,7 @@ The Wall of Wind Research Scientists are available to meet with you every Wed Sign Up for Office Hours

    --- -#### [Lehigh University, Real-Time Multi-Directional Experimental Facility](#lehigh) { #lehigh } +#### Lehigh University, Real-Time Multi-Directional Experimental Facility { #lehigh } ![](./imgs/liang-cao.jpg){: width="40%" } Lehigh’s Dr. Liang Cao is available to answer questions and discuss research proposals every other Wednesday from 2:00-3:00 pm EST. @@ -43,7 +43,7 @@ Lehigh’s Dr. Liang Cao is available to answer questions and discuss research p Sign Up for Office Hours

    --- -#### [University of California, San Diego, Large High Performance Outdoor Shake Table](#ucsd) { #ucsd } +#### University of California, San Diego, Large High Performance Outdoor Shake Table { #ucsd } ![](./imgs/koorosh_lotfizadeh.jpg){: width="40%" } @@ -53,7 +53,7 @@ Contact Dr. Lotfizadeh: klotf Sign Up for Office Hours --- -#### [Oregon State University, O.H. Hinsdale Wave Research Laboratory](#hinsdale) { #hinsdale } +#### Oregon State University, O.H. Hinsdale Wave Research Laboratory { #hinsdale } ![](./imgs/pedro-lomonaco.jpg){: width="40%" } @@ -62,7 +62,7 @@ Dr. Pedro Lomonaco, Co-Pi and Director of NHERI O.H. Hinsdale Wave Research Labo Contact Dr. Lomonaco: pedro.lomonaco@oregonstate.edu --- -### [How to Schedule Virtual Office Hours](#schedule) { #schedule } +### How to Schedule Virtual Office Hours { #schedule } To schedule a meeting with NHERI facility faculty, click the facility Eventbrite link, schedule your date and time, chose individual or group meeting, and then register. A confirmation email will include the Zoom link for your scheduled day and time. You will also receive reminder emails about your scheduled meeting. Also see our video tutorial. diff --git a/user-guide/docs/curating/policies.md b/user-guide/docs/curating/policies.md index 4e6d202..969a150 100644 --- a/user-guide/docs/curating/policies.md +++ b/user-guide/docs/curating/policies.md @@ -1,14 +1,14 @@ ### DesignSafe Data Depot Repository Mission and History -#### [Mission](#mission) { #mission } +#### Mission { #mission } The Data Depot Repository (DDR) is the platform for curation and publication of datasets generated in the course of natural hazards research. The DDR is an open access data repository that enables data producers to safely store, share, organize, and describe research data, towards permanent publication, distribution and impact evaluation. The DDR allows data consumers to discover, search for, access, and reuse published data in an effort to accelerate research discovery. The DDR is one component of the DesignSafe cyberinfrastructure, which represents a comprehensive research environment that provides cloud-based tools to manage, analyze, understand, and publish critical data for research to understand the impacts of natural hazards. DesignSafe is part of the NSF-supported Natural Hazards Engineering Research Infrastructure (NHERI), and aligns with its mission to provide the natural hazards research community with open access, shared-use scholarship, education, and community resources aimed at supporting civil infrastructure prior to, during, and following natural disasters. However, DesignSafe also supports the broader natural hazards research community that extends beyond the NHERI network. -#### [History](#history) { #history } +#### History { #history } The DDR has been in operation since 2016 and is currently supported by NSF through 2025. The DDR preserves natural hazards research data published since its inception in 2016, and also provides access to legacy data dating from about 2005. These legacy data were generated as part of the NSF-supported Network for Earthquake Engineering Simulation (NEES), a predecessor to NHERI. Legacy data and metadata belonging to NEES were transferred to the DDR for continuous preservation and access. View the published NEES data here. -#### [Community](#community) { #community } +#### Community { #community } The DDR serves the broader natural hazards research community both as data producers and consumers. This research community includes, but is not limited to, the facilities that make up NHERI network - for which we are the designated data repository. We work with each component of the NHERI network to meet the requirements and commitments of their distinct research focus and functions. As the only repository for open natural hazards research data, we welcome data produced in other national and international facilities and organizations. At large, the repository serves a broad national and international audience of natural hazard researchers, students, practitioners, policy makers, as well as the general public. @@ -18,7 +18,7 @@ The DDR serves the broader natural hazards research community both as data produ Policies for the DDR are driven by the Natural Hazards (NH) scientific community and informed by best practices in library and information sciences. The DDR operates under the leadership of the DesignSafe Management Team (DSMT), who establishes and updates policies, evaluates and recommends best practices, oversees its technical development, and prioritizes activities. The broad organizational structure under which the DDR operates is here. -#### [Repository Team](#repositoryteam) { #repositoryteam } +#### Repository Team { #repositoryteam } An interdisciplinary repository team (RT) carries out ongoing design, development and day-to-day operations, gathering requirements and discussing solutions through formal monthly and bi-weekly meetings with the NHERI community and maintaining regular communications with members of the network, including monthly meetings with the Experimental Facilities, RAPID, and CONVERGE staff. Based on these fluid communications, the RT designs functionalities, researches and develops best-practices, and implements agreed-upon solutions. The figure below shows the current formation of the RT, including their expertise. @@ -26,13 +26,13 @@ An interdisciplinary repository team (RT) carries out ongoing design, developmen Formal mechanisms are in place for external evaluators to gather feedback and conduct structured assessments, in the form of usability studies and yearly user surveys, to ensure that the repository is meeting the community’s expectations and needs. To track development the DDR curator meets every other week with the DesignSafe PI and with the head of the development team. All DDR activities are reported to the National Science Foundation on a quarterly and annual basis in terms of quantitative and qualitative progress. -#### [Community Norms for DDR](#norms) { #norms } +#### Community Norms for DDR { #norms } Within the broader conditions of use for DesignSafe we have established a set of Community Norms specific for DDR which have to be agreed upon at the point of registering an account on the platform. These norms, highlighting our existing policies, are the following: Users who either publish and use data in DDR must abide by both the TACC Acceptable Use Policy and the DesignSafe Terms of Use. -##### [For users curating and publishing data in DDR:](#norms-curators) { #norms-curators } +##### For users curating and publishing data in DDR: { #norms-curators } * Users understand that their data submissions to the DDR should follow our Data Policies and our Curation and Publication Best Practices to the best of their ability. * Users agree to use DDR to publish only open access data, which they must document in a manner that does not hinder the ability of other users to reuse or reproduce it. @@ -42,7 +42,7 @@ Users who either publish and use data in DDR must abide by both the tombstone explaining why the data is not available. Curators review both in process and published data on a monthly basis. In both cases we will work with the authors to find an adequate repository for their dataset. -#### [Data Size](#size) { #size } +#### Data Size { #size } Researchers in the natural hazards community generate very large datasets during large-scale experiments, simulations, and field research projects. At the moment the DDR does not pose limitations on the amount of data to be published, but we do recommend to be selective and publish data that is relevant to a project completeness and is adequately described for reuse. Our Data Curation Best Practices include recommendations to achieve a quality data publication. We are observing trends in relation to sizes and subsequent data reuse of these products, which will inform if and how we will implement data size publication limit policies. -#### [Data Formats](#formats) { #formats } +#### Data Formats { #formats } We do not pose file format restrictions. The natural hazards research community utilizes diverse research methods to generate and record data in both open and proprietary formats, and there is continual update of equipment used in the field. We do encourage our users to convert to open formats when possible. The DDR follows the Library of Congress Recommended Format Statement and has guidance in place to convert proprietary formats to open formats for long term preservation; see our Accepted and Recommended Data Formats for more information. However, conversion can present challenges; Matlab, for example, allows saving complex data structures, yet not all of the files stored can be converted to a csv or a text file without losing some clarity and simplicity for handling and reusing the data. In addition, some proprietary formats such as jpeg, and excel have been considered standards for research and teaching for the last two decades. In attention to these reasons, we allow users to publish the data in both proprietary and open formats. Through our Fedora repository we keep file format identification information of all the datasets stored in DDR. ### Data Curation @@ -70,17 +70,17 @@ Data curation involves the organization, description, representation, permanent Our goal is to enable researchers to curate their data from the beginning of a research project and turn it into publications through interactive pipelines and consultation with data curators. The DDR has and continues to invest efforts in developing and testing curation and publication pipelines based on data models designed with input from the NHERI community. -#### [Data Management Plan](#management) { #management } +#### Data Management Plan { #management } For natural hazards researchers submitting proposals to the NSF using any of the NHERI network facilities/resources, or alternative facilities/resources, we developed Data Management guidelines that explain how to use the DDR for data curation and publication. See Data Management Plan at: https://www.designsafe-ci.org/rw/user-guides/ and https://converge.colorado.edu/data/data-management -#### [Data Models](#models) { #models } +#### Data Models { #models } To facilitate data curation of the diverse and large datasets generated in the fields associated with natural hazards, we worked with experts in natural hazards research to develop five data models that encompass the following types of datasets: experimental, simulation, field research, hybrid simulation, and other data products (See: 10.3390/publications7030051; 10.2218/ijdc.v13i1.661) as well as lists of specialized vocabulary. Based on the Core Scientific Metadata Model, these data models were designed considering the community's research practices and workflows, the need for documenting these processes (provenance), and using terms common to the field. The models highlight the structure and components of natural hazards research projects across time, tests, geographical locations, provenance, and instrumentation. Researchers in our community have presented on the design, implementation and use of these models broadly. In the DDR web interface the data models are implemented as interactive functions with instructions that guide the researchers through the curation and publication tasks. As researchers move through the tion pipelines, the interactive features reinforce data and metadata completeness and thus the quality of the publication. The process will not move forward if requirements for metadata are not in place (See Metadata in Best Practices), or if key files are missing. -#### [Metadata](#metadata) { #metadata } +#### Metadata { #metadata } Up to date, there is no standard metadata schema to describe natural hazards data. In DDR we follow a combination of standard metadata schemas and expert-contributed vocabularies to help users describe and find data. @@ -92,7 +92,7 @@ To further describe datasets, the curation interface offers the possibility to a For purposes of metadata exchange and interoperability, the elements and tags in the data models are mapped to widely-used, standardized metadata schemas. These are: Dublin Core for description of the dataset project, DDI (Data Documentation Initiative) for social science data, DataCite for DOI assignment and citation, and PROV-O to record the structure of the dataset. Metadata mapping is substantiated as the data is ingested into Fedora. Users can download the standardized metadata in the publications landing page. -#### [Metadata and Data Quality](#dataquality) { #dataquality } +#### Metadata and Data Quality { #dataquality } The diversity and quantity of research methods, instruments, and data formats in natural hazards research is vast and highly specialized. For this reason, we conceive of data quality as a collaboration between the researchers and the DDR. In consultation with the larger NHERI network we are continuously observing and defining best practices that emerge from our community's understanding and standards. @@ -108,7 +108,7 @@ We also support citation best practices for datasets reused in our publications. Data publications review: Once a month, data curators meet to review new publications. These reviews show us how the community is using and understanding the models, and allows verifying the overall quality of the data publications. When we identify curation problems (e.g. insufficient or unclear descriptions, file or category misplacement, etc.) that could not be automatically detected, we contact the researchers and work on solving these issues. Based on the feedback, users have the possibility to amend/improve their descriptions and to version their datasets (See amends and version control). -#### [Curation and Publication Assistance ](#assistance) { #assistance } +#### Curation and Publication Assistance { #assistance } We believe that researchers are best prepared to tell the story of their projects through their data publications; our role is to enable them to communicate their research to the public by providing flexible and easy to use curation resources and guidance that abide by publication best practices. To support researchers organizing, categorizing and describing their data, we provide interactive pipelines with onboarding instructions, different modes of training and documentation, and one-on-one help. @@ -122,7 +122,7 @@ Webinars by Researchers: Various researchers in our community contribute to our ### Data Publication and Usage -#### [Protected Data](#protecteddata) { #protecteddata } +#### Protected Data { #protecteddata } Protected data are information subject to regulation under relevant privacy and data protection laws, such as HIPAA, FERPA and FISMA, as well as human subjects data containing Personally Identifiable Information (PII) and data involving vulnerable populations and or containing sensitive information. @@ -136,23 +136,23 @@ It is the user’s responsibility to adhere to these policies and the procedures For any data not subject to IRB oversight but may still contain PII, such as Google Earth images containing images of people not studied in the scope of the research project, we recommend blocking out or blurring any information that could be considered PII before publishing the data in the DDR. We still invite any researchers that are interested in seeing the raw data to contact the PI of the research project to try and attain that. See our Protected Data Best Practices for information on how to manage protected data in DDR. -#### [Subsequent Publishing](#publishing) { #publishing } +#### Subsequent Publishing { #publishing } Attending to the needs expressed by the community, we enable the possibility to publish data and other products subsequently within a project, each with a DOI. This arises from the longitudinal and/or tiered structure of some research projects such as experiments and field research missions which happen at different time periods, may involve multiple distinct teams, have the need to publish different types of materials or to release information promptly after a natural hazards event and later publish related products. Subsequent publishing is enabled in My Project interface where users and teams manage and curate their active data throughout their projects' lifecycle. -#### [Timely Data Publication ](#timely) { #timely } +#### Timely Data Publication { #timely } Although no firm deadline requirements are specified for data publishing, as an NSF-funded platform we expect researchers to publish in a timely manner, so we provide recommended timelines for publishing different types of research data in our Timely Data Publication Best Practices. -#### [Peer Review](#peerreview) { #peerreview } +#### Peer Review { #peerreview } Users that need to submit their data for revision prior to publishing and assigning a DOI have the opportunity to do so by: a) adding reviewers to their My Project, when there is no need for annonymous review, or b) by contacting the DesignSafe data curator through a Help ticket to obtain a Public Accessibility Data Delay (See below). Note that the data must be fully curated prior to requesting a Public Accessibility Delay. -#### [Public Accessibility Delay ](#accessiblity) { #accessiblity } +#### Public Accessibility Delay { #accessiblity } Many researchers request a DOI for their data before it is made publicly available to include in papers submitted to journals for review. In order to assign a DOI in the DDR, the data has to be curated and ready to be published. Once the DOI is in place, we provide services to researchers with such commitments to delay the public accessibility of their data publication in the DDR, i.e. to make the user’s data publication, via their assigned DOI, not web indexable through DataCote and or not publicly available in DDR's data browser until the corresponding paper is published in a journal, or for up to one year after the data is deposited. The logic behind this policy is that once a DOI has been assigned, it will inevitably be published, so this delay can be used to provide reviewers access to a data publication before it is broadly distributed. Note that data should be fully curated, and that while not broadly it will be eventually indexed by search engines. Users that need to amend/correct their publications will be able to do so via version control. See our Data Delay Best Practices for more information on obtaining a public accessibility delay. -#### [Data Licenses](#licenses) { #licenses } +#### Data Licenses { #licenses } DDR provides users with 5 licensing options to accommodate the variety of research outputs generated and how researchers in this community want to be attributed. The following licenses were selected after discussions within our community. In general, DDR users are keen about sharing their data openly but expect attribution. In addition to data, our community issues reports, survey instruments, presentations, learning materials, and code. The licenses are: Creative Commons Attribution (CC-BY 4.0), Creative Commons Public Domain Dedication (CC-0 1.0), Open Data Commons Attribution (ODC-BY 1.0), Open Data Commons Public Domain Dedication (ODC-PPDL 1.0), and GNU General Public License (GNU-GPL 3). During the publication process users have the option of selecting one license per publication with a DOI. More specifications of these license options and the works they can be applied to can be found in Licensing Best Practices. @@ -160,7 +160,7 @@ DDR also requires that users reusing data from others in their projects do so in The expectations of DDR and the responsibilities of users in relation to the application and compliance with licenses are included in the DesignSafe Terms of Use, the Data Usage Agreement, and the Data Publication Agreement. As clearly stated in those documents, in the event that we note or are notified that the licencing policies and best practices are not followed, we will notify the user of the infringement and may cancel their DesignSafe account. -#### [Data Citation](#citation) { #citation } +#### Data Citation { #citation } DDR abides by and promotes the Joint Declaration of Data Citation Principles amongst its users. @@ -172,7 +172,7 @@ The expectations of DDR and the responsibilities of users in relation to the app However, given that it is not feasible to know with certainty if users comply with data citation, our approach is to educate our community by reinforcing citation in a positive way. For this we implement outreach strategies to stimulate data citation. Through diverse documentation, FAQs webinars, and via emails, we regularly train our users on data citation best practices. And, by tracking and publishing information about the impact and science contributions of the works they publish citing the data that they use, we demonstrate the value of data reuse and further stimulate publishing and citing data. -#### [Data Publication Agreement](#agreement) { #agreement } +#### Data Publication Agreement { #agreement } This agreement is read and has to be accepted by the user prior to publishing a dataset. @@ -207,7 +207,7 @@ If applicable, I warrant that I am following the IRB agreements in place for my I understand that the DDR does not approve data publications before they are posted; therefore, I am solely responsible for the submission, publication, and all possible confidentiality/privacy issues that may arise from the publication. -#### [Data Usage Agreement](#datausage) { #datausage } +#### Data Usage Agreement { #datausage } Users who access, preview, download or reuse data and metadata from the DesignSafe Data Depot Repository (DDR) agree to the following policies. If these policies are not followed, we will notify the user of the infringement and may cancel their DesignSafe account. @@ -230,7 +230,7 @@ Users who access, preview, download or reuse data and metadata from the DesignSa
  • Our system logs file actions completed by registered users in the DDR including previewing, downloading or copying published data to My Data or My Projects. We only use this information in aggregate for metrics purposes and do not link it to the user’s identity.
    • -#### [Amends and Version Control](#versioncontrol) { #versioncontrol } +#### Amends and Version Control { #versioncontrol } Users can amend and version their data publications. Since the DDR came online, we have helped users correct and or improve the metadata applied to their datasets after publication. Most requests involve improving the text of the descriptions, changing the order of the authors, and adding references of papers publised using the data in the project; users also required the possibility to version their datasets. Our amends and version control policy derives from meeting our users needs. @@ -258,11 +258,11 @@ The Fedora repository manages all amends and versions so there is a record of al More information about the reasons for amends and versioning are in Publication Best Practices. -#### [Leave Data Feedback](#feedback) { #feedback } +#### Leave Data Feedback { #feedback } Users can click a “Leave Feedback” button on the projects’ landing pages to provide comments on any publication. This feedback is forwarded to the curation team for any needed actions, including contacting the authors. In addition, it is possible for users to message the authors directly as their contact information is available via the authors field in the publication landing pages. We encourage users to provide constructive feedback and suggest themes they may want to discuss about the publication in our Leave Data Feedback Best Practices -#### [Data Impact](#impact) { #impact } +#### Data Impact { #impact } We understand data impact as a strategy that includes complementary efforts at the crossroads of data discoverability, usage metrics, and scholarly communications. @@ -317,7 +317,7 @@ Data sustainability is a continuous effort that DDR accomplishes along with the
  • Identifying Data Guideline Needs for Community and Regional Resilience Modelling Workshop: http://resilience.colostate.edu/files/DATA/Agenda_DATA_20210316.pdf
    • -#### [Continuity of Access](#continuity) { #continuity } +#### Continuity of Access { #continuity } As part of the requirements of the current award we have a contingency pan in place to transfer all the DDR data, metadata and corresponding DOIs to a new awardee (should one be selected) without interruption of services and access to data. Fedora has export capabilities for transfer of data and metadata to another repository in a complete and validated fashion. The portability plan is confirmed and updated in the Operations Project Execution Plan that we present anually to the NSF. diff --git a/user-guide/docs/managingdata/datadepot.md b/user-guide/docs/managingdata/datadepot.md index 00855be..db4ff47 100644 --- a/user-guide/docs/managingdata/datadepot.md +++ b/user-guide/docs/managingdata/datadepot.md @@ -12,7 +12,7 @@ The Command-Line Data Transfer Guide for instructions. -#### [Recommended Normal Data Transfer Methods](#recommended-normaldatatransfer) { #recommended-normaldatatransfer } +#### Recommended Normal Data Transfer Methods { #recommended-normaldatatransfer } We define a "normal" data transfer as < 2GB or < 25 files or < 2 folders @@ -50,30 +50,30 @@ We define a "normal" data transfer as < 2GB or < 25 files or < 2 folde --- -### [Globus Data Transfer Guide](#globus) { #globus } +### Globus Data Transfer Guide { #globus } Globus supplies high speed, reliable, and asynchronous transfers to DesignSafe. Once setup, Globus will allow you to not only transfer files to and from DesignSafe, but also other cyberinfrastructure resources at TACC and other research centers. While the setup of Globus can take slightly longer than the other transfer methods (see Data Transfer Guide), it only needs to be performed once, making later transfers as fast (if not faster due to Globus' superior speed) than the other methods. For these reasons, Globus is the recommend approach for moving large quantities of data to and from DesignSafe. The following provides detailed instructions for setting up Globus access to DesignSafe. -#### [1. Log in to CILogon.org](#globus-step1) { #globus-step1 } +#### 1. Log in to CILogon.org { #globus-step1 } Log in to the CILogon service (https://CILogon.org). If your institution is already a member of CILogon you can search for your institution and use your institutional credentials to log in. Otherwise, you can search for ACCESS CI (XSEDE) and proceed to create an ACCESS account. -#### [2. Find the ePPN associated with your CILogon/Globus access](#globus-step2) { #globus-step2 } +#### 2. Find the ePPN associated with your CILogon/Globus access { #globus-step2 } Globus requires a unique identifier, called a eduPersonPrincipalName (ePPN), for each user. Find your ePPN associated with your Globus access by going to https://cilogon.org/ and logging in. You will find your ePPN under User Attributes -#### [3. Associate your ePPN with your DesignSafe/TACC Account](#globus-step3) { #globus-step3 } +#### 3. Associate your ePPN with your DesignSafe/TACC Account { #globus-step3 } Login to your TACC user profile here: https://accounts.tacc.utexas.edu. Select ePPN on the left menu and then enter your ePPN in the field at the top of the page and save. Allow 30 minutes for the ePPN to propagate through TACC's systems. -#### [4. Activate Your Desktop/Laptop as a Globus Endpoint and Connect](#globus-step4) { #globus-step4 } +#### 4. Activate Your Desktop/Laptop as a Globus Endpoint and Connect { #globus-step4 } After giving your ePPN time to propagate through the systems (up to 30 minutes), go to https://globus.org and log in. @@ -103,7 +103,7 @@ You can now access the files on your desktop/laptop via Globus. ![Estabilish connection to local endpoint](./imgs/globus-step4-f.png) -#### [5. Connect to the DesignSafe (TACC Corral3) Endpoint](#globus-step5) { #globus-step5 } +#### 5. Connect to the DesignSafe (TACC Corral3) Endpoint { #globus-step5 } To view both endpoint simultaneously, change the Globus' interface to the "two pane" view by toggling the buttons next to "Panels" in the upper right. @@ -128,7 +128,7 @@ After entering the appropriate path to DesignSafe on Corral, you are ready to pe ![Establish connection to Corral endpoint](./imgs/globus-step5.png) -#### [6. Perform Transfer between Your Local Enpoint and the DesignSafe (TACC Corral3) Endpoint](#globus-step6) { #globus-step6 } +#### 6. Perform Transfer between Your Local Enpoint and the DesignSafe (TACC Corral3) Endpoint { #globus-step6 } To begin your transfer, select the file/folder you wish to move to/from DesignSafe. @@ -144,27 +144,27 @@ Globus will email you when the transfer is complete. --- -### [Globus CLI Automated Transfer Guide](#globucli) { #globuscli } +### Globus CLI Automated Transfer Guide { #globuscli } Globus provides a command line interface (CLI), for those who need to perform automated data transfers. This data transfer method will likely be of most use to NHERI centers that need to bulk upload their data on a schedule. -#### [1. Follow the steps 1-3 above in Globus Data Transfer Guide](#globuscli-step1) { #globuscli-step1 } +#### 1. Follow the steps 1-3 above in Globus Data Transfer Guide { #globuscli-step1 } To set up your Globus access, follow steps 1-3 above in the Globus Data Transfer Guide . -#### [2. Activate Your Desktop/Laptop as a Globus Endpoint and Connect](#globuscli-step2) { #globuscli-step2 } +#### 2. Activate Your Desktop/Laptop as a Globus Endpoint and Connect { #globuscli-step2 } If the data you wish to transfer is located on your local machine, follow Step 4 of the Globus Data Transfer Guide to create a personal endpoint. If the data you wish to transfer is located on a server operated by your organization and does not already have a Globus Endpoint available, talk to your system administrator about creating one. -#### [3. Install the Globus CLI](#globuscli-step3) { #globuscli-step3 } +#### 3. Install the Globus CLI { #globuscli-step3 } Follow the instructions provided by Globus for installing the CLI (https://docs.globus.org/cli) *Note the recommended installation method requires a system with Python3 and the ability to run pip commands.* -#### [4. Settings for CLI Transfer](#globuscli-step4) { #globuscli-step4 } +#### 4. Settings for CLI Transfer { #globuscli-step4 } With the Globus CLI successfully installed on our local machine, we must now determine the endpoint information for DesignSafe. @@ -182,7 +182,7 @@ Search for TACC Corral3 with CILogon Authentication > Repeat the process above to attain the UUID for your local endpoint.     -#### [5. Test Globus CLI Transfer](#globuscli-step5) { #globuscli-step5 } +#### 5. Test Globus CLI Transfer { #globuscli-step5 } With the endpoint IDs, we can now do a test transfer with the Globus CLI. @@ -210,7 +210,7 @@ The full reference for the Globus CLI can found here: https://docs.globus.org/cli/reference/transfer. -#### [6. Create an Automatic Transfer Script](#globuscli-step6) { #globuscli-step6 } +#### 6. Create an Automatic Transfer Script { #globuscli-step6 } We will now create a shell script to store the transfer details (i.e., UUIDs and paths) and globus-cli syntax to allow us to quickly and reliably initiate future transfers. @@ -243,7 +243,7 @@ label=$"YourLabelHere_${label}" globus transfer --recursive --label $label "$ep1" "$ep2" ``` -#### [7. Automate Script Execution with cron](#globuscli-step7) { #globuscli-step7 } +#### 7. Automate Script Execution with cron { #globuscli-step7 } To automate the transfer we wil use the Linux scheduling utility cron to call our transfer script on a specified schedule. @@ -254,27 +254,27 @@ An example cron table entry that you can use to automatically run your transfer --- -### [Cyberduck Data Transfer Guide](#cyberduck) { #cyberduck } +### Cyberduck Data Transfer Guide { #cyberduck } Cyberduck is an open-source SSH File Transfer Protocal (sftp) client that allows you to securely connect from your laptop to DesignSafe and other Texas Advanced Computing Center (TACC) resources. -#### [1. Set up MFA using the TACC Token App](#cyberduck-step1) { #cyberduck-step1 } +#### 1. Set up MFA using the TACC Token App { #cyberduck-step1 } TACC requires multi-factor authentication (MFA) for logging directly into our resources. Go to the TACC user portal and log in with your DesignSafe/TACC credentials, click on Manage Account on the left menu, and then pair a device with your account. If needed you can explore the full MFA instructions. -#### [2. Download and Install Cyberduck](#cyberduck-step2) { #cyberduck-step2 } +#### 2. Download and Install Cyberduck { #cyberduck-step2 } Download Cyberduck and install. Note that Cyberduck is Free Software and as such is freely available to download (see link above). However, some approaches to downloading Cyberduck (such as through the Windows Store and Mac App Store) come with a registration key that disables a donation prompt. While you may purchase a registration key to support the development of Cyberduck if you wish, the **activation key is not required** to use the software for transfer files to and from DesignSafe. -#### [3. Create a New Bookmark](#cyberduck-step3) { #cyberduck-step3 } +#### 3. Create a New Bookmark { #cyberduck-step3 } Launch the Cyberduck app and select "Bookmark" > "New Bookmark". ![Figure 1. Bookmark](./imgs/cyberduck-1.png) -#### [4. Populate Bookmark](#cyberduck-step4) { #cyberduck-step4 } +#### 4. Populate Bookmark { #cyberduck-step4 } Change the top dropdown to "SFTP (SSH File Transfer Protocol)". @@ -294,7 +294,7 @@ When done close the bookmark. You will now see your newly created bookmark in th ![Figure 2. Bookmark Filled](imgs/cyberduck-2.png) -#### [5. Perform Transfer](#cyberduck-step5) { #cyberduck-step5 } +#### 5. Perform Transfer { #cyberduck-step5 } Right-click on your newly created bookmark and select "Connect to Server". You will be prompted for your TACC Token code. Input the code from your TACC Token app. @@ -306,19 +306,19 @@ To download files, select the file(s) you wish to download. Select "File" > " --- -### [Command-Line Data Transfer Guide](#cli) { #cli } +### Command-Line Data Transfer Guide { #cli } Common command-line utilities, such as scp and rsync, may also be used to transfer large amounts of data to DesignSafe. Command line tools require the shortest setup time (assuming you have a compatible terminal), however are generally found challenging for first-time users as you will need to learn unix commands. Therefore, command line transfers are only recommended in specific circumstances where other tools have been tried and found to be insufficient. -#### [1. Set up MFA using the TACC Token App](#cli-step1) { #cli-step1 } +#### 1. Set up MFA using the TACC Token App { #cli-step1 } TACC requires multi-factor authentication (MFA) for logging directly into our resources. Go to the TACC user portal and log in with your DesignSafe/TACC credentials, click on Manage Account on the left menu, and then pair a device with your account. If needed you can explore the full MFA instructions. -#### [2. Select Transfer Utility and Perform Transfer](#cli-step2) { #cli-step2 } +#### 2. Select Transfer Utility and Perform Transfer { #cli-step2 } There are several different command-line based file transfer utilities. We detail two of them here: scp and rsync. -##### [scp](#cli-step3-scp) { #cli-step3-scp } +##### scp { #cli-step3-scp } A data transfer can be performed using the secure copy (scp) utility between any Linux, Mac, or Windows (with Window's Subsystem for Linux) machine and DesignSafe. @@ -340,7 +340,7 @@ For more information execute: man scp -##### [rsync](#cli-step3-rsync) { #cli-step3-rsync } +##### rsync { #cli-step3-rsync } A data transfer can also be performed using the rsync utility between any Linux, Mac, or Windows (with Window's Subsystem for Linux) machine and DesignSafe. The rsync utility is different from the scp utility as it first compares the source and destination files prior to performing the transfer and only performs a data transfer on the file(s) if they are different. @@ -368,11 +368,11 @@ For more information execute: --- -### [Data Depot's Browser-Based Data Transfer Guide](#datadepotbrowser) { #datadepotbrowser } +### Data Depot's Browser-Based Data Transfer Guide { #datadepotbrowser } The Data Depot's browser interface allows you to conveniently upload and download small quantities of data (< 100 MB, < 25 files, < 2 folders) as well as move and copy data between directories. -#### [Upload](#datadepotbrowser-upload) { #datadepotbrowser-upload } +#### Upload { #datadepotbrowser-upload } To upload a small amount of data through your browser **login to DesignSafe** and **go to My Data**. @@ -398,7 +398,7 @@ If you wish to upload a folder, follow the same procedure as above except select -#### [Download](#datadepotbrowser-download) { #datadepotbrowser-download } +#### Download { #datadepotbrowser-download } To download a file from DesignSafe to your local desktop/laptop **select the file** you wish to download and press **Download**. @@ -408,7 +408,7 @@ To download a file from DesignSafe to your local desktop/laptop **select the fil If you would like to download an entire folder from DesignSafe, please use one of the large data transfer methods listed in this guide. -#### [Transferring Data Inside of DesignSafe](#datadepotbrowser-transferring) { #datadepotbrowser-transferring } +#### Transferring Data Inside of DesignSafe { #datadepotbrowser-transferring } You can move and copy the data inside of DesignSafe using the browser-based interface. @@ -416,7 +416,7 @@ You can move and copy the data inside of DesignSafe using the browser-based inte ![Select File for Move or Copy](./imgs/datadepotbrowser-6.png) -##### [If you selected Move](#datadepotbrowser-transferring-move) { #datadepotbrowser-transferring-move } +##### If you selected Move { #datadepotbrowser-transferring-move } Navigate to the new destination and press **Move Here**. @@ -424,7 +424,7 @@ Navigate to the new destination and press **Move Here**. ![Move Here](./imgs/datadepotbrowser-7.png) -##### [If you selected Copy](#datadepotbrowser-transferring-copy) { #datadepotbrowser-transferring-copy } +##### If you selected Copy { #datadepotbrowser-transferring-copy } **Use the drop down menu** in the top left to switch between main directories, such as My Data and My Projects, **navigate to the new destination**, and press **Copy Here**. @@ -432,11 +432,11 @@ Navigate to the new destination and press **Move Here**. --- -### [JupyterHub's Browser-Based Data Transfer Guide](#jupyterhubbrowser) { #jupyterhubbrowser } +### JupyterHub's Browser-Based Data Transfer Guide { #jupyterhubbrowser } The DesignSafe JupyterHub provides a convenient way to upload and download small amounts of data ( < 100 MB, < 25 files). -#### [To Upload a File Through Jupyter](#jupyterhubbrowser-upload) { #jupyterhubbrowser-upload } +#### To Upload a File Through Jupyter { #jupyterhubbrowser-upload } Launch Jupyter by logging into DesignSafe and going to **Workspace** > **Tools & Applications** > **Analysis** > **Jupyter** > **Select Jupyter from dropdown**. @@ -466,7 +466,7 @@ If you would like to upload an entire folder, please use one of the large data t -#### [To Download a File Through Jupyter](#jupyterhubbrowser-download) { #jupyterhubbrowser-download } +#### To Download a File Through Jupyter { #jupyterhubbrowser-download } To download a file, **select the file** then select **Download**. @@ -476,7 +476,7 @@ To download a file, **select the file** then select **Download**. If you would like to download an entire folder, please use one of the large data transfer methods listed in this guide. -### [Cloud Storage Transfer](#cloud) { #cloud } +### Cloud Storage Transfer { #cloud } DesignSafe provides users the capability to connect to their preferred cloud storage provider. @@ -484,7 +484,7 @@ Once connected, data held on the selected cloud storage provider can be easily c The three main cloud storage providers, Box, Dropbox,and Google Drive, are supported on DesignSafe. Detailed instructions for setting up these integrations is provided below. -#### [Box](#cloud-box) { #cloud-box } +#### Box { #cloud-box } **Login to DesignSafe** and go to **Workspace > Data Depot > Box.com**. @@ -502,7 +502,7 @@ Return to the Box.com section of the Data Depot. You can now copy files to and f ![Box Complete](./imgs/cloudstorage-1.png) -#### [Dropbox](#cloud-dropbox) { #cloud-dropbox } +#### Dropbox { #cloud-dropbox } Login to DesignSafe and go to Workspace > Data Depot > Dropbox.com. @@ -520,7 +520,7 @@ Return to the Dropbox.com section of the Data Depot. You can now copy files to a ![Complete Dropbox](./imgs/cloudstorage-1.png) -#### [Google Drive - CURRENTLY NOT FUNCTIONAL](#cloud-googledrive) { #cloud-googledrive } +#### Google Drive - CURRENTLY NOT FUNCTIONAL { #cloud-googledrive } GOOGLE HAS MADE CHANGES THAT WE ARE WORKING THROUGH TO REENABLE (status as of January 11, 2023) diff --git a/user-guide/docs/managingdata/experimentalfacilitychecklist.md b/user-guide/docs/managingdata/experimentalfacilitychecklist.md index 1edb269..355caea 100644 --- a/user-guide/docs/managingdata/experimentalfacilitychecklist.md +++ b/user-guide/docs/managingdata/experimentalfacilitychecklist.md @@ -1,8 +1,8 @@ -### [DesignSafe-EF Onboarding Checklist for Data Curation](#onboarding) { #onboarding } +### DesignSafe-EF Onboarding Checklist for Data Curation { #onboarding } DesignSafe has been developed as a comprehensive research environment supporting a range of activities from research planning to cloud-based data analysis to data curation/publication. We encourage users to take full advantage of the DesignSafe capabilities associated with both the Data Depot data repository and the Tools and Apps. To learn more about all of these capabilities, watch this Introductory Webinar. -### [Phase 1 - Before arriving to the Experimental Facility (EF):](#phase1) { #phase1 } +### Phase 1 - Before arriving to the Experimental Facility (EF): { #phase1 } * Create an account on DesignSafe: Account Registration * Familiarize yourself with the Data Depot and the curation process. @@ -30,7 +30,7 @@ DesignSafe has been developed as a comprehensive research environment supporting * PIs and co-PIs should be informed and contribute to curation decisions including final publication. -### [Phase 2 - At the EF:](#phase2) { #phase2 } +### Phase 2 - At the EF: { #phase2 } * The following steps will be completed as a team. * Upload project data files into your Project as soon as you gather and produce them. @@ -40,7 +40,7 @@ DesignSafe has been developed as a comprehensive research environment supporting * You may begin the curation process as soon as you start uploading files to DesignSafe. You will continue this process after your work at the EF. -### [Phase 3 – After the EF:](#phase3) { #phase3 } +### Phase 3 – After the EF: { #phase3 } * Attend Virtual Curation Office Hours. As a team, make an appointment with the DesignSafe Data Curator to discuss data management. Office hours are every Tuesday and Thursday at 1 pm central: * Curation Office Hours Zoom Link @@ -54,7 +54,7 @@ DesignSafe has been developed as a comprehensive research environment supporting * Publish your project using the Prepare to Publish button within the Publication Preview. Confirm the project metadata, files to be published, etc., and then click Request DOI and Publish. * The project will be publicly available in the Published section of the Data Depot within about 24 hours. -### [General Comments](#comments) { #comments } +### General Comments { #comments } * Publishing the data from your project quickly will help you comply with the requirements of your funding sources, allow you to cite your data with a DOI in your upcoming publications and presentations, and bring prompt attention to your work. * DesignSafe provides the possibility to publish one experiment at a time, so you do not need to finish your entire research project to publish all the experiments. diff --git a/user-guide/docs/managingdata/settingpathtodesignsafe.md b/user-guide/docs/managingdata/settingpathtodesignsafe.md index 96ce1f5..a01bd35 100644 --- a/user-guide/docs/managingdata/settingpathtodesignsafe.md +++ b/user-guide/docs/managingdata/settingpathtodesignsafe.md @@ -3,7 +3,7 @@ The data stored on DesignSafe resides on the large (40 PB), shared data resource There are four main locations for data transfers on DesignSafe: My Data, My Projects, Published, and Published (NEES), they are each presented in detail below. -### [Path to My Data](#mydata) { #mydata } +### Path to My Data { #mydata } For My Data set Path to /data/designsafe/mydata/<username>/ @@ -13,7 +13,7 @@ Replace <username> with your username. You can find your -### [Path to My Projects](#myprojects) { #myprojects } +### Path to My Projects { #myprojects } For My Projects set Path to /corral/projects/NHERI/projects/<project-uid>/ @@ -21,7 +21,7 @@ Replace <project-uid> with your projects unique identifie ![Path to My Projects](./imgs/settingpath-2.png) -### [Path to Published](#published-nheri) { #published-nheri } +### Path to Published { #published-nheri } For Published DesignSafe projects set Path to /corral/projects/NHERI/published/<PRJ-XXXX> @@ -30,7 +30,7 @@ Replace <PRJ-XXXX> with your project's number. You can fi ![Path to Published](./imgs/settingpath-3.png) -### [Path to Published (NEES)](#published-nees) { #published-nees } +### Path to Published (NEES) { #published-nees } For Published (NEES) projects set Path to /corral/projects/NHERI/public/projects/<NEES-XXXX-XXXX.groups> diff --git a/user-guide/docs/tools/advanced/cybersecurity.md b/user-guide/docs/tools/advanced/cybersecurity.md index b1f7ec0..5bb3bf3 100644 --- a/user-guide/docs/tools/advanced/cybersecurity.md +++ b/user-guide/docs/tools/advanced/cybersecurity.md @@ -15,101 +15,101 @@ DesignSafe implements cybersecurity based upon TACC's well-established approach The following is the DesignSafe Security Plan, as adapted from TACC's Security Plan. This Security Plan forms the basis by which other NHERI sites will be audited and is the baseline security posture for NHERI Awardees. Successful implementation of the Cybersecurity Program will be measured by completing the security audits and by completing resolution of any audit results. -## [1. Roles and Responsibilities](#1) { #1 } +## 1. Roles and Responsibilities { #1 } While cybersecurity is the responsibility of all stakeholders, specific duties and responsibilities of certain key individuals is made explicit here for the sake of clear accountability. -### [1.1. DesignSafe Management Team](#1.1) { #1.1 } +### 1.1. DesignSafe Management Team { #1.1 } Ultimately, overall responsibility for the success of the DesignSafe CI lies with the management team comprised of the DesignSafe PI/Project Director Ellen Rathje, co-PI's Clint Dawson, Jamie Padgett, Jean-Paul Pinelli, and Dan Stanzione, Deputy Project Director Tim Cockerill, Project Manager Natalie Henriques and Portal Lead Steve Mock. This management team will be assisted by the DesignSafe CSO in the matter of the cybersecurity program and its overall goals, objectives, and priorities in order to support the overall mission of DesignSafe. The senior management team is also responsible for ensuring that adequate resources are applied to the security program to ensure its success. -### [1.2. DesignSafe Chief Security Officer](#1.2) { #1.2 } +### 1.2. DesignSafe Chief Security Officer { #1.2 } The DesignSafe CSO Nathaniel Mendoza, also TACC's Information Security Officer, directs DesignSafe's day-to-day management of its security program, including maintaining a secure environment for the DesignSafe CI, providing security advice to the DesignSafe user community, conducting regular security audits, and coordinating all security related interactions among the various participating NHERI organizations as the leader of the Security Working Group. -### [1.3. NHERI Awardees](#1.3) { #1.3 } +### 1.3. NHERI Awardees { #1.3 } To ensure that proper security measures are taken by each of the NHERI Awardees (each EF, NCO, SimCenter, and RAPID), local site security is ultimately the responsibility of each NHERI Awardee Principal Investigator. Each awardee will identify a main point of contact to participate in the NHERIwide Security Working Group, coordinate with the DesignSafe CSO for security audits, and be a member of the incident response team. -### [1.4. Security Working Group](#1.4) { #1.4 } +### 1.4. Security Working Group { #1.4 } The DesignSafe CSO is the leader of this working group, and members are the local site security points of contact from each of the other NHERI awardees – one each from the seven EF's, one from the RAPID facility, one from the SimCenter, and one from the NCO. Communications will be maintained via a monthly Zoom virtual meeting and via an email list. The annual security audits will be done virtually as well. -## [2. Administrative Safeguards](#2) { #2 } +## 2. Administrative Safeguards { #2 } -### [2.1. Risk Assessment](#2.1) { #2.1 } +### 2.1. Risk Assessment { #2.1 } -#### [2.1.1.Risk Assessment Policy and Procedures](#2.1.1) { #2.1.1 } +#### 2.1.1.Risk Assessment Policy and Procedures { #2.1.1 } DesignSafe's risk assessment policy and procedures are developed, reviewed, updated and disseminated by the DesignSafe CSO. This is done annually, or as needed if urgent security information becomes available and new resources are brought online in the information system. Risk assessment identifies threats to and vulnerabilities of DesignSafe's information system. -#### [2.1.2.Risk Assessment](#2.1.2) { #2.1.2 } +#### 2.1.2.Risk Assessment { #2.1.2 } DesignSafe risk assessment takes into account vulnerabilities, threat sources and security controls that are planned or in place to determine the resulting level of residual risk posed to DesignSafe's operations, assets or individuals based on the operation of the information system. Risk assessments are conducted and results documented as threats are identified and addressed. -### [2.2. Audit](#2.2) { #2.2 } +### 2.2. Audit { #2.2 } DesignSafe's comprehensive cybersecurity approach includes a security audit at each of the NHERI Awardees performed once a year. The audits use security best practices to verify that each server-class system operating at a NHERI Awardee site is operating in a manner to limit the potential for security incidents and breaches. Security incidents and data breaches could invalidate data being collected by scientists, damage experimental equipment, and spread the damage to the DesignSafe resources. No system can be perfectly secure, but regular audits of the system provide vital information for the regular upkeep and secure maintenance of the server systems. -#### [2.2.1.Schedule for Audits](#2.2.1) { #2.2.1 } +#### 2.2.1.Schedule for Audits { #2.2.1 } Each NHERI Awardee together with the DesignSafe CSO will determine an appropriate time schedule for performing the audit. This will be coordinated within 6 months after NSF awards are made with each NHERI Awardee. The audits will generally be done once a year, and will be performed virtually. However, in the event that a security incident occurs then further audits may be done. In all cases, the timing for the audit will be decided in consultation with the NHERI Awardee, such that the site operations are minimally affected and the resources of the site IT staff are optimally utilized. -#### [2.2.2.Actions following the Audits](#2.2.2) { #2.2.2 } +#### 2.2.2.Actions following the Audits { #2.2.2 } If there are audit findings, the DesignSafe CSO will recommend corrective actions for the NHERI Awardee to implement. A formal report will be generated once a year that summarizes the results of the audits for each NHERI Awardee. The report will identify the assets that were a part of the audit, where the audit did find vulnerabilities and security breaches, and remediation actions, both short term and long term. This report will not be for public disclosure, keeping in view the security sensitive nature of the information, but will be made available to the NSF. -## [3. Technical Safeguards](#3) { #3 } +## 3. Technical Safeguards { #3 } -### [3.1. Proactive Security Monitoring and Detection](#3.1) { #3.1 } +### 3.1. Proactive Security Monitoring and Detection { #3.1 } The DesignSafe resources are protected within the TACC datacenter by a firewall and a Bro Intrusion Detection system that monitors 100% of the network traffic and can automatically block IP's. Further automated analysis and detection is accomplished by ingesting logs from all datacenter resources into TACC's Splunk Operational Intelligence system. TACC's 24/7/365 Operations staff receive notifications from these monitoring systems and can take corrective action and notify the CSO which initiates the formal incident response. Additionally, users can report an incident via the DesignSafe Help system that will also notify TACC's Operations staff. All connection to the DesignSafe CI, for example from EF site servers, must be encrypted and all data transport is encrypted e.g. using SSH, HTTPS, etc. -### [3.2. Vulnerability Scanning](#3.2) { #3.2 } +### 3.2. Vulnerability Scanning { #3.2 } DesignSafe uses appropriate vulnerability scanning tools and techniques, scanning for vulnerabilities in the information system weekly using a third-party software tool and ad-hoc as needed such as after major systems changes or when significant new vulnerabilities potentially affecting the system are identified and reported. -## [4. Administrative Safeguards](#4) { #4 } +## 4. Administrative Safeguards { #4 } -### [4.1. Authentication and Authorization](#4.1) { #4.1 } +### 4.1. Authentication and Authorization { #4.1 } DesignSafe's plan for Authentication and Authorization is utilizes TACC's well-established and proven infrastructure for a secure CI environment. -### [4.2. Access Control](#4.2) { #4.2 } +### 4.2. Access Control { #4.2 } Identification, authentication, and authorization are controls that facilitate access to and protect DesignSafe resources and data. Access to non-public resources will be achieved by unique user credentials and will require authentication. DesignSafe will assign a username and password for identification and authentication purposes to each individual that has a need to access DesignSafe resources. In all cases, only the minimum privileges necessary to complete required tasks are assigned to that individual. Privileges assigned to each individual will be reviewed on a periodic basis and modified or revoked upon a change in status within the DesignSafe community. All DesignSafe resources must use only encrypted authentication and authorization mechanisms unless otherwise authorized by the CSO. -## [5. Policy and Procedures](#5) { #5 } +## 5. Policy and Procedures { #5 } -### [5.1. Creating User Accounts](#5.1) { #5.1 } +### 5.1. Creating User Accounts { #5.1 } DesignSafe user accounts will be created as TACC user accounts, and users will be required to provide identification information that enables TACC user services personnel to ensure we are compliant with University of Texas, state, and federal laws and regulations per standard TACC user policies. We will also federate account creation with InCommon, such that users can link their InCommon identity with a TACC identity and use their local institution credentials. -### [5.2. User Credentials](#5.2) { #5.2 } +### 5.2. User Credentials { #5.2 } DesignSafe will initially use single-factor authentication via a user password. Multi-factor authentication is being phased into TACC's authentication infrastructure, and if deemed necessary will be applied to the DesignSafe CI. For multi-factor authentication, users would have a password and in addition a second mechanism of a short-lived access code provided by a fob or via mobile device app. The use of group accounts for administrative purposes and shared passwords for those accounts will be minimized where technically feasible. DesignSafe staff requiring privileged user access will be using RSASecurID fobs for controlling root access to resources. Credentials may be used only by the authorized user. Passwords or accounts should never be shared with anyone. Account owners will be held responsible for any actions performed using their accounts. DesignSafe staff will never ask users to disclose their passwords in any manner. Passwords should never be written down and left in plain sight, or stored in plain text online. -### [5.3. Inactive Account Expiration](#5.3) { #5.3 } +### 5.3. Inactive Account Expiration { #5.3 } DesignSafe accounts that are inactive for 120 days will be deactivated, and the user will need to request reactivation of the account. -## [6. Physical Safeguards](#6) { #6 } +## 6. Physical Safeguards { #6 } -### [6.1. Physical Access Authorization](#6.1) { #6.1 } +### 6.1. Physical Access Authorization { #6.1 } DesignSafe controls physical access points (including designated entry/exit points) to facilities containing information systems (except for those areas within the facilities officially designated as publicly accessible) and to verify individual access authorizations before granting access to the facilities. Senior management decides who is authorized to enter controlled areas, and the CISO or designated personnel are responsible for enabling access. -### [6.2. Access Control for Transmission Medium](#6.2) { #6.2 } +### 6.2. Access Control for Transmission Medium { #6.2 } Adequate physical protection is in place for wiring closets and communication demark areas to prevent accidental damage, disruption, or intentional physical tampering of transmission lines. -## [7. Awareness and Training](#7) { #7 } +## 7. Awareness and Training { #7 } Awareness of the DesignSafe Cybersecurity Plan for NHERI Awardees will be achieved via the Security Working Group, and assurance of awareness and compliance is achieved via the aforementioned audits. @@ -117,7 +117,7 @@ NHERI Awardees shall adhere to their local University cybersecurity policies, an TACC follows the UT-Austin security awareness and training policy. Upon hiring, all UT-Austin staff are required to complete designated modules in the UT-Austin Compliance Training System www.utexas.edu/hr/hrpro/comply/training.html based on job function, including CW 170 IT Security Awareness. Staff members are required to perform this testing every two (2) years. The Compliance Training System maintains a log of completed training and automatically notifies a staff member when retraining is required. All training modules are developed and facilitated by UT-Austin's University Compliance Services (UCS). -## [8. Incident Response and Notification Procedures](#8) { #8 } +## 8. Incident Response and Notification Procedures { #8 } Upon notification of a possible security incident, the DesignSafe CSO will lead a formal incident response. The DesignSafe Security Working Group will be informed that a response is being initiated, and the response team will be formed based upon the extent of the incident. It will be necessary to quickly suspend the suspected user account(s), services, or systems to prevent an escalation of the incident. The team will analyze all available information, interrogate any persons involved, determine corrective measures, and assure corrections are implemented and effective prior to allowing any accounts, services or systems to be brought back online. An incident report will be generated and shared with the Security Working Group. Relevant information from the report will be shared with the DesignSafe Management Team and NSF as appropriate. diff --git a/user-guide/docs/tools/advanced/dsfaq.md b/user-guide/docs/tools/advanced/dsfaq.md index 7016b08..145bee5 100644 --- a/user-guide/docs/tools/advanced/dsfaq.md +++ b/user-guide/docs/tools/advanced/dsfaq.md @@ -4,7 +4,7 @@ An Expanding Collection of the Most Frequently Asked Questions --- -## [FAQ Categories](#categories) { #categories } +## FAQ Categories { #categories }
    -## [Users and Accounts](#faq-users) { #faq-users } +## Users and Accounts { #faq-users } Q: Who can be a DesignSafe User?
    A: Any natural hazards researcher or practitioner that wants an environment to store, analyze, curate, publish, and discover data with a community of peers. @@ -51,7 +51,7 @@ A: With an account, you can: A: Upon receiving your new user account, you will be sent an email to join our Slack team to participate in discussions and give your input on the site.
    -## [Training](#faq-training) { #faq-training } +## Training { #faq-training } Q: Does DesignSafe provide training on how to use the Data Depot and the applications in Tools & Applications?
    A: Upcoming training opportunities are posted in the DesignSafe Learning Center, announced via email, and posted on DesignSafe Slack. Webinars covering DesignSafe features are hosted roughly monthly during the academic year. @@ -60,7 +60,7 @@ A: YouTube Channel.
    -## [Data Depot](#faq-datadepot) { #faq-datadepot } +## Data Depot { #faq-datadepot } Q: What data can I upload to the Data Depot?
    A: There are no restrictions on file types for the data that you upload to the Data Depot. We do have special handling procedures for Managing Protected Data such as regulated/secure/PHI/PII/IRB/human-subjects data, so do not upload these data types directly into the Data Depot. All other types of data that you need to perform your research is welcomed and encouraged! @@ -82,13 +82,13 @@ A: This will require contacting your IT department to make sure your company/uni A: The Published directory in the Data Depot holds the NEES Public Data. Projects published using DesignSafe will be hosted there as well. The DOIs for the NEES Public Data point to the Data Depot.
    -## [Tools & Applications](#faq-workspace) { #faq-workspace } +## Tools & Applications { #faq-workspace } Q: I don't see the application I need in the Tools & Applications. How can I use my application?
    A: There are a couple of different approaches that can be taken. If this is an open source application that is used by many researchers throughout this research community, then we can work with you to determine if it is a candidate to be added to the Tools & Applications in the portal. Please submit a ticket to start the conversation. Another approach is that you could request an allocation on one of TACC's HPC resources, and then you would log directly into the HPC system to run your application.
    -## [MATLAB](#faq-matlab) { #faq-matlab } +## MATLAB { #faq-matlab } Q: Do I need to provide my own license to run MATLAB?
    A: No, you don’t need to provide your own license to run MATLAB in the Discovery Workspace. Our license with MathWorks allows for academic use and you can be from any academic institution. We have a process whereby you request access to MATLAB and we then verify you are an academic user prior to adding you to the license. Simply click on MATLAB in the Discovery Workspace to start the process of requesting access. @@ -103,7 +103,7 @@ A: If you are starting up with a folder that has a large number of files and/or A: If you do not specify a location, the default output is shown in the grayed-out text in the Job Output Archive Location field in the job submission form, which is your My Data/Archive/Jobs/YYYY-MO-DD/JobName folder.
    -## [Jupyter](#faq-jupyter) { #faq-jupyter } +## Jupyter { #faq-jupyter } Q: What is Jupyter?
    A: The Jupyter Notebook is a web application that allows you to create and share documents that contain live code, equations, visualizations and explanatory text. A more detailed description including a list of more than 40 supported programming languages can be found on the Jupyter website. @@ -112,7 +112,7 @@ A: The Jupyter Notebook is a web application that allows you to create and share A: Many people use Jupyter in a similar fashion as they use MATLAB to analyze and plot their data. We will be sharing example Jupyter Notebooks soon that you can copy into your My Data and customize for your research. We also provide Jupyter training.
    -## [OpenSees](#faq-opensees") { #faq-opensees } +## OpenSees { #faq-opensees } Q: What research is enabled by OpenSees?
    A: The Open System for Earthquake Engineering Simulation (OpenSees) is a software framework for simulating the seismic response of structural and geotechnical systems.
    @@ -128,7 +128,7 @@ A: OpenSees Express performs analysis using a single Tcl script, and runs on a v A: If you do not specify a location, the default output is shown in the grayed-out text in the Job Output Archive Location field in the job submission form, which is your My Data/Archive/Jobs/YYYY-MO-DD/JobName folder.
    -## [ADCIRC](#faq-adcirc) { #faq-adcirc } +## ADCIRC { #faq-adcirc } Q: What research is enabled by ADCIRC?
    A: ADCIRC is a system of computer programs for solving time dependent, free surface circulation and transport problems in two and three dimensions. These programs utilize the finite element method in space allowing the use of highly flexible, unstructured grids. Typical ADCIRC applications have included: @@ -150,7 +150,7 @@ A: If you do not specify a location, the default output is shown in the grayed-o A: If you do not specify a location, the default output is shown in the grayed out text in the Job Output Archive Location field in the job submission form, which is your My Data/Archive/Jobs/YYYY-MO-DD/JobName folder.
    -## [OpenFOAM](#faq-openfoam) { #faq-openfoam } +## OpenFOAM { #faq-openfoam } Q: What research is enabled by OpenFOAM?
    A: OpenFOAM has an extensive range of features to solve anything from complex fluid flows involving chemical reactions, turbulence and heat transfer, to acoustics, solid mechanics and electromagnetics.
    @@ -160,7 +160,7 @@ A more detailed description can be found on the -## [ParaView](#faq-paraview) { #faq-paraview } +## ParaView { #faq-paraview } Q: What data analysis and visualization capabilities are enabled by ParaView?
    A: ParaView is an open-source data analysis and visualization application. ParaView users can quickly build visualizations to analyze their data using qualitative and quantitative techniques. @@ -171,7 +171,7 @@ A more detailed description can be found on the     Experimental Facilities on our website. Research instruments at the facilities include wind tunnels, shake tables, centrifuges, wave pools, and mobile shaker trucks. @@ -180,7 +180,7 @@ A: You can explore the NSF NHERI Program Exp A: Each facility hosts workshops to provide prospective users with the knowledge of a facility's capabilities and discuss details toward developing research proposals, such as to the NSF Engineering for Natural Hazards Program.
    -## [Citing DesignSafe](#faq-citation) { #faq-citation } +## Citing DesignSafe { #faq-citation } Q: How do I cite my use of DesignSafe in my papers, presentations, and publications?
    A: DesignSafe has published a marker paper that you can cite as a reference: diff --git a/user-guide/docs/tools/advanced/hpcallocations.md b/user-guide/docs/tools/advanced/hpcallocations.md index 7640b3b..73e6472 100644 --- a/user-guide/docs/tools/advanced/hpcallocations.md +++ b/user-guide/docs/tools/advanced/hpcallocations.md @@ -1,14 +1,14 @@ # HPC Allocations -## [Introduction](#intro) { #intro } +## Introduction { #intro } The following are the policies governing allocations for compute time and storage on the DesignSafe platform, a comprehensive natural hazards research environment for experimental, theoretical, and computational science. The allocation policy for DesignSafe considers that users may have a range of use cases, from short term analysis work to long term data storage, for very large or very small experiments. All registered users are provided access to the Tools & Applications in a controlled manner that limits the amount of computing capacity, and they may request an additional allocation for various purposes such as the need for larger scale computation or to run applications that are not provided via the portal. New use cases for DesignSafe may emerge—please don’t hesitate to contact the project team if you have additional needs. -## [Who is eligible for an Account](#account) { #account } +## Who is eligible for an Account { #account } DesignSafe is broadly available to any researcher or practitioner working in open Natural Hazards Engineering and Social Science research. By “open,” the expectation is that any research performed on DesignSafe is unclassified, and will result in publication in a broadly available journal or conference. We also strongly encourage you to publish your datasets in our CoreTrustSeal certified data repository. -## [Who is eligible for Allocations](#allocation) { #allocation } +## Who is eligible for Allocations { #allocation } All registered users are provided access to the Tools & Applications in a controlled manner that limits the amount of computing capacity, and may request an additional Allocation for various purposes such as the need for larger scale computation or to run applications that are not provided via the portal. An additional allocation via DesignSafe is headed by a project PI, typically a faculty member or research scientist at a US-based academic institution (private sector users are also welcome, see below). The PI may then add additional researchers to their allocation at their discretion. PIs are responsible for ensuring that any users added to their allocation comply with the terms and conditions for use of the DesignSafe and TACC resources. Collaborative projects involving non-U.S. researchers are encouraged as long as they include substantive intellectual participation by the U.S. researchers. In joint research projects, foreign collaborators are eligible to make use of that allocation in a manner consistent with the request. While a PI is typically a faculty member or staff researcher at a U.S. academic institution, there are a number of other cases where an individual may be eligible to be a PI: @@ -20,11 +20,11 @@ All registered users are provided access to the Tools & Applications in a co
  • U.S. commercial organizations, especially small businesses with strong capabilities in scientific or engineering research or education may apply for an allocation. The DesignSafe project is interested in supporting projects that couple industrial research resources and perspectives with those of universities; therefore, it especially welcomes requests from cooperative projects involving both universities and the private commercial sector. It is necessary for these projects to submit their work in an open forum, and make the work readily available to the public.
    • -## [Allocation Types and Constraints](#description) { #description } +## Allocation Types and Constraints { #description } DesignSafe provides access to a variety of computing and storage resources, accessible through several different interfaces: via the web-based Tools & Applications portal, or via API's, or via direct command-line access to TACC's HPC systems. Web-based access via Tools & Applications has built-in constraints that limit the amount of compute time. For users requiring access via API or direct command-line access to HPC resources, they can request additional allocations of computational time and/or data storage. TACC has several different HPC resources and we will provide allocation on the resource that is best suited to achieve your research goals. -### [Allocation Sizes and Limits](#allocationsizes) { #allocationsizes } +### Allocation Sizes and Limits { #allocationsizes } Additional allocations available via DesignSafe enable researchers to directly access TACC's HPC systems via the command line interface. The following describes the types and sizes of allocations available via DesignSafe. Some users will ultimately require even larger amounts of compute time than is offered under this policy, and when that happens we will recommend other allocation methods for NSF-supported resources that are available such as Frontera Allocations or ACCESS Allocations. @@ -37,7 +37,7 @@ Additional allocations available via DesignSafe enable researchers to directly a Allocations are provided through NSF funding at no direct cost to the end user to anyone who meets the eligibility criteria above. Users who are not eligible for an NSF-provided allocation, may choose to purchase additional compute time or storage capacity from TACC. These services will be provided based on TACC's services rates in effect at the time of purchase. For example as of this writing in January 2024, storage is available for approximately $60/TB/year, and compute time is available for approximately $0.50 per node hour. -### [Allocation Limit per PI](#allocationlimit) { #allocationlimit } +### Allocation Limit per PI { #allocationlimit } An individual may be a PI on only one active DesignSafe additional allocation request at a given time. Several distinct research activities can be combined in a single allocation request, however, the allocation request for each activity must be justified, and any allocation-size limits apply to the aggregate request. @@ -45,11 +45,11 @@ The single-allocation rule is designed to minimize the effort required by PIs fo Similarly, to minimize the effort required to gain access to DesignSafe, closely collaborating researchers should submit a single collaborative allocation request rather than several individual requests. For example, a PI and associated post-doctoral researchers; investigators supported by the same funding grant; and researchers in the same lab group should consider submitting a request describing and justifying the various sub-activities. One of the collaborators is designated as the PI, and others can be designated as co-PIs. -### [Allocation Duration](#duration) { #duration } +### Allocation Duration { #duration } Allocations for DesignSafe resources are made for up to a 12-month period. PIs can continue their activities in subsequent years through annual renewal requests. At the end of each 12-month allocation period, projects forfeit any unused compute SUs. Users with a Startup Allocation need not wait the full 12 months to apply for a Research Allocation, and can apply as soon as they receive preliminary results suitable for their request. Educational Allocations will typically be limited in duration to the academic semester of the class in which they take place. -## [Review Criteria for Additional Allocation Proposals](#criteria) { #criteria } +## Review Criteria for Additional Allocation Proposals { #criteria } DesignSafe additional allocations are reviewed for merit by a committee consisting of members of the Allocation Advisory Board. The board makes recommendations to the project team based solely on the merit of the proposal, and not on overall availability of the resource. The board will review the proposal and make a recommendation to the project team based on the following criteria: @@ -63,15 +63,15 @@ DesignSafe additional allocations are reviewed for merit by a committee consisti Proposals that are deemed competitive by the Allocation Advisory Board are then reviewed by the project team for technical feasibility (i.e., can the project be implemented in the DesignSafe environment?) and for the availability of time on the resource. For Research Allocations, proposals will be reviewed at least once per quarter. Startup and Educational Allocation requests receive a fast track review, bypassing the Allocation Advisory Board. DesignSafe project staff will review these requests according to the merit review criteria above, and will act on these requests within two weeks of their submission. -### [Conflict of Interest and Confidentiality](#coi) { #coi } +### Conflict of Interest and Confidentiality { #coi } Every effort is made to avoid conflicts of interest. Reviewers are not allowed to review or be present for the discussion of requests from their home institution, former students, postdocs, advisors, or current and recent collaborators. If in the opinion of a PI a certain individual has a conflict of interest, the PI may request that the individual not act as reviewer on their request. All reviews remain confidential and are made available only to the PI and Co-PIs submitting the request, assigned reviewers from the Allocation Advisory Board, DesignSafe project staff involved in the allocation review process, and NSF program officers for the DesignSafe project. While the contents of reviews and details of allocation requests and experiments remain confidential, a list of projects receiving allocations and the general area of research may be made public on the DesignSafe web site and used in project reports and presentations. -## [How to apply for an Additional Allocation](#apply) { #apply } +## How to apply for an Additional Allocation { #apply } A proposal for an additional allocation includes information about the eligibility of the requestor, a description of the research to be performed and its sources of support, and a justification for the amount of resources requested. Detailed guidance is in the following sections. When the proposal is complete it can be submitted via a help ticket. -### [Allocation Proposal Guidance](#guidance) { #guidance } +### Allocation Proposal Guidance { #guidance } A Startup Allocation request consists of the following components: @@ -114,7 +114,7 @@ A Research Allocation request consists of the following components:
  • References (optional, no page limit)
    • -### [Document Formatting](#formatting) { #formatting } +### Document Formatting { #formatting } While readability is of greatest importance, documents must satisfy the following minimum requirements. Documents that conform to NSF proposal format guidelines will satisfy these guidelines. @@ -137,15 +137,15 @@ A font size of less than 10 points may be used for mathematical formulas or equa
  • Page Numbering: Page numbers should be included in each file by the submitter.
    • -### [Supplemental Requests](#supplements) { #supplements } +### Supplemental Requests { #supplements } A supplement is a request for additional resources during an existing allocation's one year time frame. Its purpose is to support changes in the original computational research plan that are required to achieve the scientific goals of the project. This may include altered or new goals, or support for projects proceeding more rapidly than anticipated or that require more resources than anticipated. Supplement awards are highly dependent upon availability of resources. Supplemental requests require a progress report that documents what has been done with the originally awarded allocation, and makes the case for additional resources. -### [Extension Requests](#extensions) { #extensions } +### Extension Requests { #extensions } Extensions of allocation periods beyond the normal 12-month duration can be requested. This request brings no new allocation, but keeps unused allocations from expiring. A brief reason is required for not using the awarded allocation, but no formal documentation is needed. -### [Guidelines for a Successful Allocation Request](#guidelines) { #guidelines } +### Guidelines for a Successful Allocation Request { #guidelines } Well-written research allocation requests will meet the following guidelines: @@ -160,7 +160,7 @@ The purpose of DesignSafe is to accelerate research progress in natural hazards The project team is interested in receiving as many excellent requests as possible, and encourages any potential investigators who wish to use the resource to not hesitate in contacting the team for additional help in preparing this request, with questions about the process, or to discuss any type of project that may not fit the structure described here. The project team also welcomes input in improving this process, and encourages feedback on both this document and the process. -## [Acknowledgement of Support](#support) { #support } +## Acknowledgement of Support { #support } An acknowledgement of support from the DesignSafe project and the National Science Foundation should appear in any publication of material, whether copyrighted or not, that describes work which benefited from access to DesignSafe cyberinfrastructure resources. If you wish to acknowledge DesignSafe, the following paper can be used as a reference: diff --git a/user-guide/docs/tools/hazard/hazardapps.md b/user-guide/docs/tools/hazard/hazardapps.md index ddd7b5b..5eb97da 100644 --- a/user-guide/docs/tools/hazard/hazardapps.md +++ b/user-guide/docs/tools/hazard/hazardapps.md @@ -1,27 +1,27 @@ ## Hazard Applications -### [Managing Applications](#managing) { #managing } +### Managing Applications { #managing } -#### [Requesting New Applications](#managing-request) { #managing-request } +#### Requesting New Applications { #managing-request } DesignSafe regularly adds new software applications in support of natural hazards engineering research. You may contact DesignSafe by submitting a help ticket if you would like to request the addition of a software application to the Workspace. -#### [Getting Your Own HPC Application](#managing-installing) { #managing-installing } +#### Getting Your Own HPC Application { #managing-installing } For those researchers with larger computational needs on the order of tens of thousands, or even millions of core-hours, or if you have a software application that we don't support in the web portal, you may request your own allocation of computing time on TACC's HPC systems. Your files can still be stored in the Data Depot, allowing you to share your research results with your team members, as well as curate and publish your findings. -#### [Commercial/Licensed Applications](#managing-commercial) { #managing-commercial } +#### Commercial/Licensed Applications { #managing-commercial } The DesignSafe infrastructure includes support for commercial/licensed software. Wile in some cases licenses can be provided by the DesignSafe project itself, not all vendors will make licenses available for larger open communities at reasonable cost. You may contact DesignSafe by submitting a help ticket if you have questions regarding a commercial software application. -### [Hurricane Data Analysis User Guide](#hurricane) { #hurricane } +### Hurricane Data Analysis User Guide { #hurricane } The Hurricane Data Analysis (HDA) is a graphical user interface (GUI) MATLAB program used to convert sensor raw data to physical data. The hurricane Matthew data was collected by deploying a Wireless Sensor Network system on the roof of a Satellite Beach, Florida house. The wireless system measured the pressure distribution over the roof top. The wind speed and wind direction data were obtained from the FCMP tower and Melbourne airport weather station. The pressure data is analyzed by selecting optimum sample period which yielded a steady value for the peak suction pressure. This averaging time was found to be around 300 seconds. The RMS pressure fluctuations are quantified with respect to the 300-second mean pressure. The mean pressure data correlated well with the wind speed data and the mathematical expressions are developed for different segments, before hurricane, during hurricane and after hurricane. Wireless Sensor Network is an autonomous full-scale hurricane data measurement system with 30 pressure and temperature sensors, and an anemometer to measure pressure, temperature, speed and wind direction respectively. All sensors are connected wirelessly through 3 routers to a base modem, which is connected to the laptop for collecting the data. The data from the laptop was uploaded through a cellular network at five minutes interval to a Box account, which provides cloud storage and file sharing collaboration. The entire wireless sensor network system was monitored using Team Viewer remote desktop application. All the data collected in the Box was then uploaded into the DesignSafe-ci “My Data”, which can be shared with researchers. To perform the analysis a unique graphical user interface (GUI) application was created using MATLAB, which is capable of analyzing the complete data set on a single run. The user can run the GUI code instantly on the cloud and provide plots of the physical data. -#### [How to Start a Hurricane Data Analysis Interactive Session in the Workspace](#hurricane-start) { #hurricane-start } +#### How to Start a Hurricane Data Analysis Interactive Session in the Workspace { #hurricane-start } -### [Activate Ls-Dyna on DesignSafe](#activate) { #activate } +### Activate Ls-Dyna on DesignSafe { #activate } DesignSafe (through TACC) has negotiated with LSTC to allow LS-DYNA access on TACC systems for academic research. Users can submit a ticket (https://www.designsafe-ci.org/help/new-ticket/) requesting LS-DYNA access and are granted access upon verification with LSTC that they have an existing academic departmental license or that you acquire such license. @@ -38,7 +38,7 @@ A Request Activation button is also available in Workspace > Simula Once activated, Ls-Dyna will appear in Workspace > My Apps tab. -### [How to launch LS-Dyna](#launch) { #launch } +### How to launch LS-Dyna { #launch } Examples in this guide: @@ -49,7 +49,7 @@ Examples in this guide:
  • Launching batch of jobs via Command Line Interface.
    • -#### [Launching LS-Pre/Post](#launch-prepost) { #launch-prepost } +#### Launching LS-Pre/Post { #launch-prepost }
    • Select the LS-Pre/Post app from the drop-drown menu at (Workspace > Simulation > LS-DYNA):
      • @@ -86,7 +86,7 @@ Examples in this guide: ![](./imgs/ls-dyna-5.png) -#### [Launching a single job via DesignSafe web portal](#launch-singlejob) { #launch-singlejob } +#### Launching a single job via DesignSafe web portal { #launch-singlejob }
      • Select LS-DYNA (Serial) from the LS-DYNA app in (Workspace > My Apps > LS-DYNA):
        • @@ -121,7 +121,7 @@ Examples in this guide:
      -#### [Launching batch of jobs via DesignSafe web portal](#launch-batch) { #launch-batch } +#### Launching batch of jobs via DesignSafe web portal { #launch-batch }
      • Select LS-DYNA (Parallel) from the LS-DYNA app in (Workspace > My Apps > LS-DYNA):
        • @@ -162,7 +162,7 @@ Examples in this guide:
      -#### [Launching batch of jobs via Command Line Interface (CLI)](#launch-batchcli) { #launch-batchcli } +#### Launching batch of jobs via Command Line Interface (CLI) { #launch-batchcli }
      • Connect to Frontera using SSH Client. See TACC's [Data Transfer & Management Guide](https://docs.tacc.utexas.edu/hpc/frontera/): diff --git a/user-guide/docs/tools/simulation/opensees.md b/user-guide/docs/tools/simulation/opensees.md index 4d9ad4d..7bb4045 100644 --- a/user-guide/docs/tools/simulation/opensees.md +++ b/user-guide/docs/tools/simulation/opensees.md @@ -16,7 +16,7 @@ OpenSeesSP is an OpenSees interpreter intended for high performance computers fo OpenSeesMP is an OpenSees interpreter intended for high performance computers for performing finite element simulations with parameteric studies and very large models on parallel machines. OpenSeesMP requires understanding of parallel processing and the capabilities to write parallel scripts. -### [How to Submit an OpenSees Job in the Workspace](#submit) { #submit } +### How to Submit an OpenSees Job in the Workspace { #submit } 1. Select the OpenSees application from the simulation tab in the workspace. @@ -55,7 +55,7 @@ OpenSeesMP is an OpenSees interpreter intended for high performance computers fo -### [DesignSafe Tutorial: OpenSees & DesignSafe, October 31, 2018](#tutorial) { #tutorial } +### DesignSafe Tutorial: OpenSees & DesignSafe, October 31, 2018 { #tutorial }