diff --git a/downcase-ids/.buildinfo b/downcase-ids/.buildinfo new file mode 100644 index 000000000..30bff2ac9 --- /dev/null +++ b/downcase-ids/.buildinfo @@ -0,0 +1,4 @@ +# Sphinx build info version 1 +# This file hashes the configuration used when building these files. When it is not found, a full rebuild will be done. +config: f6a51926a732dce8963d38833c37ce9a +tags: 645f666f9bcd5a90fca523b33c5a78b7 diff --git a/downcase-ids/.nojekyll b/downcase-ids/.nojekyll new file mode 100644 index 000000000..e69de29bb diff --git a/downcase-ids/_images/2.0_files_app.png b/downcase-ids/_images/2.0_files_app.png new file mode 100644 index 000000000..03f4813dc Binary files /dev/null and b/downcase-ids/_images/2.0_files_app.png differ diff --git a/downcase-ids/_images/app-dev-tutorial-ps-to-quota-published.png b/downcase-ids/_images/app-dev-tutorial-ps-to-quota-published.png new file mode 100644 index 000000000..d534f9672 Binary files /dev/null and b/downcase-ids/_images/app-dev-tutorial-ps-to-quota-published.png differ diff --git a/downcase-ids/_images/app-links-no-group.png b/downcase-ids/_images/app-links-no-group.png new file mode 100644 index 000000000..32a1e4472 Binary files /dev/null and b/downcase-ids/_images/app-links-no-group.png differ diff --git a/downcase-ids/_images/app-sharing-1.png b/downcase-ids/_images/app-sharing-1.png new file mode 100644 index 000000000..005033124 Binary files /dev/null and b/downcase-ids/_images/app-sharing-1.png differ diff --git a/downcase-ids/_images/app-sharing-2.png b/downcase-ids/_images/app-sharing-2.png new file mode 100644 index 000000000..8beb0886f Binary files /dev/null and b/downcase-ids/_images/app-sharing-2.png differ diff --git a/downcase-ids/_images/app-sharing-3.png b/downcase-ids/_images/app-sharing-3.png new file mode 100644 index 000000000..0e6f76aa0 Binary files /dev/null and b/downcase-ids/_images/app-sharing-3.png differ diff --git a/downcase-ids/_images/app-sharing-4.png b/downcase-ids/_images/app-sharing-4.png new file mode 100644 index 000000000..f56279e6d Binary files /dev/null and b/downcase-ids/_images/app-sharing-4.png differ diff --git a/downcase-ids/_images/app-sharing-5.png b/downcase-ids/_images/app-sharing-5.png new file mode 100644 index 000000000..a809ebf8d Binary files /dev/null and b/downcase-ids/_images/app-sharing-5.png differ diff --git a/downcase-ids/_images/app-sharing-6.png b/downcase-ids/_images/app-sharing-6.png new file mode 100644 index 000000000..6dfaebdd6 Binary files /dev/null and b/downcase-ids/_images/app-sharing-6.png differ diff --git a/downcase-ids/_images/app-sharing-mode-after.png b/downcase-ids/_images/app-sharing-mode-after.png new file mode 100644 index 000000000..6d9280b9c Binary files /dev/null and b/downcase-ids/_images/app-sharing-mode-after.png differ diff --git a/downcase-ids/_images/app-sharing-mode-before.png b/downcase-ids/_images/app-sharing-mode-before.png new file mode 100644 index 000000000..0067619c3 Binary files /dev/null and b/downcase-ids/_images/app-sharing-mode-before.png differ diff --git a/downcase-ids/_images/app-sharing-permissions-after.png b/downcase-ids/_images/app-sharing-permissions-after.png new file mode 100644 index 000000000..6a90a99bd Binary files /dev/null and b/downcase-ids/_images/app-sharing-permissions-after.png differ diff --git a/downcase-ids/_images/app-sharing-permissions-before.png b/downcase-ids/_images/app-sharing-permissions-before.png new file mode 100644 index 000000000..9800f07f8 Binary files /dev/null and b/downcase-ids/_images/app-sharing-permissions-before.png differ diff --git a/downcase-ids/_images/bc-card-w-hostlink.png b/downcase-ids/_images/bc-card-w-hostlink.png new file mode 100644 index 000000000..f7b55107b Binary files /dev/null and b/downcase-ids/_images/bc-card-w-hostlink.png differ diff --git a/downcase-ids/_images/cancel_session.png b/downcase-ids/_images/cancel_session.png new file mode 100644 index 000000000..4413b686d Binary files /dev/null and b/downcase-ids/_images/cancel_session.png differ diff --git a/downcase-ids/_images/custom-pages-documentation.png b/downcase-ids/_images/custom-pages-documentation.png new file mode 100644 index 000000000..55242d928 Binary files /dev/null and b/downcase-ids/_images/custom-pages-documentation.png differ diff --git a/downcase-ids/_images/custom_pinned_apps.png b/downcase-ids/_images/custom_pinned_apps.png new file mode 100644 index 000000000..98c25af98 Binary files /dev/null and b/downcase-ids/_images/custom_pinned_apps.png differ diff --git a/downcase-ids/_images/customization_homedirmissing_default.png b/downcase-ids/_images/customization_homedirmissing_default.png new file mode 100644 index 000000000..347643bbc Binary files /dev/null and b/downcase-ids/_images/customization_homedirmissing_default.png differ diff --git a/downcase-ids/_images/customization_homedirmissing_pammkdir.png b/downcase-ids/_images/customization_homedirmissing_pammkdir.png new file mode 100644 index 000000000..b5a497529 Binary files /dev/null and b/downcase-ids/_images/customization_homedirmissing_pammkdir.png differ diff --git a/downcase-ids/_images/customization_xdmod.png b/downcase-ids/_images/customization_xdmod.png new file mode 100644 index 000000000..aedc5be7e Binary files /dev/null and b/downcase-ids/_images/customization_xdmod.png differ diff --git a/downcase-ids/_images/customization_xdmod_jobcomposer_warning_1.png b/downcase-ids/_images/customization_xdmod_jobcomposer_warning_1.png new file mode 100644 index 000000000..2284caed2 Binary files /dev/null and b/downcase-ids/_images/customization_xdmod_jobcomposer_warning_1.png differ diff --git a/downcase-ids/_images/customization_xdmod_jobcomposer_warning_2.png b/downcase-ids/_images/customization_xdmod_jobcomposer_warning_2.png new file mode 100644 index 000000000..3c2e9d8ed Binary files /dev/null and b/downcase-ids/_images/customization_xdmod_jobcomposer_warning_2.png differ diff --git a/downcase-ids/_images/dashboard-announcement.png b/downcase-ids/_images/dashboard-announcement.png new file mode 100644 index 000000000..b48c7e1bc Binary files /dev/null and b/downcase-ids/_images/dashboard-announcement.png differ diff --git a/downcase-ids/_images/dashboard_branding_logo_and_colors.png b/downcase-ids/_images/dashboard_branding_logo_and_colors.png new file mode 100644 index 000000000..dee874a61 Binary files /dev/null and b/downcase-ids/_images/dashboard_branding_logo_and_colors.png differ diff --git a/downcase-ids/_images/dashboard_motd.png b/downcase-ids/_images/dashboard_motd.png new file mode 100644 index 000000000..c58c74570 Binary files /dev/null and b/downcase-ids/_images/dashboard_motd.png differ diff --git a/downcase-ids/_images/dashboard_navbar_branding_bluered.png b/downcase-ids/_images/dashboard_navbar_branding_bluered.png new file mode 100644 index 000000000..345cb2146 Binary files /dev/null and b/downcase-ids/_images/dashboard_navbar_branding_bluered.png differ diff --git a/downcase-ids/_images/def-apps-menu.png b/downcase-ids/_images/def-apps-menu.png new file mode 100644 index 000000000..dad526942 Binary files /dev/null and b/downcase-ids/_images/def-apps-menu.png differ diff --git a/downcase-ids/_images/def-category-menu.png b/downcase-ids/_images/def-category-menu.png new file mode 100644 index 000000000..95e2a332e Binary files /dev/null and b/downcase-ids/_images/def-category-menu.png differ diff --git a/downcase-ids/_images/def-full-menu.png b/downcase-ids/_images/def-full-menu.png new file mode 100644 index 000000000..a77b1f1d8 Binary files /dev/null and b/downcase-ids/_images/def-full-menu.png differ diff --git a/downcase-ids/_images/def-navigation-link.png b/downcase-ids/_images/def-navigation-link.png new file mode 100644 index 000000000..a64979fc9 Binary files /dev/null and b/downcase-ids/_images/def-navigation-link.png differ diff --git a/downcase-ids/_images/def-page-links.png b/downcase-ids/_images/def-page-links.png new file mode 100644 index 000000000..46840c58f Binary files /dev/null and b/downcase-ids/_images/def-page-links.png differ diff --git a/downcase-ids/_images/def-profiles-links.png b/downcase-ids/_images/def-profiles-links.png new file mode 100644 index 000000000..093eccd1b Binary files /dev/null and b/downcase-ids/_images/def-profiles-links.png differ diff --git a/downcase-ids/_images/def-templates-menu.png b/downcase-ids/_images/def-templates-menu.png new file mode 100644 index 000000000..65132e207 Binary files /dev/null and b/downcase-ids/_images/def-templates-menu.png differ diff --git a/downcase-ids/_images/develop_dashboard_grey_background.png b/downcase-ids/_images/develop_dashboard_grey_background.png new file mode 100644 index 000000000..8b5d636fd Binary files /dev/null and b/downcase-ids/_images/develop_dashboard_grey_background.png differ diff --git a/downcase-ids/_images/develop_dashboard_pinned_apps.png b/downcase-ids/_images/develop_dashboard_pinned_apps.png new file mode 100644 index 000000000..47afcee54 Binary files /dev/null and b/downcase-ids/_images/develop_dashboard_pinned_apps.png differ diff --git a/downcase-ids/_images/files-open-in-terminal.png b/downcase-ids/_images/files-open-in-terminal.png new file mode 100644 index 000000000..41d01fc3a Binary files /dev/null and b/downcase-ids/_images/files-open-in-terminal.png differ diff --git a/downcase-ids/_images/files_menu_shortcuts_osc.png b/downcase-ids/_images/files_menu_shortcuts_osc.png new file mode 100644 index 000000000..afe811c70 Binary files /dev/null and b/downcase-ids/_images/files_menu_shortcuts_osc.png differ diff --git a/downcase-ids/_images/flow_access_dashboard.png b/downcase-ids/_images/flow_access_dashboard.png new file mode 100644 index 000000000..009a8d030 Binary files /dev/null and b/downcase-ids/_images/flow_access_dashboard.png differ diff --git a/downcase-ids/_images/flow_access_passenger_app.png b/downcase-ids/_images/flow_access_passenger_app.png new file mode 100644 index 000000000..cc6a3f40d Binary files /dev/null and b/downcase-ids/_images/flow_access_passenger_app.png differ diff --git a/downcase-ids/_images/flow_access_usr_app_via_app_sharing.png b/downcase-ids/_images/flow_access_usr_app_via_app_sharing.png new file mode 100644 index 000000000..cb7639241 Binary files /dev/null and b/downcase-ids/_images/flow_access_usr_app_via_app_sharing.png differ diff --git a/downcase-ids/_images/flow_authentication.png b/downcase-ids/_images/flow_authentication.png new file mode 100644 index 000000000..70974976d Binary files /dev/null and b/downcase-ids/_images/flow_authentication.png differ diff --git a/downcase-ids/_images/flow_linux_host_adapter.png b/downcase-ids/_images/flow_linux_host_adapter.png new file mode 100644 index 000000000..7f747e169 Binary files /dev/null and b/downcase-ids/_images/flow_linux_host_adapter.png differ diff --git a/downcase-ids/_images/flow_rstudio_job.png b/downcase-ids/_images/flow_rstudio_job.png new file mode 100644 index 000000000..ecc0bde02 Binary files /dev/null and b/downcase-ids/_images/flow_rstudio_job.png differ diff --git a/downcase-ids/_images/flow_start_shell_session.png b/downcase-ids/_images/flow_start_shell_session.png new file mode 100644 index 000000000..0c9139e18 Binary files /dev/null and b/downcase-ids/_images/flow_start_shell_session.png differ diff --git a/downcase-ids/_images/flow_vnc_desktop_job.png b/downcase-ids/_images/flow_vnc_desktop_job.png new file mode 100644 index 000000000..ea9e79533 Binary files /dev/null and b/downcase-ids/_images/flow_vnc_desktop_job.png differ diff --git a/downcase-ids/_images/form-display-attribute.png b/downcase-ids/_images/form-display-attribute.png new file mode 100644 index 000000000..73cba5e07 Binary files /dev/null and b/downcase-ids/_images/form-display-attribute.png differ diff --git a/downcase-ids/_images/grouped_pinned_apps.png b/downcase-ids/_images/grouped_pinned_apps.png new file mode 100644 index 000000000..64db436e2 Binary files /dev/null and b/downcase-ids/_images/grouped_pinned_apps.png differ diff --git a/downcase-ids/_images/help_menu_links.png b/downcase-ids/_images/help_menu_links.png new file mode 100644 index 000000000..99d4f275c Binary files /dev/null and b/downcase-ids/_images/help_menu_links.png differ diff --git a/downcase-ids/_images/interactive-app-compression-quality.png b/downcase-ids/_images/interactive-app-compression-quality.png new file mode 100644 index 000000000..cf88cb15f Binary files /dev/null and b/downcase-ids/_images/interactive-app-compression-quality.png differ diff --git a/downcase-ids/_images/interactive-app-shell-link.png b/downcase-ids/_images/interactive-app-shell-link.png new file mode 100644 index 000000000..32f7b5cf7 Binary files /dev/null and b/downcase-ids/_images/interactive-app-shell-link.png differ diff --git a/downcase-ids/_images/interactive-apps-menu.png b/downcase-ids/_images/interactive-apps-menu.png new file mode 100644 index 000000000..b01374d39 Binary files /dev/null and b/downcase-ids/_images/interactive-apps-menu.png differ diff --git a/downcase-ids/_images/linux_host_undetermined.png b/downcase-ids/_images/linux_host_undetermined.png new file mode 100644 index 000000000..d4bbd86f7 Binary files /dev/null and b/downcase-ids/_images/linux_host_undetermined.png differ diff --git a/downcase-ids/_images/main-navigation-menu.png b/downcase-ids/_images/main-navigation-menu.png new file mode 100644 index 000000000..dc4ffc510 Binary files /dev/null and b/downcase-ids/_images/main-navigation-menu.png differ diff --git a/downcase-ids/_images/ood_container_view.png b/downcase-ids/_images/ood_container_view.png new file mode 100644 index 000000000..269247b94 Binary files /dev/null and b/downcase-ids/_images/ood_container_view.png differ diff --git a/downcase-ids/_images/ood_overview.png b/downcase-ids/_images/ood_overview.png new file mode 100644 index 000000000..83b42ad11 Binary files /dev/null and b/downcase-ids/_images/ood_overview.png differ diff --git a/downcase-ids/_images/ood_system_view.png b/downcase-ids/_images/ood_system_view.png new file mode 100644 index 000000000..eadf6492d Binary files /dev/null and b/downcase-ids/_images/ood_system_view.png differ diff --git a/downcase-ids/_images/pinned_apps.png b/downcase-ids/_images/pinned_apps.png new file mode 100644 index 000000000..704556b8d Binary files /dev/null and b/downcase-ids/_images/pinned_apps.png differ diff --git a/downcase-ids/_images/plantuml-6a8fdab5b21c9c0c998d0a21b362a10878872ac3.png b/downcase-ids/_images/plantuml-6a8fdab5b21c9c0c998d0a21b362a10878872ac3.png new file mode 100644 index 000000000..7e51298ab Binary files /dev/null and b/downcase-ids/_images/plantuml-6a8fdab5b21c9c0c998d0a21b362a10878872ac3.png differ diff --git a/downcase-ids/_images/plantuml-be1e1017662db91033e2642c95916bc1e948d557.png b/downcase-ids/_images/plantuml-be1e1017662db91033e2642c95916bc1e948d557.png new file mode 100644 index 000000000..ce7b42d2c Binary files /dev/null and b/downcase-ids/_images/plantuml-be1e1017662db91033e2642c95916bc1e948d557.png differ diff --git a/downcase-ids/_images/support_ticket_custom_form.png b/downcase-ids/_images/support_ticket_custom_form.png new file mode 100644 index 000000000..9c15f487f Binary files /dev/null and b/downcase-ids/_images/support_ticket_custom_form.png differ diff --git a/downcase-ids/_images/support_ticket_form.png b/downcase-ids/_images/support_ticket_form.png new file mode 100644 index 000000000..19ace60fa Binary files /dev/null and b/downcase-ids/_images/support_ticket_form.png differ diff --git a/downcase-ids/_images/support_ticket_menu.png b/downcase-ids/_images/support_ticket_menu.png new file mode 100644 index 000000000..0cd832a01 Binary files /dev/null and b/downcase-ids/_images/support_ticket_menu.png differ diff --git a/downcase-ids/_sources/architecture.rst.txt b/downcase-ids/_sources/architecture.rst.txt new file mode 100644 index 000000000..a45ab2414 --- /dev/null +++ b/downcase-ids/_sources/architecture.rst.txt @@ -0,0 +1,115 @@ +.. _architecture: + +Architecture +============ + +Below are some diagrams of OnDemand's architecture: + +#. Overview is a high level visual generated from Powerpoint. +#. System context and Container context diagrams below follow the `C4 `_. + model for software diagrams, are more technically detailed and are built using draw.io +#. Request flow diagram is a sequence diagram built using plantuml. + +Overview +-------- + + +.. figure:: /architecture/ood_overview.png + +#. Apache is the server front end, running as the Apache user, and accepting all requests from users and serves four primary functions: + + #. Authenticates user. + #. Starts Per-User NGINX processes (PUNs). + #. Reverse proxies each user to her PUN via Unix domain sockets. + #. Reverse proxies to interactive apps running on compute nodes (RStudio, Jupyter, VNC desktop) via TCP sockets. + +#. The Per-User NGINX serves web apps in Ruby and NodeJS and is how users submit jobs and start interactive apps. + + +System context +----------------------- + +Users use OnDemand to interact with their HPC resources through a web browser. + +.. figure:: /architecture/ood_system_view.png + +All the gray components are specific to a given site and outside the OnDemand +system. + +Container context +----------------------- + +.. tip:: + + In the C4 nomenclature, 'containers' are one level below the system context. This is + not to be confused with Linux containers via cgroups and namespaces (i.e. Docker or + Singularity or `OCI containers `_). + +The Front-end proxy is the only component that is shared with all clients. +The Front-end proxy will create Per User Nginx (PUN) processes (light blue boxes labeled "Per User Instance"). + +.. figure:: /architecture/ood_container_view.png + +* Everything contained in the dotted line is a part of the OnDemand system (see blue box in System context diagram). +* Everything outside of it in gray is site specific components. +* The "Per User Instance" light blue boxes are replicated for every user accessing the system. + +Request Flow +------------- + +This is the request flow through the OnDemand system. A user initiates a +request through a browser and this illustrates how that request propagates +through the system to a particular application (including the dashboard). + +.. uml:: architecture/request-flow.uml + +Other Request Flow Diagrams +---------------------------- + +================ +Dashboard Access +================ + +.. figure:: /app-flow-diagrams/flow_access_dashboard.png + +============= +Passenger App +============= + +.. figure:: /app-flow-diagrams/flow_access_passenger_app.png + +================ +User App Sharing +================ + +.. figure:: /app-flow-diagrams/flow_access_usr_app_via_app_sharing.png + +============== +Authentication +============== + +.. figure:: /app-flow-diagrams/flow_authentication.png + +================== +Linux Host Adapter +================== + +.. figure:: /app-flow-diagrams/flow_linux_host_adapter.png + +=========== +Rstudio Job +=========== + +.. figure:: /app-flow-diagrams/flow_rstudio_job.png + +============= +Shell Session +============= + +.. figure:: /app-flow-diagrams/flow_start_shell_session.png + +=============== +VNC Desktop Job +=============== + +.. figure:: /app-flow-diagrams/flow_vnc_desktop_job.png \ No newline at end of file diff --git a/downcase-ids/_sources/authentication.rst.txt b/downcase-ids/_sources/authentication.rst.txt new file mode 100644 index 000000000..e62b44fcb --- /dev/null +++ b/downcase-ids/_sources/authentication.rst.txt @@ -0,0 +1,44 @@ +.. _authentication: + +Authentication +============== + +Open OnDemand supports most authentication modules that work with Apache HTTP +Server version 2.4. The following :ref:`authentication-overview` section +provides an introduction to setting up these generic authentication modules +with an Open OnDemand installation. Tutorials will also be provided with the +focus on setting up some of the more common authentication modules (e.g., +OpenID Connect with KeyCloak). + +After installing Open OnDemand you **must** add authentication of some kind +to generate the correct Apache configuration. When no authentication is +supplied Apache will only serve a static page that directs you here. + +No Open OnDemand functionality is available without authentication. + +.. note:: + + If you managed to install an Apache authentication module at your center + that currently does not have a tutorial listed below we would greatly + appreciate it if you could take the time to contribute a detailed + walkthrough. + + +.. tip:: + + :ref:`Dex ` is a very good starting option if you can connect + to LDAP or Active Directory and not an institutional Single Sign-On service. + +.. toctree:: + :maxdepth: 2 + + authentication/overview + authentication/oidc + authentication/dex + authentication/shibboleth + authentication/cas + authentication/tutorial-oidc-keycloak-rhel7 + authentication/duo-2fa-with-keycloak + authentication/adfs-with-auth-mellon + authentication/nsf-access + authentication/insecure diff --git a/downcase-ids/_sources/authentication/adfs-with-auth-mellon.rst.txt b/downcase-ids/_sources/authentication/adfs-with-auth-mellon.rst.txt new file mode 100644 index 000000000..0e91bc045 --- /dev/null +++ b/downcase-ids/_sources/authentication/adfs-with-auth-mellon.rst.txt @@ -0,0 +1,113 @@ +.. _authentication-adfs-with-auth-mellon: + +SAML Authentication with Active Directory Federated Services (ADFS) and mod_auth_mellon +======================================================================================== + +The following details how to use ADFS infrastructure via SAML authentication to authenticate to an OpenOnDemand deployment. + +Prepare the Host +-------------------------------------------------- +Before beginning, retrieve the following information from the ADFS administrator: + +#. The SAML 2.0 service URL (e.g., https://adfs.organization.com/ADFS/ls) +#. The IdP metadata URL (e.g., https://adfs.organization.com/ADFS/metadata.xml) +#. Ensure SSL is properly configured and any organizational certificate authorities are properly integrated into the host's trust store, see :ref:`add-ssl` + +Install mod_auth_mellon +-------------------------------------------------- + +#. Ensure Software Collections is enabled on the system +#. Install the mod_auth_mellon module: + +.. tabs:: + + .. tab:: EL7 + + .. code-block:: shell + + yum install httpd24-mod_auth_mellon httpd24-mod_ssl + + .. tab:: EL8+ + + .. code-block:: shell + + yum install mod_auth_mellon mod_ssl + + .. tab:: Ubuntu + + .. code-block:: shell + + apt install libapache2-mod-auth-mellon + +Configure mod_auth_mellon +-------------------------------------------------- + +Note that this configuration assumes that SAML has been configured such that the returned NameID directly maps to a Unix user on the OOD host. For more information, see https://jdennis.fedorapeople.org/doc/mellon-user-guide/mellon_user_guide.html + +#. Download the IDP metadata file + + .. code-block:: shell + + cd /etc/httpd/mellon/ + wget https://adfs.organization.com/ADFS/metadata.xml -O idpmetadata.xml + +#. Generate the mellon metadata + + .. code-block:: shell + + export mellon_endpoint="https://$(hostname)/mellon" + /usr/libexec/mod_auth_mellon/mellon_create_metadata.sh "${mellon_endpoint}/metadata" "${mellon_endpoint}" + mv *.cert ./mellon.cert + mv *.key ./mellon.key + mv *.xml ./mellon_metadata.xml + +#. Create a mellon configuration file + + .. code-block:: shell + + vi /etc/httpd/conf.d/00-mellon.conf + +#. Add the following to the ``00-mellon.conf`` file + + .. code-block:: xml + + + MellonSPPrivateKeyFile /etc/httpd/mellon/mellon.key + MellonSPCertFile /etc/httpd/mellon/mellon.cert + MellonSPMetadataFile /etc/httpd/mellon/mellon_metadata.xml + MellonIdPMetadataFile /etc/httpd/mellon/idpmetadata.xml + + MellonEndpointPath /mellon + MellonEnable "auth" + + +#. Convert the key and cert files into pfx format + + .. code-block:: shell + + openssl pkcs12 -export -inkey /etc/httpd/mellon/mellon.key -in /etc/httpd/mellon/mellon.cert -out /etc/httpd/mellon/mellon.pfx + +#. Provide the ``mellon.pfx`` and ``mellon_metadata.xml`` files to your ADFS administrator. The files can then be imported into the ADFS system. + +Configure OOD +-------------------------------------------------- + +#. Edit the ``ood_portal.yml`` file to include the following: + + .. code-block:: yaml + + # /etc/ood/config/ood_portal.yml + --- + # ... + # Your other custom configuration options... + # ... + + auth: + - 'AuthType Mellon' + - 'Require valid-user' + +#. Restart the HTTPD + + .. code-block:: shell + + systemctl restart httpd diff --git a/downcase-ids/_sources/authentication/cas.rst.txt b/downcase-ids/_sources/authentication/cas.rst.txt new file mode 100644 index 000000000..5354efab6 --- /dev/null +++ b/downcase-ids/_sources/authentication/cas.rst.txt @@ -0,0 +1,13 @@ +.. _authentication-cas: + +CAS +--- + +Several HPC centers running OnDemand have successfully configured authentication using Central Authentication Service (CAS). + +See `this Discourse `__ topic regarding several different examples configuring CAS authentication with OnDemand. + +Related links: + +- `mod_auth_cas `__ +- `CAS project website `__ diff --git a/downcase-ids/_sources/authentication/dex.rst.txt b/downcase-ids/_sources/authentication/dex.rst.txt new file mode 100644 index 000000000..371208195 --- /dev/null +++ b/downcase-ids/_sources/authentication/dex.rst.txt @@ -0,0 +1,318 @@ +.. _authentication-dex: + +OpenID Connect with Dex +======================= + +`Dex`_ is a lightweight OpenID Connect authentication provider written in Go, and is the default authentication mechanism shipped with Open OnDemand. + +Installing OnDemand Dex package +------------------------------- + +First the OnDemand yum repos must be enabled, see :ref:`install-software`. + +Install the ``ondemand-dex`` package: + +.. tabs:: + + .. tab:: yum/dnf + + .. code-block:: sh + + sudo yum install ondemand-dex + + + .. tab:: apt + + .. code-block:: sh + + sudo apt-get install ondemand-dex + +Installing OnDemand Dex from source +----------------------------------- + +Requirements: + +- Go version 1.16.x with the ``go`` binary in ``PATH`` +- Git +- Make + +Build and install the ondemand-dex binary: + + .. code-block:: sh + + GOPATH=$(go env GOPATH) + go get github.com/dexidp/dex + cd $GOPATH/src/github.com/dexidp/dex + make build + sudo install -m 0755 bin/dex /usr/sbin/ondemand-dex + +Add the ``ondemand-dex`` user and group: + + .. code-block:: sh + + sudo groupadd -r ondemand-dex + sudo useradd -r -d /var/lib/ondemand-dex -g ondemand-dex -s /sbin/nologin -c "OnDemand Dex" ondemand-dex + +Get ``ondemand-dex`` repo and install web files and systemd unit file + + .. code-block:: sh + + cd /tmp + git clone https://github.com/OSC/ondemand-dex + sudo mkdir /usr/share/ondemand-dex + sudo cp -R ondemand-dex/web /usr/share/ondemand-dex/web + sudo install -m 0644 ondemand-dex/examples/ondemand-dex.service /etc/systemd/system/ondemand-dex.service + +Configuring OnDemand Dex +------------------------ + +OnDemand Dex is configured by modifying the Open OnDemand Portal :ref:`ood-portal-generator-configuration` file :file:`/etc/ood/config/ood_portal.yml`. + +The default location for Dex configurations is :file:`/etc/ood/dex/config.yaml`. + +When changes are needed for OnDemand Dex :ref:`restart-apache` and then restart Dex with: + + .. code-block:: sh + + sudo systemctl restart ondemand-dex + +.. warning:: + + If OnDemand is configured to use SSL and SSL certificates are not configured in Dex, + the default behavior is for Dex to use copies of the OnDemand certificates for SSL. + This means when the OnDemand certificates are updated it's necessary to run + ``update_ood_portal`` to make new copies of the certificates and restart ``ondemand-dex``. + +Managing the OnDemand Dex service +--------------------------------- + +The service for OnDemand Dex is ``ondemand-dex``: + + .. code-block:: sh + + sudo systemctl enable ondemand-dex.service + sudo systemctl start ondemand-dex.service + +OnDemand Dex behind Apache reverse proxy +---------------------------------------- + +By default Dex sits behing Apache and is accessed via a reverse proxy. +OnDemand Dex behind the reverse proxy logic will force Dex to listen only on ``localhost`` and only +via HTTP. + +To disable Dex behind a reverse proxy set ``dex_uri`` to ``false`` or ``null`` + + .. code-block:: yaml + + dex_uri: false + +When Dex is not behind a reverse proxy firewall adjustments may be needed. +See :ref:`Dex Firewall ` for instructions on opening Dex ports through your firewall. + +Dex Firewall +------------ + +.. _dex-firewall: + +.. note:: + + The Dex firewall changes are only needed when ``dex_uri`` is set to ``false`` or ``null``. + +By default when using SSL, Dex will use port ``5554`` for the communication between OnDemand and Dex as well as login interactions with users accessing OnDemand. The port used for non-SSL is ``5556``. The port being used by Dex must be externally accessible. + +Firewalld example: + .. code-block:: sh + + $ sudo firewall-cmd --zone=public --add-port=5554/tcp --permanent + $ sudo firewall-cmd --reload + +Iptables example: + .. code-block:: sh + + $ sudo iptables -I INPUT -p tcp -m tcp --dport 5554 -j ACCEPT + $ sudo iptables-save > /etc/sysconfig/iptables + +Configuring OnDemand Dex for LDAP +--------------------------------- + +.. _dex-ldap: + + +Requirements: + +- an LDAP server preferably with SSL support (``openldap.my_center.edu:636``) + +The following is an example configuration using OpenLDAP. + + .. code-block:: yaml + :emphasize-lines: 5- + + # /etc/ood/config/ood_portal.yml + --- + + # ... + dex: + connectors: + - type: ldap + id: ldap + name: LDAP + config: + host: openldap.my_center.edu:636 + insecureSkipVerify: false + bindDN: cn=admin,dc=example,dc=org + bindPW: admin + userSearch: + baseDN: ou=People,dc=example,dc=org + filter: "(objectClass=posixAccount)" + username: uid + idAttr: uid + emailAttr: mail + nameAttr: gecos + preferredUsernameAttr: uid + groupSearch: + baseDN: ou=Groups,dc=example,dc=org + filter: "(objectClass=posixGroup)" + userMatchers: + - userAttr: DN + groupAttr: member + nameAttr: cn + .. note:: + + For documentation on Dex LDAP configuration please see the `Dex LDAP docs`_ + + .. note:: + + If you supply a ``bindPW`` in this file it's recommended to change the file permissions on :file:`/etc/ood/config/ood_portal.yml` to be ``0600`` make the file only readable by ``root``: + + .. code-block:: sh + + sudo chown root:root /etc/ood/config/ood_portal.yml + sudo chmod 0600 /etc/ood/config/ood_portal.yml + +Customizing OnDemand Dex +------------------------ + +The theme for Dex can be customized to be site-specific, see :ref:`customize_dex_theme`. + +OnDemand Dex configuration reference +------------------------------------ + +.. _dex-configuration: + +The OnDemand Dex configuration works by attempting to expose all Dex configuration options as keys nested under the ``dex`` key in :file:`/etc/ood/config/ood_portal.yml`. + +The following reference is for :file:`/etc/ood/config/ood_portal.yml` values set under the ``dex`` key. + +.. describe:: ssl (Boolean, null) + + Boolean to set if SSL is used, is ``true`` if OnDemand is configured for SSL, otherwise this defaults to ``false``. + This value is used to determine which listen ports to use for Dex as well as OIDC configurations for OnDemand + +.. describe:: http_port (String, Integer) + + The HTTP port used by Dex, default is ``5556``. + Used to define ``web -> http`` in the Dex configuration as well as OIDC configurations + +.. describe:: https_port (String, Integer) + + The HTTPS port used by Dex, default is ``5554``. This value is only set if SSL is enabled. + Used to define ``web -> https`` in the Dex configuration as well as OIDC configurations + +.. describe:: tls_cert (String, null) + + The path to TLS cert used by Dex. + The default is to use the SSL certificate for OnDemand if OnDemand is configured with SSL. + Used to define ``web -> tlsCert`` in the Dex configuration. + If using the OnDemand certificate, a copy is made to ``/etc/ood/dex``. + The ``ondemand-dex`` user must be able to read this file if configured. + +.. describe:: tls_key (String, null) + + The path to TLS key used by Dex. + The default is to use the SSL key for OnDemand if OnDemand is configured with SSL. + Used to define ``web -> tlsKey`` in the Dex configuration. + If using the OnDemand key, a copy is made to ``/etc/ood/dex``. + The ``ondemand-dex`` user must be able to read this file if configured. + +.. describe:: storage_file (String) + + The path to the Dex SQLite storage file. Defaults to ``/etc/ood/dex/dex.db``. + Used to define ``storage -> config -> file`` in the Dex configuration. + +.. describe:: client_id (String) + + The client ID used for the OnDemand OIDC client. + The default is to use the ``servername`` for OnDemand, and if that is not defined the host's FQDN is used. + Sets ``staticClients[0] -> id`` in the Dex configuration as well as OnDemand OIDC configurations. + +.. describe:: client_secret (String) + + The client secret used for the OnDemand OIDC client. + The default is a randomly generated secret stored in ``/etc/ood/dex/ondemand.secret``. + The value for this configuration can either be the secret string or path to file storing the secret. + If using a file, the ``ondemand-dex`` user must be able to read the file. + Sets ``staticClients[0] -> secret`` in the Dex configuration as well as OnDemand OIDC configurations. + +.. describe:: client_redirect_uris (Array) + + Additional OIDC client URIs to authorize for the OnDemand client. + The values provided for this are merged with the default redirect URI generated for OnDemand. + Sets ``staticClients[0] -> redirectURIs`` in the Dex configuration as well as OnDemand OIDC configurations. + +.. describe:: client_name (String) + + The default OIDC client name for Dex. Defaults to ``OnDemand``. + Sets ``staticClients[0] -> name`` in the Dex configuration. + +.. describe:: connectors (Array) + + This defines the external connectors used to authenticate users with Dex. + If this value is not provided the default behavior is to set a static password of ``password`` for user ``ood@localhost``. + This value is passed directly to the Dex configuration for ``connectors``. + For an example of LDAP configuration see :ref:`Configuring OnDemand Dex for LDAP `. + +.. describe:: frontend (Hash) + + This defines various changes for the themes and frontend look of Dex. + The value provided is passed directly to the Dex configuration for ``frontend``. + If ``dir`` key is not set the default of ``/usr/share/ondemand-dex/web`` is used. + If ``theme`` key is not set the default of ``ondemand`` is used. + + Default + + .. code-block:: yaml + + frontend: + dir: "/usr/share/ondemand-dex/web" + theme: "ondemand" + +.. describe:: grpc (Hash) + + The configuration for Dex's gRPC API. + Value is passed directly to the Dex configuration + + Example: + + .. code-block:: yaml + + grpc: + addr: "127.0.0.1:5557" + tlsCert: "/etc/ood/dex/grpc-server.crt" + tlsKey: "/etc/ood/dex/grpc-server.key" + tlsClientCA: "/etc/ood/dex/grpc-ca.crt" + +.. describe:: expiry (Hash) + + The configuration for Dex's expirations. + Value is passed directly to the Dex configuration + + Example: + + .. code-block:: yaml + + expiry: + signingKeys: "6h" + idTokens: "24h" + +.. _dex: https://github.com/dexidp/dex +.. _dex ldap docs: https://dexidp.io/docs/connectors/ldap/ diff --git a/downcase-ids/_sources/authentication/duo-2fa-with-keycloak.rst.txt b/downcase-ids/_sources/authentication/duo-2fa-with-keycloak.rst.txt new file mode 100644 index 000000000..3a99aeba6 --- /dev/null +++ b/downcase-ids/_sources/authentication/duo-2fa-with-keycloak.rst.txt @@ -0,0 +1,66 @@ +.. _authentication-duo-2fa-with-keycloak: + +Two Factor Auth using Duo with Keycloak +======================================= + +These are the steps to setup two factor authentication with Duo using Keycloak. + +Install Keycloak Duo SPI +-------------------------------------------------- + +#. Clone the Keycloak Duo SPI repo + + .. code:: + + git clone https://github.com/OSC/keycloak-duo-spi.git + cd keycloak-duo-spi + git submodule update --init + +#. Edit ``pom.xml`` and ensure ``keycloak.version`` matches the version of Keycloak you are running. + +#. Build (with Docker) - produces ``target/keycloak-duo-spi-jar-with-dependencies.jar`` + + .. code:: + + docker run --rm -it -v $(pwd):/keycloak-duo-spi -w /keycloak-duo-spi \ + ohiosupercomputer/keycloak_duo_spi_buildbox:latest mvn clean test package + +#. Build (without Docker) - produces ``target/keycloak-duo-spi-jar-with-dependencies.jar`` + + .. code:: + + yum -y install maven + cd build/duo_java/DuoWeb + mvn clean test install + cd ../../.. + mvn clean test package + +#. Copy the JAR file and necessary template files to Keycloak and instruct Keycloak to install the SPI + + .. code:: + + sudo install -o keycloak -g keycloak -m 0644 target/keycloak-duo-spi-jar-with-dependencies.jar \ + /opt/keycloak-9.0.0/standalone/deployments/keycloak-duo-spi-jar-with-dependencies.jar + sudo install -o keycloak -g keycloak -m 0644 src/main/resources/duo-mfa.ftl \ + /opt/keycloak-9.0.0/themes/base/login/duo-mfa.ftl + sudo install -o keycloak -g keycloak -m 0644 /dev/null \ + /opt/keycloak-9.0.0/standalone/deployments/keycloak-duo-spi-jar-with-dependencies.jar.dodeploy + +Configure Duo SPI +-------------------------------------------------- + +#. Log into your Keycloak instance +#. Choose the realm to configure in upper left corner, eg ``ondemand`` +#. Choose ``Realm Settings`` in the left menu then ``Security Defenses`` tab +#. Add ``frame-src https://*.duosecurity.com/ 'self';`` to the beginning of the value for ``Content-Security-Policy`` +#. Choose ``Authentication`` in the left menu +#. While on ``Flows`` tab ensure the dropdown for the flow name is ``Browser`` and click ``Copy`` +#. Name the new flow ``browser-with-duo`` +#. For all items below ``Username Password Form`` delete them by choosing ``Actions`` then ``Delete`` +#. Choose ``Actions`` for ``Browser-with-duo Forms`` and choose ``Add Execution`` +#. Select the ``Duo MFA`` provider and click ``Save`` +#. Click ``Actions`` for ``Duo MFA`` and select ``Config``. Fill in all values as appropriate and select ``Save``. +#. Select ``Required`` for ``Duo MFA`` +#. Choose the ``Bindings`` tab and set ``Browser Flow`` to ``browser-with-duo`` and choose ``Save`` + +Users logging into Keycloak will be required to verify their identity using Duo. diff --git a/downcase-ids/_sources/authentication/insecure.rst.txt b/downcase-ids/_sources/authentication/insecure.rst.txt new file mode 100644 index 000000000..fc64851f7 --- /dev/null +++ b/downcase-ids/_sources/authentication/insecure.rst.txt @@ -0,0 +1,12 @@ +.. _authentication-insecure: + +Other Insecure Options +====================== + +There are other insecure options that you may find Apache supports. + +It's left to the reader to try those out. Open OnDemand developers +highly discourage users attempting to authenticate with Apache's +BASIC auth like PAM and LDAP as they are really quite insecure. + +Questions on these topics will be linked back to this page. diff --git a/downcase-ids/_sources/authentication/nsf-access.rst.txt b/downcase-ids/_sources/authentication/nsf-access.rst.txt new file mode 100644 index 000000000..662ea9e17 --- /dev/null +++ b/downcase-ids/_sources/authentication/nsf-access.rst.txt @@ -0,0 +1,100 @@ +.. _nsf-access: + +NSF ACCESS +---------- + +If your site is a part of the `National Science Foundation`_'s (NSF) +`ACCESS`_ program (formerley `XSEDE`_) you can use their Identity Provider (IDP) +to authenticate users for your Open OnDemand instance. + +OIDC Client Registration +************************ + +You should read the `ACCESS IDP documentation`_ on how to register your Open OnDemand +instance as an Open ID Connect (OIDC) client. +ACCESS uses `CILogon`_ to provide a bridge from campus authentication, via the InCommon Federation, +to OAuth/OIDC-based research cyberinfrastructure (CI). + +Once you've registered your Open OnDemand instance, you can then configure it accordingly. +Since `ACCESS`_ uses Open ID Connect (OIDC) you can see our :ref:`oidc documentation ` +for more details on how to configure Open OnDemand with what CILogon has provided in +registering your application. + +Here's an example you can use to get started. Note that ``oidc_client_id`` and ``oidc_client_secret`` +are commented out because they are specific to your site. + +.. code-block:: yaml + :emphasize-lines: 3-4 + + oidc_uri: "/oidc" + oidc_provider_metadata_url: "https://cilogon.org/.well-known/openid-configuration" + # oidc_client_id: "cilogon:/client_id/..." + # oidc_client_secret: "..." + oidc_remote_user_claim: "sub" + oidc_scope: "openid email profile org.cilogon.userinfo" + oidc_session_inactivity_timeout: 28800 + oidc_session_max_duration: 28800 + oidc_state_max_number_of_cookies: "10 true" + oidc_settings: + OIDCPassIDTokenAs: "serialized" + OIDCPassRefreshToken: "On" + OIDCPassClaimsAs: "environment" + OIDCStripCookies: "mod_auth_openidc_session mod_auth_openidc_session_chunks mod_auth_openidc_session_0 mod_auth_openidc_session_1" + OIDCAuthRequestParams: "idphint=https%3A%2F%2Faccess-ci.org%2Fidp" + + +Shibboleth and InCommon +*********************** + +If your campus already runs Shibboleth authentication, you have an alternative to the Open ID Connect +configuration above. + +The SAML metadata for idp.access-ci.org is published by InCommon and can be downloaded using the +Metadata Query (MDQ) Service from https://mdq.incommon.org/entities/https%3A%2F%2Faccess-ci.org%2Fidp . +Alternatively, you can download the metadata from https://identity.access-ci.org/access-metadata.xml +and configure it in a local file. + +See our :ref:`shibboleth documentation ` for more information on +Shibboleth authentication. + +Mapping Users +************* + +`ACCESS`_ users have allocations on many `ACCESS`_ resource, of which you are one. +This means they have disparate usernames on all these systems and a unique username +on _your_ system. + +So you'll need an additional utility provided by access `ACCESS`_, namely the +`access-oauth-mapfile`_. + +Follow the instructions to install that utility and you'll get a lookup table +in ``/etc/grid-security/access-oauth-mapfile`` like so: + +.. code-block:: sh + + annie-oakley@access-ci.org aoakley + + +You can set the `user_map_cmd`_ in ``ood_portal.yml`` to search this file and return +the local user given the ACCESS username. + +.. code-block:: sh + + #!/bin/bash + + MAPPED_USER=$(grep "$1" ./delme.txt | awk '{print $2}') + if [[ "$MAPPED_USER" != "" ]]; then + echo -n "$MAPPED_USER" + else + echo "$1-not-found" + fi + + +.. _mod_auth_openidc: https://github.com/zmartzone/mod_auth_openidc +.. _National Science Foundation: https://www.nsf.gov/ +.. _ACCESS: https://access-ci.org/ +.. _XSEDE: https://www.xsede.org/ +.. _ACCESS IDP documentation: https://identity.access-ci.org/about-access-idp +.. _CILogon: https://www.cilogon.org/faq +.. _access-oauth-mapfile: https://github.com/access-ci-org/access-oauth-mapfile +.. _user_map_cmd: ood-portal-generator-user-map-cmd diff --git a/downcase-ids/_sources/authentication/oidc.rst.txt b/downcase-ids/_sources/authentication/oidc.rst.txt new file mode 100644 index 000000000..52c476c54 --- /dev/null +++ b/downcase-ids/_sources/authentication/oidc.rst.txt @@ -0,0 +1,63 @@ +.. _authentication-oidc: + +OpenID Connect +-------------- + +This is a summary of using OpenID Connect for authentication. Two example OpenID Connect identity providers we have documented include :ref:`Dex ` and :ref:`Keycloak `. + +The following prerequisites need to be satisfied: + +- A OIDC IdP server deployed, e.g., ``idp.example.com`` (outside of scope of this document) +- The `mod_auth_openidc`_ installed on the OnDemand Server. + +.. note:: + + The OnDemand repos have the ``mod_auth_openidc`` RPM for RHEL 8 and Rocky 8 that are newer than what the OS provides to make use of some newer features. + +The following is an example :program:`ood-portal-generator` configuration file: + +.. code-block:: yaml + + # /etc/ood/config/ood_portal.yml + --- + servername: ondemand.example.com + + # Use OIDC authentication + auth: + - "AuthType openid-connect" + - "Require valid-user" + + # Use OIDC logout + logout_redirect: "/oidc?logout=https%3A%2F%2Fondemand.example.com" + + # Capture system user name from authenticated user name + user_map_cmd: "/opt/ood/ood_auth_map/bin/ood_auth_map.regex" + + oidc_uri: "/oidc" + oidc_provider_metadata_url: "https://idp.example.com/.well-known/openid-configuration" + oidc_client_id: "ondemand.example.com" + oidc_client_secret: "b972c25d-88a4-41fa-87f7-a12ff167dd13" + oidc_remote_user_claim: "preferred_username" + oidc_scope: "openid profile email groups" + oidc_session_inactivity_timeout: 28800 + oidc_session_max_duration: 28800 + oidc_state_max_number_of_cookies: "10 true" + oidc_settings: + OIDCPassIDTokenAs: "serialized" + OIDCPassRefreshToken: "On" + OIDCPassClaimsAs: "environment" + OIDCStripCookies: "mod_auth_openidc_session mod_auth_openidc_session_chunks mod_auth_openidc_session_0 mod_auth_openidc_session_1" + OIDCResponseType: "code" + +.. note:: + + It's recommended to secure :file:`/etc/ood/config/ood_portal.yml` by changing the file permissions to be ``0600`` and make the file only readable by ``root``: + + .. code-block:: sh + + sudo chmod root:root /etc/ood/config/ood_portal.yml + sudo chmod 0600 /etc/ood/config/ood_portal.yml + +Build the Apache configuration file and install it. + +.. _mod_auth_openidc: https://github.com/zmartzone/mod_auth_openidc diff --git a/downcase-ids/_sources/authentication/overview.rst.txt b/downcase-ids/_sources/authentication/overview.rst.txt new file mode 100644 index 000000000..fcb790ae8 --- /dev/null +++ b/downcase-ids/_sources/authentication/overview.rst.txt @@ -0,0 +1,22 @@ +.. _authentication-overview: + +Overview +======== + +Configuring Open OnDemand to work with an Apache authentication module can be +broken down into three procedures: + +#. Install and configure the Apache authentication module +#. Setup the mapping of the remote user to the local user +#. Configure logout + +These are discussed in more detail in the following sections. + +.. toctree:: + :maxdepth: 2 + :numbered: + :caption: Setup Authentication Module + + overview/configure-authentication + overview/map-user + overview/configure-logout diff --git a/downcase-ids/_sources/authentication/overview/configure-authentication.rst.txt b/downcase-ids/_sources/authentication/overview/configure-authentication.rst.txt new file mode 100644 index 000000000..4ada1e5d2 --- /dev/null +++ b/downcase-ids/_sources/authentication/overview/configure-authentication.rst.txt @@ -0,0 +1,96 @@ +.. _authentication-overview-configure-authentication: + +Configure Apache Authentication +=============================== + +Configure Authentication Module +------------------------------- + +Any Apache authentication module specific configuration directives (e.g., +``OIDCCLientID``, ``CASLoginURL``, ...) should reside outside of the +:file:`/opt/rh/httpd24/root/etc/httpd/conf.d/ood-portal.conf` configuration +file. The Apache configuration files are loaded in lexical order, so placing +these module specific configuration directives in the file: + +.. code-block:: text + + /opt/rh/httpd24/root/etc/httpd/conf.d/auth-config.conf + +will cause your authentication configuration directives to be loaded before +:file:`/opt/rh/httpd24/root/etc/httpd/conf.d/ood-portal.conf`. If there are any +secrets inside this file you can ensure privacy by adding restrictive file +permissions: + +.. code-block:: sh + + sudo chmod 640 /opt/rh/httpd/root/etc/httpd/conf.d/auth-config.conf + +Add to OnDemand Portal +---------------------- + +.. danger:: + + **Never** directly edit the + :file:`/opt/rh/httpd24/root/etc/httpd/conf.d/ood-portal.conf` to include + this authentication module within your Open OnDemand portal. This could + cause future upgrades of OnDemand to break. + +Edit the YAML configuration file for the :ref:`ood-portal-generator` located +under :file:`/etc/ood/config/ood_portal.yml`. For example, to add support for +an authentication module with ``AuthType`` of ``my-auth``, you would modify the +:file:`/etc/ood/config/ood_portal.yml` file as such: + +.. code-block:: yaml + + # /etc/ood/config/ood_portal.yml + --- + # ... + # Your other custom configuration options... + # ... + + auth: + - 'AuthType my-auth' + - 'Require valid-user' + +You would then build and install the new Apache configuration file with: + +.. code-block:: sh + + sudo /opt/ood/ood-portal-generator/sbin/update_ood_portal + +Finally you will need to restart your Apache HTTP Server for the changes to +take effect. + +Sanitize Session Information +---------------------------- + +You will need to sanitize any session-specific request headers that may be +passed to the backend web servers that a user is proxied to. For most Apache +authentication modules there are module-specific directives that can be enabled +to wipe session information from being passed as headers (e.g., +``OIDCStripCookies ...``). In other cases you may have to use regular +expressions to search for the session cookies and remove them manually. + +For example, Shibboleth does not have a directive to strip session information +from the cookies, so we accomplish this with the following options in our +:ref:`ood-portal-generator` configuration file: + +.. code-block:: yaml + + # /etc/ood/config/ood_portal.yml + --- + # ... + # Your other custom configuration options... + # ... + + auth: + - 'AuthType shibboleth' + - 'ShibRequestSetting requireSession 1' + - 'RequestHeader edit* Cookie "(^_shibsession_[^;]*(;\s*)?|;\s*_shibsession_[^;]*)" ""' + - 'RequestHeader unset Cookie "expr=-z %{req:Cookie}"' + - 'Require valid-user' + +where we use a regular expression to replace any ``shibsession`` cookies with +empty strings and delete the cookie header if it becomes empty. + +.. _apache http server 2.4: https://www.softwarecollections.org/en/scls/rhscl/httpd24/ diff --git a/downcase-ids/_sources/authentication/overview/configure-logout.rst.txt b/downcase-ids/_sources/authentication/overview/configure-logout.rst.txt new file mode 100644 index 000000000..03c513a74 --- /dev/null +++ b/downcase-ids/_sources/authentication/overview/configure-logout.rst.txt @@ -0,0 +1,36 @@ +.. _authentication-overview-configure-logout: + +Configure Logout +================ + +The logout link on the dashboard is ``/logout``. OnDemand's Apache config has a separate directive to handle ``/logout``, which by default redirects the user to ``/pun/sys/dashboard/logout``, which is a default logout page displayed by the dashboard. Because authentication handled by Apache, this approach enables the logout URL to be changed based on the authentication strategy used. + +To change the logout_redirect URL, set ``logout_redirect: "https:://URL/TO/LOGOUT/USER"`` in the ood-portal-generator config at ``/etc/ood/config/ood_portal.yml`` and regenerate the config. + + +.. describe:: logout_redirect (String, null) + + the URI the user is redirected to when accessing the logout URI above + + Default + Redirect to the dashboard's log out page + + .. code-block:: yaml + + logout_redirect: "/pun/sys/dashboard/logout" + + Using OpenID Connect Apache module + Redirect to the mod_auth_oidc logout location: + + .. code-block:: yaml + + logout_redirect: "/oidc?logout=https%3A%2F%2Fondemand.my-center.edu" + + Using Shibboleth Apache module + If the Shibboleth IdP server deployed is at idp.my-center.edu, this is an example redirect with mod_auth_shib: + + .. code-block:: yaml + + logout_redirect: "/Shibboleth.sso/Logout?return=https%3A%2F%2Fidp.my-center.edu%2Fidp%2Fprofile%2FLogout" + + See :ref:`authentication-shibboleth` for more details regarding setting up authentication with the Shibboleth Apache module. diff --git a/downcase-ids/_sources/authentication/overview/map-user.rst.txt b/downcase-ids/_sources/authentication/overview/map-user.rst.txt new file mode 100644 index 000000000..3ced63b90 --- /dev/null +++ b/downcase-ids/_sources/authentication/overview/map-user.rst.txt @@ -0,0 +1,203 @@ +.. _authentication-overview-map-user: + +Setup User Mapping +================== + +Every authenticated HTTP request sent to the OnDemand portal has a ``REMOTE_USER``. +This ``REMOTE_USER`` can be an email like ``annie.oakley@osc.edu`` and needs to be +mapped to a Linux system user like ``aoakley``. + +This is what we call user mapping. Mapping apache's ``REMOTE_USER`` from an +authenticated request to a Linux system user. + +Mapping to the local system user not only restricts access of OnDemand to local users +but it is also required by the OnDemand proxy to traffic the HTTP data to the user's +corresponding per-user NGINX (PUN) server. + +Versions prior to 2.0 relied on :ref:`user_map_cmd ` to do this. +Since 2.0 you should use the simpler and faster :ref:`user_map_match `. + +Both with variations will be discussed here. + + +Remote User +----------- + +It's worth discussusing where ``REMOTE_USER`` is coming from. When apache +has successfully authenticates a request it sets the variable ``REMOTE_USER`` +from, well, the remote. + +This is generally a return value from the authentication system like an +:ref:`open id connect provider `. It's common for this +to be an email address. + +You *can* configure `user_env`_ to use something other than ``REMOTE_USER``, but +it's unlikely you should need to. + +If you're using an OpenID Connect provider you may need to set +`oidc_remote_user_claim`_ as this setting +tells apache how to set ``REMOTE_USER`` from the claim response. + + +Reguluar Expression User Mapping +-------------------------------- + +The simplest and fastest way to map a ``REMOTE_USER`` to a system user is through +:ref:`user_map_match `. It isn't directly +regular expression matching, but it's close enough for most use cases. +See it's documentation for examples and more. + +Dex Automatic Configuration +--------------------------- + +When using the OpenId Connector `dex`_ and setting `oidc_remote_user_claim`_ +to ``email`` we automatically set `user_map_match`_ to ``^([^@]+)@.*$`` as +a convienience. + +User Map Command for Advanced Mappings +-------------------------------------- + +Versions prior to 2.0 provided a default `user_map_cmd`_ in +``/opt/ood/ood_auth_map/bin/ood_auth_map.regex``. We no longer distribute +this file. + +Sites instead need to write their own mapping script should they need +this capability. Set this custom mapping script in the `user_map_cmd`_ +configuration and be sure to make this mapping script executable. + +.. warning:: + Be aware, this script is executed on every request. + +Let's take a simple example. It uses bash's builtin regular expression matching +against ``([^@]+)@osc.edu`` - an osc dot edu email address. If that matches against +``$1`` (the ``REMOTE_USER``) after it's url-decoded, then we return an all lowercase +version of the first part of an email address. + +The contract this script has with ood is that ``REMOTE_USER`` is url-encoded and +passed into it as the first arguement, ``$1``. + +The script will return 0 and output the match if it can correctly map the user. +Otherwise, if it fails, it will output nothing and exit 1. + +.. code-block:: sh + + #!/bin/bash + + function urldecode() { : "${*//+/ }"; echo -e "${_//%/\\x}"; } + + REX="([^@]+)@osc.edu" + INPUT_USER=$(urldecode $1) + + if [[ $INPUT_USER =~ $REX ]]; then + MATCH="${BASH_REMATCH[1]}" + echo "$MATCH" | tr '[:upper:]' '[:lower:]' + else + # can't write to standard out or error, so let's use syslog + logger -t 'ood-mapping' "cannot map $INPUT_USER" + + # and exit 1 + exit 1 + fi + + +If I were to run and test this script - it would return values like these: + +.. code-block:: sh + + $ /opt/site/custom_mapping.sh 'Annie.Oakley%40osc.edu' + annie.oakley + $ /opt/site/custom_mapping.sh 'jessie%40osc.edu' + jessie + $ /opt/site/custom_mapping.sh 'jessie.owens%40harvard.edu' + $ echo $? + $ 1 + $ journalctl -t ood-mapping + -- Journal begins at Tue 2020-06-02 06:45:03 EDT, ends at Wed 2022-01-19 15:11:37 EST. -- + Jan 19 15:03:14 localhost.localdomain ood-mapping[149352]: cannot map jessie.owens@harvard.edu + $ + +.. _gridmap_user_mapping: + +File User Mapping +----------------- + +This script parses a mapfile with each entry given in the following format: + +:: + + "authenticated_username" local_username + + +and separated by newlines. The script will systematically parse each line in +the mapfile looking for a match to the ``authenticated_username``. When a match +is found it breaks from the scan and outputs the ``local_username`` to +``STDOUT``. + +.. warning:: + Be aware, this script is executed and reads a user mapping file on every request. + +.. code-block:: sh + + /opt/ood/ood_auth_map/bin/ood_auth_map.mapfile [OPTIONS] + +.. program:: ood_auth_map.mapfile + +The options for this script are: + +.. option:: -f , --file + + Default: ``/etc/grid-security/grid-mapfile`` + + File used to scan for matches. + +Examples for the MapFile script +******************************* + +To scan the default grid-mapfile using a URL-encoded authenticated username: + +.. code-block:: sh + + $ /opt/ood/ood_auth_map/bin/ood_auth_map.mapfile 'http%3A%2F%2Fcilogon.org%2FserverA%2Fusers%2F58606%40cilogon.org' + bob + $ + +To scan a custom mapfile using an authenticated username: + +.. code-block:: sh + + $ /opt/ood/ood_auth_map/bin/ood_auth_map.mapfile --file '/path/to/mapfile' 'opaque_remote_username' + bob + $ + +If no match is found within the mapfile for the supplied +authenticated username that an empty string is returned instead: + +.. code-block:: sh + + $ /opt/ood/ood_auth_map/bin/ood_auth_map.mapfile 'this_remote_username_does_not_exist' + + $ + +Debugging User Mapping +---------------------- + +When debugging user mapping, it's always helpful to increase the `lua_log_level`_ to +debug. + +In doing so you'll see messages like that detail the mapping input, output and +times like ``Mapped 'jeff@localhost' => 'jeff' [0.089 ms]``. + +The full message would look like this. + +.. code-block:: sh + + /var/log/httpd/error.log:[Wed Jan 19 20:45:36.955855 2022] [lua:debug] [pid 39:tid 140070995539712] @/opt/ood/mod_ood_proxy/lib/ood/user_map.lua(21): [client 10.0.2.100:40172] Mapped 'jeff@localhost' => 'jeff' [0.089 ms], referer: http://localhost:5556/ + + + +.. _dex: authentication-dex +.. _user_map_match: ood-portal-generator-user-map-match +.. _user_map_cmd: ood-portal-generator-user-map-cmd +.. _user_env: ood-portal-generator-user-env +.. _oidc_remote_user_claim: ood-portal-generator-user-map-match +.. _lua_log_level: ood-portal-generator-lua-log-level diff --git a/downcase-ids/_sources/authentication/shibboleth.rst.txt b/downcase-ids/_sources/authentication/shibboleth.rst.txt new file mode 100644 index 000000000..3cdcc968c --- /dev/null +++ b/downcase-ids/_sources/authentication/shibboleth.rst.txt @@ -0,0 +1,71 @@ +.. _authentication-shibboleth: + +Shibboleth +---------- + +The following prerequisites need to be satisfied: + +- A Shibboleth IdP server deployed, e.g., ``idp.my-center.edu`` (outside of + scope of this document) +- The `Apache module for Shibboleth`_ installed on the OnDemand Server and + properly configured with its own Apache config (outside of scope of this + document) + +.. warning:: + + It is required you turn on ``ShibCompatValidUser`` in your Apache config + when setting up the Shibboleth module for Apache above. + + .. code-block:: apache + + # /path/to/httpd/conf.d/auth_shib.conf + + # + # Turn this on to support "require valid-user" rules from other + # mod_authn_* modules, and use "require shib-session" for anonymous + # session-based authorization in mod_shib. + # + ShibCompatValidUser On + +Then you can modify your :program:`ood-portal-generator` configuration file as +such: + +.. code-block:: yaml + + # /etc/ood/config/ood_portal.yml + --- + + # Use Shibboleth authentication + auth: + - "AuthType shibboleth" + - "ShibRequestSetting requireSession 1" + - "RequestHeader edit* Cookie \"(^_shibsession_[^;]*(;\\s*)?|;\\s*_shibsession_[^;]*)\" \"\"" + - "RequestHeader unset Cookie \"expr=-z %{req:Cookie}\"" + - "Require valid-user" + + # Use Shibboleth logout + logout_redirect: /Shibboleth.sso/Logout?return=https%3A%2F%2Fidp.my-center.edu%2Fidp%2Fprofile%2FLogout + + # Capture system user name from authenticated user name + user_map_match: '([^@]+)@my-center.edu' + + +In the example above: + +- The ``user_map_match`` uses regular expressions to map the authenticated user + ``bob@my-center.edu`` to their system user name ``bob``. +- The ``RequestHeader`` settings are used to strip private session information + from being sent to the backend web servers. +- The ``logout_redirect`` will first redirect the user to log out of the Open + OnDemand portal followed by redirecting the user to log out of the Shibboleth + IdP server. + +Remember, in order to apply modifications to the :file:`/etc/ood/config/ood_portal.yml` +you need to generate the new Apache configuration file with: + +.. code-block:: sh + + sudo /opt/ood/ood-portal-generator/sbin/update_ood_portal + + +.. _apache module for shibboleth: https://wiki.shibboleth.net/confluence/display/SHIB2/NativeSPApacheConfig diff --git a/downcase-ids/_sources/authentication/tutorial-oidc-keycloak-rhel7.rst.txt b/downcase-ids/_sources/authentication/tutorial-oidc-keycloak-rhel7.rst.txt new file mode 100644 index 000000000..04bb5182b --- /dev/null +++ b/downcase-ids/_sources/authentication/tutorial-oidc-keycloak-rhel7.rst.txt @@ -0,0 +1,47 @@ +.. _authentication-tutorial-oidc-keycloak-rhel7: + +OpenID Connect with KeyCloak on RHEL7 +===================================== + +This tutorial shows installing Keycloak as an OpenID Connect Identity Provider and configuring OnDemand as an OpenID Client to authenticate with this provider. + +Using ``https://ondemand-dev.hpc.osc.edu`` as the example host with OnDemand installed, at the end of the tutorial: + +#. Keycloak is running and accessible at ``https://ondemand-idpdev.hpc.osc.edu`` +#. In both cases Apache is handling requests. Apache proxies requests for ``https://ondemand-idpdev.hpc.osc.edu`` to the Keycloak server running on the default port of 8080. +#. Attempting to access OnDemand at ``https://ondemand-dev.hpc.osc.edu`` redirects the user to ``https://ondemand-idpdev.hpc.osc.edu`` to first authenticate. + +At OSC in production we do two things differently from this tutorial: + +#. we keep Keycloak on a separate host from OnDemand +#. we configure Keycloak to use MySQL for Keycloak's database instead of the default H2 file database + +These steps have been omitted from the tutorial. For most cases for OnDemand, +the default H2 database is probably sufficient. Also by installing Keycloak on +the same host as OnDemand, we don't need to provision separate SSL certificates +and host, which simplifies the tutorial. + +If your site is interested in either of these things and needs assistance, +please let us know by contacting us on the OnDemand Discourse at https://discourse.osc.edu/c/open-ondemand. + +.. warning:: + + In production we recommend installing Keycloak on a separate host from OnDemand. + +.. note:: + + This tutorial has only been verified to work with Keycloak 9.0.0. + + + + +.. toctree:: + :maxdepth: 2 + :numbered: + :caption: Tutorial + + tutorial-oidc-keycloak-rhel7/install-keycloak + tutorial-oidc-keycloak-rhel7/configure-keycloak-webui + tutorial-oidc-keycloak-rhel7/install_mod_auth_openidc + tutorial-oidc-keycloak-rhel7/add-custom-theme + tutorial-oidc-keycloak-rhel7/configure-cilogon diff --git a/downcase-ids/_sources/authentication/tutorial-oidc-keycloak-rhel7/add-custom-theme.rst.txt b/downcase-ids/_sources/authentication/tutorial-oidc-keycloak-rhel7/add-custom-theme.rst.txt new file mode 100644 index 000000000..73eded32e --- /dev/null +++ b/downcase-ids/_sources/authentication/tutorial-oidc-keycloak-rhel7/add-custom-theme.rst.txt @@ -0,0 +1,36 @@ +.. _authentication-tutorial-oidc-keycloak-rhel7-add-custom-theme: + +Add Custom Theme +================================================================ + +Custom themes are added to a realm by + +1. adding the theme directory to ``/opt/keycloak-9.0.0/themes`` +2. selecting the theme via Admin Web UI >> Realm Settings >> Themes + +Each theme is selectable based on the directory name of the theme. Themes can +extend other themes. + +Here are two links to get started with a custom theme: + +1. Currently version `v2.3.1 of OSC's Keycloak theme `__ + can be used as a starting point for modification. This theme is based off of + the default ``keycloak`` theme which itself is based off the ``base`` theme. + Files to modify include: + + - ``login/resources/img/ondemand-logo.png`` add a logo with this name here + - ``login/resources/img/favicon.ico`` replace with your own or remove + - ``login/messages/messages_en.properties`` replace text with text + appropriate for your center + +2. See the Keycloak documentation for themes: https://www.keycloak.org/docs/latest/server_development/index.html#_themes + +Remember after adding a theme you still need to configure your realm in the +Keycloak admin UI to use the theme for the login pages. + +.. note:: + + Soon we will offer an ood-keycloak base theme that be easier to extended to + provide most of the common themeing a site might like to perform. It will + also work well for OTP views. + diff --git a/downcase-ids/_sources/authentication/tutorial-oidc-keycloak-rhel7/configure-cilogon.rst.txt b/downcase-ids/_sources/authentication/tutorial-oidc-keycloak-rhel7/configure-cilogon.rst.txt new file mode 100644 index 000000000..48b477663 --- /dev/null +++ b/downcase-ids/_sources/authentication/tutorial-oidc-keycloak-rhel7/configure-cilogon.rst.txt @@ -0,0 +1,69 @@ +.. _authentication-tutorial-oidc-keycloak-rhel7-configure-cilogon: + +Configure Keycloak with CILogon +=============================== + +We will now use Keycloak's admin Web UI to setup the ability to log existing users in with CILogon. + +When a user logs in with CILogon for the first time they will be redirected back to Keycloak to log in with their local (ie LDAP) +credentials. This performs a mapping of their CILogon identity with their Keycloak identity. + +.. warning:: + + CILogon can only map a single external identity to a Keycloak account. This means if a user logs in with Institution A they + must remove their mapping in order to log in with Institution B. + +Register your Keycloak instance with CILogon +--------------------------------------------- + +#. Go to ``https://cilogon.org/oauth2/register`` and fill out the form + + #. The Home URL will be the base URL of your Keycloak instance, eg: ``https://ondemand-idpdev.hpc.osc.edu``. + #. The callback URL will be ``https://ondemand-idpdev.hpc.osc.edu/auth/realms//broker/cilogon/endpoint``. + Replace ``https://ondemand-idpdev.hpc.osc.edu`` with your Keycloak instance + #. The box for "Is this a public client?" should not be checked + #. For "Scopes" be sure to check "profile" and "org.cilogon.userinfo" + +You will be provided a Client ID and a Client Secret, be sure to save these values. +Your registered client will not be usable until you receive an email from CILogon stating your client has been approved. + +Add the CILogon Identity Provider +------------------------------------------ + +#. Log into ``https://ondemand-idpdev.hpc.osc.edu`` as the admin user +#. Select your desired realm in the upper left corner +#. Choose "Identity Providers" in the left menu +#. Select the "Add provider..." drop down and choose "OpenID Connect v1.0" +#. Fill in the fields as noted below + + #. Alias: cilogon (This must be cilogon as this alias is used in the callback URL) + #. Display Name: CILogon + #. Enabled: ON + #. First Login Flow: browser + #. Authorization URL: https://cilogon.org/authorize + #. Token URL: https://cilogon.org/oauth2/token + #. User Info URL: https://cilogon.org/oauth2/userinfo + #. Client Authentication: Client secret sent as post + #. Client ID: + #. Client Secret: + #. Default Scopes: "openid profile org.cilogon.userinfo" + +#. Click "Save" + +Support users removing CILogon mappings +------------------------------------------ + +In order for a user to remove an existing CILogon mapping in Keycloak they must navigate to ``https://ondemand-idpdev.hpc.osc.edu/auth/realms//account/identity``. +Replace ``ondemand-idpdev.hpc.osc.edu`` with the web URL for your Keycloak instance. + +The URL can be added to the OnDemand Help dropdown with custom text to make it easier for users to access their Keycloak identity page. + +#. Add ``OOD_DASHBOARD_HELP_CUSTOM_URL`` to ``/etc/ood/config/apps/dashboard/env`` that points to the URL of the identity page for your Keycloak instance. + Example: ``https://ondemand-idpdev.hpc.osc.edu/auth/realms/osc/account/identity`` +#. Update ``/etc/ood/config/locales/en.yml`` with the text to be used for the Identity provider Help link + + .. code:: + + en: + dashboard: + nav_help_custom: Manage Federated Identity diff --git a/downcase-ids/_sources/authentication/tutorial-oidc-keycloak-rhel7/configure-keycloak-webui.rst.txt b/downcase-ids/_sources/authentication/tutorial-oidc-keycloak-rhel7/configure-keycloak-webui.rst.txt new file mode 100644 index 000000000..6ac157a96 --- /dev/null +++ b/downcase-ids/_sources/authentication/tutorial-oidc-keycloak-rhel7/configure-keycloak-webui.rst.txt @@ -0,0 +1,70 @@ +.. _authentication-tutorial-oidc-keycloak-rhel7-configure-keycloak-webui: + +Configure Keycloak +================== + +We will now use Keycloak's admin Web UI to setup LDAP and add OnDemand as an +OIDC client that will authenticate with Keycloak. + +Add a new realm +------------------------------------------ + +#. Log into ``https://ondemand-idpdev.hpc.osc.edu`` as the admin user +#. Hover over "Master" on left and click "Add Realm" +#. Type in name "ondemand" and click "Create". The new realm is loaded. +#. Click Login tab, then adjust parameters: + + #. Remember Me: ON + #. Login with email: OFF + +#. Click Save. + +Configure LDAP +------------------------------------------ + +#. Choose User Federation on the left (verify ondemand realm is current realm) +#. Select "ldap" for provider + + #. Import Users set to OFF + #. Edit Mode set to READ_ONLY + #. Vendor set to other – for OpenLDAP + #. User Object Classes set to posixAccount – OSC specific and odd + #. Connection URL: ldaps://ldap1.infra.osc.edu:636 ldaps://ldap2.infra.osc.edu:636 – using multiple to demonstrate more than 1 + #. User DN: ou=People,dc=osc,dc=edu + #. Auth Type: none – OSC specific as we allow anonymous binds + #. Use Truststore SPI: never – OSC specific since our LDAP certificates are already trusted since from InCommon, leaving default is probably acceptable if no truststoreSpi defined in XML configs + +#. Save + +.. warning:: + + These LDAP settings are what we set for OSC. Your configuration may vary from + this. If you run into any problems, please let us know so that once a + solution is reached we can document those problem areas here. Contact us on + the OnDemand Discourse at https://discourse.osc.edu/c/open-ondemand. + +Add OnDemand as a client +-------------------------------------------------- + +#. Choose Clients, then click Create in top right corner + + #. Client ID: ondemand-dev.hpc.osc.edu + #. Client Protocol: openid-connect + #. Save (leave Root URL blank) + +#. Then edit Settings for the newly created client: + + #. Access Type: confidential + #. Direct Access Grants Enabled: off + #. Valid Redirect URIs: Press the ``+`` button to the right of the URI field so you can insert two URLs: + + #. ``https://ondemand-dev.hpc.osc.edu/oidc`` + #. ``https://ondemand-dev.hpc.osc.edu`` + + #. Scroll to bottom and click "Save" + +#. Finally, get the client secret to use with OnDemand installation: + + #. Select the "Credentials" tab of the "Client" you are viewing i.e. "Clients >> ondemand-dev.hpc.osc.edu" + #. Copy the value for "secret" for future use in this tutorial (and keep it secure). + diff --git a/downcase-ids/_sources/authentication/tutorial-oidc-keycloak-rhel7/install-keycloak.rst.txt b/downcase-ids/_sources/authentication/tutorial-oidc-keycloak-rhel7/install-keycloak.rst.txt new file mode 100644 index 000000000..d2c22232f --- /dev/null +++ b/downcase-ids/_sources/authentication/tutorial-oidc-keycloak-rhel7/install-keycloak.rst.txt @@ -0,0 +1,197 @@ +.. _authentication-tutorial-oidc-keycloak-rhel7-install-keycloak: + +Install Keycloak +================ + +We will install and launch Keycloak server behind Apache. + +Login to the host where you will install Keycloak. In this tutorial, we are +installing Keycloak on the same host as OnDemand, which is webdev07.hpc.osc.edu. + +.. warning:: + + In production we recommend installing Keycloak on a separate host from OnDemand. + + +Initial Installation Steps +-------------------------- + +#. Download and unpack Keycloak 9.0.0 (from https://www.keycloak.org/downloads.html) + + .. code-block:: sh + + cd /opt + sudo wget https://downloads.jboss.org/keycloak/9.0.0/keycloak-9.0.0.tar.gz + sudo tar xzf keycloak-9.0.0.tar.gz + + +#. Add keycloak user and change ownership of files + + .. code-block:: sh + + sudo groupadd -r keycloak + sudo useradd -m -d /var/lib/keycloak -s /sbin/nologin -r -g keycloak keycloak + + If ``-m`` doesn't work, do this: + + .. code-block:: sh + + sudo install -d -o keycloak -g keycloak /var/lib/keycloak + + This makes a home directory, which is needed when running API calls as keycloak user. Finally we set proper permissions: + + .. code-block:: sh + + sudo chown keycloak: -R keycloak-9.0.0 + +#. Restrict access to keycloak-9.0.0/standalone, which will contain + sensitive data for the Keycloak server + + .. code-block:: sh + + cd keycloak-9.0.0 + sudo -u keycloak chmod 700 standalone + + +#. Install JDK 1.8.0 + + .. code-block:: sh + + sudo yum install java-1.8.0-openjdk-devel + + +#. Added 'admin' to '/opt/keycloak-9.0.0/standalone/configuration/keycloak-add-user.json', (re)start server to load user. + + If you are not already there: + + .. code-block:: sh + + cd /opt/keycloak-9.0.0 + + Generate a password to use for the admin user: + + .. code-block:: sh + + openssl rand -hex 20 # generate a password to use for admin user + sudo -u keycloak ./bin/add-user-keycloak.sh --user admin --password KEYCLOAKPASS --realm master + + **Replace KEYCLOAKPASS with a good password and save password for later use** + +#. Modify ``standalone/configuration/standalone.xml`` to enable proxying to Keycloak: + + Simplest is to run these three commands: + + .. code-block:: sh + + sudo -u keycloak ./bin/jboss-cli.sh 'embed-server,/subsystem=undertow/server=default-server/http-listener=default:write-attribute(name=proxy-address-forwarding,value=true)' + sudo -u keycloak ./bin/jboss-cli.sh 'embed-server,/socket-binding-group=standard-sockets/socket-binding=proxy-https:add(port=443)' + sudo -u keycloak ./bin/jboss-cli.sh 'embed-server,/subsystem=undertow/server=default-server/http-listener=default:write-attribute(name=redirect-socket,value=proxy-https)' + + Or you can use a config.cli file that contains these commands. We have + provided an example file to make use of in this gist, with blocks commented + out so you can wget the file, edit as appropriate, and run via: + + .. code-block:: sh + + sudo -u keycloak ./bin/jboss-cli.sh --file=config.cli + + Where the config.cli looks like: + + .. literalinclude:: example-keycloak-jboss-config.cli + +Start Keycloak Server +------------------------- + +#. Create keycloak.service to start and stop the server: + + .. code-block:: sh + + sudo cat > /etc/systemd/system/keycloak.service < + - AuthType Basic + - AuthName "Private" + - AuthUserFile "/opt/rh/httpd24/root/etc/httpd/.htpasswd" + - RequestHeader unset Authorization + + AuthType openid-connect + Require valid-user + + LuaHookFixups nginx.lua nginx_handler + + + #. Update the ``Redirect "logout"`` directive + + .. code-block:: diff + + - Redirect "/logout" "/pun/sys/dashboard/logout" + + Redirect "/logout" "/oidc?logout=https%3A%2F%2Fondemand-dev.hpc.osc.edu" + + #. Add the ```` directive + + .. code-block:: none + + # OpenID Connect redirect URI: + # + # https://ondemand-dev.hpc.osc.edu:443/oidc + # #=> handled by mod_auth_openidc + # + + AuthType openid-connect + Require valid-user + + +Add Keycloak config to OnDemand Apache for mod_auth_openidc +----------------------------------------------------------- + +#. Add the file /opt/rh/httpd24/root/etc/httpd/conf.d/auth_openidc.conf with the contents: + + .. code-block:: none + + OIDCProviderMetadataURL https://ondemand-idpdev.hpc.osc.edu/auth/realms/ondemand/.well-known/openid-configuration + OIDCClientID "ondemand-dev.hpc.osc.edu" + OIDCClientSecret "1111111-1111-1111-1111-111111111111" + OIDCRedirectURI https://ondemand-dev.hpc.osc.edu/oidc + OIDCCryptoPassphrase "4444444444444444444444444444444444444444" + + # Keep sessions alive for 8 hours + OIDCSessionInactivityTimeout 28800 + OIDCSessionMaxDuration 28800 + + # Set REMOTE_USER + OIDCRemoteUserClaim preferred_username + + # Don't pass claims to backend servers + OIDCPassClaimsAs environment + + # Strip out session cookies before passing to backend + OIDCStripCookies mod_auth_openidc_session mod_auth_openidc_session_chunks mod_auth_openidc_session_0 mod_auth_openidc_session_1 + + #. OIDCClientID: replace with the client id specified when installing the client in Keycloak admin interface + #. OIDCClientSecret: replace ``1111111-1111-1111-1111-1111111111111`` with client secret specified from the Credentials tab of the client in Keycloak admin interface + #. OIDCCryptoPassphrase: replace ``4444444444444444444444444444444444444444`` with random generated password. I used ``openssl rand -hex 40``. + #. Verify the OIDCProviderMetadataURL uses the correct realm and the port Apache exposes to the world for Keycloak by accessing the URL. + +#. Change permission on file to be readable by apache and no one else: + + .. code-block:: sh + + sudo chgrp apache /opt/rh/httpd24/root/etc/httpd/conf.d/auth_openidc.conf + sudo chmod 640 /opt/rh/httpd24/root/etc/httpd/conf.d/auth_openidc.conf + +#. Then restart OnDemand's Apache. OnDemand should now be authenticating using KeyCloak. + + Stop both servives: + + .. code-block:: sh + + sudo systemctl restart httpd24-httpd + +.. note:: + + We prevent OIDC_CLAIM headers from being passed through to the PUN + by specifying in this file to pass claims as environment, instead of + as HTTP headers, since Apache won't pass any environment off to the + PUN when proxying requests, but would pass HTTP headers. diff --git a/downcase-ids/_sources/customizations.rst.txt b/downcase-ids/_sources/customizations.rst.txt new file mode 100644 index 000000000..beb735c7b --- /dev/null +++ b/downcase-ids/_sources/customizations.rst.txt @@ -0,0 +1,1493 @@ +.. _customizations: + +Customizations +============== + +.. tip:: + Check out the :ref:`pun-environment` for an overview of how environment variables can be + added. + +.. _disabling_applications: + +Disabling applications +---------------------- + +OnDemand is comprised of a few components. Each of which you can disable or limit +access by simply changing the file permissions of the application. + +All the applications OnDemand installs are located in `/var/www/ood/apps/sys`. +So, for example, if you wished to disable the file browser you would simply +change it's directory to 700 so it's unreadable by regular users. + +When this directory is unreadable by regular users, the functionality +it provides will be disabled. + +.. code-block:: sh + + sudo chmod 700 /var/www/ood/apps/sys/files + +Alternatively, if you wished to limit access you can do so through group +permissions. For example, if you wanted to limit access to the file browser +to only members in the Unix group ``staff``, you would simply apply the +applicable file permission such that anonymous users cannot access the +directory while members of the ``staff`` Unix group can. + +.. code-block:: sh + + sudo chmod 750 /var/www/ood/apps/sys/files + sudo chown root:staff /var/www/ood/apps/sys/files + + +.. _configure_announcements: + +Announcements +------------- + +To add an announcement message that appears at the top of the dashboard you can create a file at ``/etc/ood/config/announcement.(md|yml)`` or ``/etc/ood/config/announcements.d/any_file_name.(md|yml)``. + +On each request the dashboard will check for the existence of this file. If it exists, the contents will be converted using markdown converter to HTML and displayed inside a bootstrap alert. + +For example, if I create an announcement.md file with the contents: + + .. code-block:: md + + **NOTICE:** There will be a two day downtime on February 21-22, 2017. OSC + OnDemand will be unavailable during this period. For details, please visit + [http://bit.ly/2jhfyh7](http://bit.ly/2jhfyh7). + +the user would see this message at the top of the dashboard: + +.. figure:: /images/dashboard-announcement.png + :align: center + + Example of the Dashboard announcement. + +If the announcement file has the extension ``yml`` and is a yaml file it is first rendered using ERB and then the resulting file is parsed as YAML. The valid keys are: + +.. list-table:: Announcement configuration keys. + + * - Key + - Description + * - type + - The type of announcment. Values can be ``warning``, ``info``, ``success``, or ``danger``. + * - msg + - The announcement's message. + * - dismissable + - Specify if the announcment is dismissable or not with ``true`` or ``false``. + Defaults to ``true``. + * - required + - Specify if the announcment is required or not with ``true`` or ``false``. + Defaults to ``false``. When this is set to ``true``, the user will not be + do anything until the announcment has been accepted. + +.. tip:: + You can use ``required`` announcements to present users with a ToS (terms of service), + EULA (end user license agreement) or similar. + +Because the announcement is rendered via ERB you can do some interesting things, like stop showing the announcement past a specified date: + + .. code-block:: erb + + type: warning + msg: | + <% if Time.now < Time.new(2018, 9, 24, 12, 0, 0) %> + A **Ruby Partial Downtime** for 4 hours on Monday, September 24 from 8:00am to 12:00pm + will prevent SSH login to Ruby nodes and Ruby VDI sessions. + <% end %> + +.. note:: Warnings about the announcement file being missing may be present in users' nginx logs. Despite the warning the Dashboard will still function normally without those files being present. + + The file path for the announcement message can be customize using configuration properties, see :ref:`OnDemand configuration documentation `. + +.. _motd_customization: + +Message of the Day (MOTD) +------------------------- + +You can configure the Dashboard to display the /etc/motd file on the front page - the same file that is displayed when ssh-ing to a login node. + +To display a MOTD file on the Dashboard ensure that the environment variables ``$MOTD_PATH`` and ``$MOTD_FORMAT`` are set, where + + .. code-block:: sh + + MOTD_PATH="/etc/motd" # this supports both file and RSS feed URIs + MOTD_FORMAT="txt" # markdown, txt, rss, markdown_erb, txt_erb + +.. tip:: + The ``_erb`` formats support ERB rendering to generate more dynamic messages. + +.. warning:: + Some MOTD formats like ``rss``, ``markdown`` and ``markdown_erb`` can contain malicious + HTML content. For your safety, by default, the Open OnDemand system will not render + HTML. We provide :ref:`a configuration to enable HTML rendering in MOTD ` + should you need to render HTML. + +.. figure:: /images/dashboard_motd.png + :align: center + + Message of the Day appears in the body of the index page. + +We recommend setting this in ``/etc/ood/config/apps/dashboard/env``. + + +Branding +------------------- + +.. _branding: + +You can customize the logo, favicon, title, and navbar colors of OnDemand. + +.. figure:: /images/dashboard_branding_logo_and_colors.png + :align: center + + +We recommend setting these values using the :ref:`OnDemand configuration properties `. +Currently only the dashboard uses the colors in the navbar. + +.. list-table:: Branding + :header-rows: 1 + :stub-columns: 1 + + * - Feature + - Property + - Details + * - Title + - dashboard_title + - The title appears in the navbar and is controlled by the property ``dashboard_title``. The default value is "Open OnDemand". + * - Logo + - dashboard_logo + - The default value for ``dashboard_logo`` is ``/public/logo.png`` and this should be the URL to the logo. By default if you place a logo.png at ``/var/www/ood/public/logo.png`` it will be accessible via the URL ``https://your.ondemand.institution.edu/public/logo.png``. SVG logo format is also supported. + * - Logo height + - dashboard_logo_height + - The CSS height of the dashboard logo. + * - Favicon + - public_url + - The favicon is expected to exist at the path ``$public_url/favicon.ico``. For a default OOD installation the favicon will be located at ``/var/www/ood/public/favicon.ico``. + * - Brand background color + - brand_bg_color + - Controls the background color of the navbar in the dashboard + * - Brand foreground color + - brand_link_active_bg_color + - Controls the background color the active link in the navbar in the dashboard + * - Replace header title with logo + - dashboard_header_img_logo + - Value should be url to logo i.e. ``/public/logo.png``. the background color the active link in the navbar in the dashboard + * - Use white text on black background for navbar. + - navbar_type + - By default we use ``inverse`` for this value, which specifies to use `Bootstrap 3's inverted navbar `_ where text is white and background is black (or dark grey). You can set this to ``default`` to use black text on light grey background if it fits your branding better. + +.. note:: It is possible to configure these settings using environment variables, although this is deprecated. + For information about the properties and environment variables, see the :ref:`OnDemand configuration documentation `. + +.. figure:: /images/dashboard_navbar_branding_bluered.png + :align: center + + Nav bar if I set ``brand_bg_color`` to ``#0000ff`` and ``brand_link_active_bg_color`` to ``#ff0000`` and ``dashboard_title`` to ``OSC OnDemand`` + +Custom CSS files +................ + +For more control on the look and feel of the dashboard pages, use custom CSS files. +These CSS files will be added to all dashboard pages and loaded last to have precedence over default styles. + +Add your CSS file references to the ``custom_css_files`` array property. +Drop the files into the Apache public assets folder, the default location is: ``/var/www/ood/public``. +The system will prepend the ``/public`` URL path, based on the ``public_url`` property, to load the files correctly. + +**Example:** to load the following CSS file: ``/var/www/ood/public/myfolder/custom-branding.css``, add the configuration below. +This will result in a CSS tag added to all dashboard pages with the path: ``/public/myfolder/custom-branding.css``. + +.. code-block:: yaml + + custom_css_files: ["/myfolder/custom-branding.css"] + +.. _help_menu_guide: + +.. include:: customizations/overriding-partials.inc + +Add URLs to Help Menu +--------------------- + +These URLs can be specified, which will appear in the Help menu and on other locations of the Dashboard. We recommend setting this in ``/etc/ood/config/apps/dashboard/env``. + +.. list-table:: Dashboard URLs + :header-rows: 1 + :stub-columns: 1 + + * - Name + - Environment variable + - Example value + * - Support URL + - OOD_DASHBOARD_SUPPORT_URL + - https://www.osc.edu/contact/supercomputing_support + * - User Documentation + - OOD_DASHBOARD_DOCS_URL + - https://www.osc.edu/ondemand + * - Developer Documentation + - OOD_DASHBOARD_DEV_DOCS_URL + - https://osc.github.io/ood-documentation/master/app-development.html (link appears in Develop dropdown if developer mode enabled for user) + * - Change Password URL + - OOD_DASHBOARD_PASSWD_URL + - https://my.osc.edu + * - Custom Help URL (Also requires locale ``en.dashboard.nav_help_custom``) + - OOD_DASHBOARD_HELP_CUSTOM_URL + - https://idp.osc.edu/auth/realms/osc/account/identity + +Since OnDemand 2.1, custom links can be added to the Help menu using the configuration property ``help_menu``. +Links will be inserted at the end of the core links already included in the menu by the OnDemand codebase. + +``help_menu`` supports all the link definitions developed for the custom navigation configuration. +For more information on how to create custom links, see for example :ref:`adding urls to menus `. + +For information about how to configure properties, see the :ref:`OnDemand configuration documentation `. + +.. code-block:: yaml + + help_menu: + - group: "Documentation" + - title: "Jupyter Docs" + icon: "fas://book" + url: "https://mydomain.com/path/jupyter" + - title: "Support Docs" + icon: "fas://book" + url: "https://mydomain.com/path/support/docs" + - group: "Custom Pages" + - page: "rstudio_guide" + title: "RStudio Guide" + icon: "fas://window-restore" + - group: "Profiles" + - profile: "team1" + title: "Team 1" + icon: "fas://user" + +.. figure:: /images/help_menu_links.png + :align: center + + +.. _add-shortcuts-to-files-menu: + +Add Shortcuts to Files Menu +--------------------------- + +The Files menu by default has a single link to open the Files app in the user's +Home Directory. More links can be added to this menu, for Scratch space and +Project space directories. + +Adding more links currently requires adding a custom initializer to the +Dashboard app. Ruby code is placed in the initializer to add one or more Ruby +``FavoritePath`` (or ``Pathname`` for backwards compatibility) objects to the ``OodFilesApp.candidate_favorite_paths`` array, a +global attribute that is used in the Dashboard app. + +``FavoritePath`` is instantiated with a single ``String`` or ``Pathname`` argument, the +directory path, and with an optional keyword argument ``title`` specifying a +human readable title for that path. + +Start by creating the file +:file:`/etc/ood/config/apps/dashboard/initializers/ood.rb` as such: + +.. code-block:: ruby + + # /etc/ood/config/apps/dashboard/initializers/ood.rb + + Rails.application.config.after_initialize do + OodFilesApp.candidate_favorite_paths.tap do |paths| + # add project space directories + projects = User.new.groups.map(&:name).grep(/^P./) + paths.concat projects.map { |p| FavoritePath.new("/fs/project/#{p}") } + + # add User scratch space directory + paths << FavoritePath.new("/fs/scratch/#{User.new.name}") + + # Project scratch is given an optional title field + paths.concat projects.map { |p| FavoritePath.new("/fs/scratch/#{p}", title: "Scratch") } + end + end + +- The variable ``paths`` is an array of ``FavoritePath`` objects that define a list + of what will appear in the Dashboard menu for Files +- At OSC, the pattern for project paths follows + :file:`/fs/project/{project_name}`. So above we: + + #. get an array of all user's groups by name + #. filter that array for groups that start with ``P`` (i.e., ``PZS0002``, + ``PAW0003``, ...) + #. using ``map`` we turn this array into an array of ``FavoritePath`` objects to + all the possible project directories the user could have. + #. extend the paths array with this list of paths + +- For possible scratch space directories, we look for either + :file:`/fs/scratch/{project_name}` or :file:`/fs/scratch/{user_name}` +- Additionally project scratch directories have a 'title' attribute and will + with in the dropdown with both the title and the path. + +The following example adds all directories within the specified base directories to the favorite paths. This approach is ideal when there is no specific directory naming logic to follow. It also appropriately handles Access Control Lists (ACLs). If a directory does not exist, no error is raised, making this configuration easily exportable to different clusters. + + +.. code-block:: ruby + + # /etc/ood/config/apps/dashboard/initializers/ood.rb + + Rails.application.config.after_initialize do + OodFilesApp.candidate_favorite_paths.tap do |paths| + + # Hash of base paths to check for additional directories with titles + # location => Title + base_paths = { + '/home/share/' => 'Shared home', + '/srv/scratch/shares/' => 'Shared scratch', + '/srv/scratch/groups/' => 'Group scratch', + '/srv/fast/users/' => 'Fast user' + # Add more paths and titles here if needed + } + + base_paths.each do |base_path, title| + # Check if the base path exists and is a directory, to avoid error + next unless Dir.exist?(base_path) + + # Get all entries in the current base path + Dir.entries(base_path).each do |entry| + # Construct the full path for the current entry + full_path = File.join(base_path, entry) + + # Skip if it's not a directory or if it's a special entry like '.' or '..' + next unless File.directory?(full_path) && !['.', '..'].include?(entry) + + # Check if the directory is readable and executable + if File.readable?(full_path) && File.executable?(full_path) + # Access the value of the current base_path using the `title` variable + paths << FavoritePath.new(full_path, title: "#{title}: #{File.basename(full_path)}") + end + end + end + end + end + + # The variable ``base_paths`` is an hash (``dirname`` => ``Title``) of all directories you want to parse. For the directory ``OSC_test`` in ``/srv/scratch/groups/``; the favorite path will be displayed as following + + | Group scratch: OSC_test | /srv/scratch/groups/OSC_test | + + On each request, the Dashboard will check for the existence of the directories + in ``OodFilesApp.candidate_favorite_paths`` array and whichever directories + exist and the user has access to will appear as links in the Files menu under + the Home Directory link. + + +.. figure:: /images/files_menu_shortcuts_osc.png + :align: center + + Shortcuts to scratch and project space directories in Files menu in OSC OnDemand. + +- You must restart the Dashboard app to see a configuration change take effect. + This can be forced from the Dashboard itself by selecting + *Help* → *Restart Web Server* from the top right menu. + +If you access the Dashboard, and it crashes, then you may have made a mistake +in ``ood.rb`` file, whose code is run during the initialization of the Rails +app. + +.. include:: customizations/profiles.inc + +.. include:: customizations/main-navigation.inc +.. include:: customizations/interactive-apps-menu.inc + + +Disable uploads or downloads +---------------------------- + +By default, Open OnDemand will allow users to upload and download files. +Both can be disabled through configuration. + +Use :ref:`download_enabled for disabling downloads ` +and :ref:`upload_enabled for disabling uploads `. + +.. tip:: + Use File Access Controls (FACLs) on the ondemand.d files to enable + or disable uploads or downloads based on group permissions. + +.. _set-upload-limits: + +Set Upload Limits +----------------- + +By default, the file size upload limit is 10737420000 bytes (~10.7 GB). + +If you want set this to a lower value, set the ``FILE_UPLOAD_MAX`` configuration +in the file apps' configuration file ``/etc/ood/config/apps/dashboard/env``. + +If you want to set it to a higher value set ``nginx_file_upload_max`` +in ``/etc/ood/config/nginx_stage.yml`` to the desired value. If you have +``FILE_UPLOAD_MAX`` set from above, unset it. + +If the values differ, the files app will choose the smaller of the two as the maximum +upload limit. + +.. warning:: + Both of these configurations are expected to be numbers only (no characters) + and in units of bytes. The default value of 10737420000 bytes is ~10.7 GB or ~10.0 Gib. + + Values like ``1000M`` or ``20G`` will not be accepted and may cause errors. + +If you want to disable file upload altogether, set ``FILE_UPLOAD_MAX`` to 0 and leave +the ``nginx_file_upload_max`` configuration alone (or comment it out so the default +is used). + +Set Download Limits +------------------- + +By default, the maximum file download size is 10.7 GB (10737418240 bytes). +If you wish to change this, you can set the ``OOD_DOWNLOAD_DIR_MAX`` configuration environment +variable in the ``/etc/ood/config/apps/dashboard/env`` file to the desired value in bytes. + +For example, to set the limit to 5 GB, you can add the following line to the ``/etc/ood/config/apps/files/env`` file: + +.. code-block:: + + OOD_DOWNLOAD_DIR_MAX=5368709120 + +Note that this will limit the download size for all users of the Open OnDemand instance. + +.. warning:: + This configuration value is expected to be numbers only (no characters) + and in units of bytes. The default value of 10737420000 bytes is ~10.7 GB or ~10.0 Gib. + + Values like ``1000M`` or ``20G`` will not be accepted and may cause errors. + +.. _set-file-allowlist: + +Block or Allow Directory Access +------------------------------- + +By default, all directories are open and accessible through Open OnDemand (barring POSIX file permissions. Open OnDemand +can never read files the user cannot read). + +By setting a colon delimited `OOD_ALLOWLIST_PATH` environment variable, the Job Composer, File Editor, and Files app +respect the allowlist in the following manner: + +1. Users will be prevented from navigating to, uploading, downloading, viewing, or editing files that are not an eventual child of the allowlisted paths +2. Users will be prevented from copying a template directory from an arbitrary path in the Job Composer if the arbitrary path that is not an eventual child of the allowlisted paths +3. Users should not be able to get around this using symlinks + +We recommend setting this environment variable in ``/etc/ood/config/nginx_stage.yml`` as a YAML mapping (key value pairs) in the mapping (hash/dictionary) ``pun_custom_env`` i.e. below would a list that allows access to home directories, project space, and scratch space at OSC: + +.. code:: yaml + + pun_custom_env: + OOD_ALLOWLIST_PATH: "/users:/fs/project:/fs/scratch" + +.. warning:: This allowlist is not enforced across every action a user can take in an app (including the developer views in the Dashboard). Also, it is enforced via the apps themselves, which is not as robust as using cgroups on the PUN. + +.. include:: customizations/disabling-users.inc + +.. _set-default-ssh-host: + +Set Default SSH Host +-------------------- + +.. warning:: + The shell app does not work out of the box because all SSH hosts have to be explicitly allowed + through the allowlist (see the section below). + + Because there are no hosts configured, no hosts are allowed. + +In ``/etc/ood/config/apps/shell/env`` set the env var ``OOD_DEFAULT_SSHHOST`` to change the default ssh host. +Since 1.8, there is no out of the box default (in previous versions it was 'localhost', but this has been removed). + +This will control what host the shell app ssh's to when the URL accessed is ``/pun/sys/shell/ssh/default`` which is the URL other apps will use (unless there is context to specify the cluster to ssh to). + +Since 1.8 you can also set the default ssh host in the cluster configuration as well. Simply add +default=true attribute to the login section like the example below. + +.. code-block:: yaml + + # /etc/ood/config/clusters.d/my_cluster.yml + --- + v2: + metadata: + title: "My Cluster" + login: + host: "my_cluster.my_center.edu" + default: true + +.. _set-ssh-allowlist: + +Set SSH Allowlist +----------------- + +In 1.8 and above we stopped allowing ssh access by default. Now you have explicitly set +what hosts users will be allowed to connect to in the shell application. + +Every cluster configuration with ``v2.login.host`` that is not hidden (it has +``v2.metadata.hidden`` attribute set to true) will be added to this allowlist. + +To add other hosts into the allow list (for example compute nodes) add the configuration +``OOD_SSHHOST_ALLOWLIST`` to the ``/etc/ood/config/apps/shell/env`` file. + +This configuration is expected to be a colon (:) separated list of GLOBs. + +Here's an example of of this configuration with three such GLOBs that allow for shell +access into any compute node in our three clusters. + +.. code:: shell + + # /etc/ood/config/apps/shell/env + OOD_SSHHOST_ALLOWLIST="r[0-1][0-9][0-9][0-9].ten.osc.edu:o[0-1][0-9][0-9][0-9].ten.osc.edu:p[0-1][0-9][0-9][0-9].ten.osc.edu" + +.. _enable-shell-ping-pong: + +Enable and configure Shell Ping Pong +------------------------------------ + +Version 3.1 added the ability for the shell application to send and receive ping pong +messages to keep the connection alive, and thus the terminal session alive. + +The drawback to this is that these persistant connections can actually outlive your +authentication timeout settings. Meaning users can have active shell sessions for much +longer than your authentication systems would normally allow. This is because the +connection was made while you were authenticated and it persists after your session +has expired. + +So, to keep a conservative security posture, Open OnDemand disables ping pongs by +default letting apache timeout these connections more freely. + +In addition to enabling or disabling ping pongs, there are other settings you may wish +to change. + +All of these configurations are environment variables are to be placed in +``/etc/ood/config/apps/shell/env``. + +Setting ``OOD_SHELL_PING_PONG`` to anything will enable ping pongs. Removing it or +commenting it out will disable ping pongs (it's disabled by default). + +``OOD_SHELL_INACTIVE_TIMEOUT_MS`` controls how long a connection can be inactive +for (in milliseconds) before being closed. It defaults to 300000 milliseconds (5 minutes). + +``OOD_SHELL_MAX_DURATION_MS`` controls how long a connection can exist regardless +of activity (in milliseconds). After this duration, the connection will be closed +regardless of activity. It's default is 3600000 milliseconds (1 hour). + +.. code:: shell + + # /etc/ood/config/apps/shell/env + + OOD_SHELL_INACTIVE_TIMEOUT_MS=300000 + OOD_SHELL_MAX_DURATION_MS=3600000 + # OOD_SHELL_PING_PONG=false + + +Set OOD SSH Port +----------------- + +As of version 2.1 you are allowed to configure a non-standard ssh port. + +To change the ssh port for submitting jobs in OOD, you need to add the configuration +``OOD_SSH_PORT`` to the ``/etc/ood/config/apps/dashboard/env`` file. + +Here's an example of of this configuration. + +.. code:: shell + + # /etc/ood/config/apps/dashboard/env + OOD_SSH_PORT="2222" + + +Shell App SSH Command Wrapper +----------------------------- + +.. _ssh-wrapper: + +Since OOD 1.7 you can use an ssh wrapper script in the shell application instead of just the ssh command. + +This is helpful when you pass add additional environment variable through ssh (``-o SendEnv=MY_ENV_VAR``) or ensure some ssh command options be used. + +To use your ssh wrapper configure ``OOD_SSH_WRAPPER=/usr/bin/changeme`` to point to your script in ``/etc/ood/config/apps/shell/env``. Also be sure to make your script executable. + +Here's a simple example of what a wrapper script could look like. + +.. code:: shell + + #!/bin/bash + + args="-o SendEnv=MY_ENV_VAR" + + exec /usr/bin/ssh "$args" "$@" + +Fix Unauthorized WebSocket Connection in Shell App +-------------------------------------------------- + +If you see a 401 error when attempting to launch a Shell app session, where the request URL starts with wss:// and the response header includes ``X-OOD-Failure-Reason: invalid origin``, you may need to set the ``OOD_SHELL_ORIGIN_CHECK`` configuration option. + +There is a security feature that adds proper CSRF_ protection using both the Origin request header check and a CSRF_ token check. + +The Origin check uses X-Forwarded-Proto_ and X-Forwarded-Host_ that Apache mod_proxy_ sets to build the string that is used to compare with the Origin request header the browser sends in the WebSocket upgrade request. + +In some edge cases this string may not be correct, and as a result valid WebSocket connections will be denied. In this case you can either set ``OOD_SHELL_ORIGIN_CHECK`` env var to the correct https string, or disable the origin check altogether by setting ``OOD_SHELL_ORIGIN_CHECK=off`` (or any other value that does not start with "http") in the ``/etc/ood/config/apps/shell/env`` file. + +Either way the CSRF token will still provide protection from this vulnerability. + +.. code:: text + + # /etc/ood/config/apps/shell/env + # to disable it, just configure it with something that doesn't start with http + OOD_SHELL_ORIGIN_CHECK='off' + + # to change it simply specify the http(s) origin you want to verify against. + OOD_SHELL_ORIGIN_CHECK='https://my.other.origin' + +.. _CSRF: https://owasp.org/www-community/attacks/csrf +.. _X-Forwarded-Proto: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-Forwarded-Proto +.. _X-Forwarded-Host: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-Forwarded-Host +.. _mod_proxy: https://httpd.apache.org/docs/2.4/mod/mod_proxy.html + +Custom Job Composer Templates +----------------------------- + +Below explains how job templates work for the Job Composer and how you can add your own. `Here is an example of the templates we use at OSC for the various clusters we have `_ + + +Job Templates Overview +...................... + +"Job Composer" attempts to model a simple but common workflow. When creating a new batch job to run a simulation a user may: + +1. copy the directory of a job they already ran or an example job +2. edit the files +3. submit a new job + +"Job Composer" implements these steps by providing the user job template directories and the ability to make copies of them: (1) Copy a directory, (2) Edit the files, and (3) Submit a new job. + +1. Copy a directory of a job already ran or an example job + + 1. User can create a new job from a "default" template. A custom default template can be defined at ``/etc/ood/config/apps/myjobs/templates/default`` or under the app deployment directory at ``/var/www/ood/apps/sys/myjobs/templates/default``. If no default template is specified, the default is ``/var/www/ood/apps/sys/myjobs/example_templates/torque`` + 2. user can select a directory to copy from a list of "System" templates the admin copied to ``/etc/ood/config/apps/myjobs/templates`` or under the app deployment directory at ``/var/www/ood/apps/sys/myjobs/templates`` during installation + 3. user can select a directory to copy from a list of "User" templates that the user has copied to ``$HOME/ondemand/data/sys/myjobs/templates`` + 4. user can select a job directory to copy that they already created through "Job Composer" from ``$HOME/ondemand/data/sys/myjobs/projects/default`` + +2. Edit the files + + 1. user can open the copied job directory in the File Explorer and edit files using the File Editor + +3. Submit a new job + + 1. user can use the Job Options form specify which host to submit to, what file is the job script + 2. user can use the web interface to submit the job to the batch system + 3. after the job is completed, the user can open the directory in the file explorer to view results + +Job Template Details +.................... + +A template consists of a folder and a `manifest.yml` file. + +The folder contains files and scripts related to the job. + +The manifest contains additional metadata about a job, such as a name, the default host, the submit script file name, and any notes about the template. + +.. code:: yaml + + name: A Template Name + host: ruby + script: ruby.sh + notes: Notes about the template, such as content and function. + +In the event that a job is created from a template that is missing from the `manifest.yml`, "Job Composer" will assign the following default values: + +- ``name`` The name of the template folder. +- ``host`` The cluster id of the first cluster with a valid resource_mgr listed in the OOD cluster config +- ``script`` The first ``.sh`` file appearing in the template folder. +- ``notes`` The path to the location where a template manifest should be located. + +Job Composer Script Size Limit +------------------------------ + +Since 1.7 the Job composer shows users 'Suggested file(s)' and 'Other valid file(s)'. Other valid files are +_any_ files less than ``OOD_MAX_SCRIPT_SIZE_KB`` which defaults to 65 (meaning 65kb). + +To reconfigure this, simply set the environment variable in the job composers' env file +``/etc/ood/config/apps/myjobs/env`` like so: + +.. code:: sh + + # show any file less than or equal to 15 kb + OOD_MAX_SCRIPT_SIZE_KB=15 + +Hiding Job Arrays +------------------------------ + +When composing a new job, the job arrays field is shown on supported clusters. To Hide this field even on +supported clusters, an option was added. + +To reconfigure this, simply set the environment variable in the job composers' env file +``/etc/ood/config/apps/myjobs/env`` like so: + +.. code:: sh + + # Don't show job arrays field even on supported clusters + OOD_HIDE_JOB_ARRAYS=True + +Custom Error Page for Missing Home Directory on Launch +------------------------------------------------------ + +Some sites have the home directory auto-create on first ssh login, for example +via ``pam_mkhomedir.so``. This introduces a problem if users first access the system +through OnDemand, which expects the existence of a user’s home directory. + +In OnDemand <= 1.3 if the user's home directory was missing a non-helpful single +string error would display. Now a friendly error page displays. This error page +can be customized by adding a custom one to ``/etc/ood/config/pun/html/missing_home_directory.html``. + +The default error page looks like this: + +.. figure:: /images/customization_homedirmissing_default.png + :align: center + +An example of a custom error page has been provided at ``/opt/ood/nginx_stage/html/missing_home_directory.html.example.pam_mkhomedir`` and can be copied to ``/etc/ood/config/pun/html/missing_home_directory.html``. This example directs the user to first click a link to open the shell app which will create the home directory. The shell app's default host must be configured to be a host that is appropriate for this purpose. The custom error page looks like this: + +.. figure:: /images/customization_homedirmissing_pammkdir.png + :align: center + + + +See `this Discourse discussion `_ for details. + +.. _dashboard_pinned_apps: + +Pinning Applications to the Dashboard +------------------------------------- + +In version 2.0 you can now pin app Icons to the dashboard that link to the application form. + +When configured a widget like the one below will appear on the dashboard's landing page. + +.. figure:: /images/pinned_apps.png + +The configuration for what apps to pin allows for three variants. + +You can configure specific apps with a string of the type ``router/app_name``. +For example ``sys/jupyter`` is the system installed app named jupyter. + +Secondly you can configure globs like ``sys/*`` to pin all system installed apps. Or +Maybe ``sys/minimal_*`` to pin all system installed apps that begin with 'minimal'. + +Lastly you can choose to pin apps based off of fields in their ``manifest.yml`` file. +You can match by type, category, subcategory and metadata fields. These matches are +cumulative. Meaning an app has to match *all* of these to be pinned. In the examples below +there is a configuration of type sys and category minimal. This configuration will only pin +system installed apps that are in the minimal category. An app has to meet *both* these +criteria to be pinned to the dashboard. + + +Full examples are below: + +.. code:: yaml + + # /etc/ood/config/ondemand.d/ondemand.yml + pinned_apps: + - sys/jupyter # pin a specific system installed app called 'jupyter' + + - sys/bc_desktop/desk1 # pinned desktop app must contain exact desktop name - 'desk1' + + - 'sys/*' # pin all system install apps. This also works for usr/* and dev/* + + - category: 'minimal' # pin all the apps in the 'minimal' category + + - type: sys # pin all system installed apps in the minimal category. + category: 'minimal' + + # pin all system installed apps in the minimal category and the + # class instruction subcategory + - type: sys + category: 'minimal' + subcategory: 'class_instruction' + + # pin all system installed apps in the minimal category, the + # class instruction subcategory and the metadata field 'field_of_science' + # with an exact match on biology + - type: sys + category: 'minimal' + subcategory: 'class_instruction' + field_of_science: 'biology' + + # pin any app with an exact match on the metadata field_of_science of biology + - field_of_science: 'biology' + + # pin any app with a glob match *bio* on the metadata field_of_science + - field_of_science: '*bio*' + + +Administrators can also configure the pinned apps to be grouped by any field +in the ``manifest.yml`` including metadata fields with the ``pinned_apps_group_by`` +configuration. + +This will create a row and a heading for each group like so (the image was generated +from grouping by category): + +.. figure:: /images/grouped_pinned_apps.png + +One can also change the menu length in the 'App's menu item. If you've +pinned more than 6 apps and you want to them to show up in this dropdown +list, simply increase the length with the option below. + +.. code:: yaml + + # /etc/ood/config/ondemand.d/ondemand.yml + pinned_apps_menu_length: 6 # the default number of items in the dropdown menu list + pinned_apps_group_by: category # defaults to nil, no grouping + +Pinned Apps customizations +.......................... + +To customize the text, icon, or color of the pinned app tile, +use the ``tile`` configuration property in the application ``manifest`` or ``form``. +The ``form`` values will take precedence over any set in the ``manifest``. +All the values are optional and any set will override the default from the application. + +.. code:: yaml + + tile: + title: "Custom Title" + icon: fa://desktop + border_color: "red" + sub_caption: | + Custom Text Line 1 + Text Line 2 + Text Line 3 + +The CSS for the pinned app tiles has been optimized to support upto three lines of text for the ``sub_caption`` property. + +.. figure:: /images/custom_pinned_apps.png + +.. _dashboard_custom_layout: + +Custom layouts in the dashboard +------------------------------- + +Administrators can now customize what widgets appear on the dashboard and how they're +laid out on the page. + +In it's simplest form this feature allows for a rearrangement of existing widgets. As +of 2.1 the existing widgets are: + +- ``pinned_apps`` - Pinned Apps described above +- ``recently_used_apps`` - the four most recently used interactive applications. + Launching these applications will start a new interactive session with the previously submitted parameters. +- ``sessions`` - the three most recent active interactive sessions +- ``motd`` - the Message of the Day +- ``xdmod_widget_job_efficiency`` - the XDMoD widget for job efficiency +- ``xdmod_widget_jobs`` - the XDMoD widget for job information + +This feature also allows for administrators to *add* custom widgets. +Simply drop new files into ``/etc/ood/config/apps/dashboard/views/widgets`` and reference them +in the configuration. These partial files can be any format Rails recognizes, notably ``.html`` or +``.html.erb`` extensions. + +Also if you use subdirectories under widgets, they can be referenced by relative paths. For example +``views/widgets/cluster/_my_cluster_widget.html.erb`` would be referenced in the configuration +as ``cluster/my_cluster_widget``. + +.. warning:: + + Rails expects files to be prefixed with an underscore. For example if you configured ``my_new_widget`` + the filename should be ``_my_new_widget.html``. + +Without setting this configuration, the dashboard will arrange itself depending on what features are +enabled. For example if both pinned apps and XDMoD features are enabled it will arrange itself accordingly +based on a default layout. + +Here's the default configuration when all of these features are enabled. + +.. code:: yaml + + # /etc/ood/config/ondemand.d/ondemand.yml + dashboard_layout: + rows: + - columns: + - width: 8 + widgets: + - pinned_apps + - motd + - width: 4 + widgets: + - xdmod_widget_job_efficiency + - xdmod_widget_jobs + +``rows`` are an array of row elements. Each row element has a ``columns`` field which is an array +column elements. Each column element two fields. A ``width`` field that specifies the width in the +`bootstrap grid layout`_ which defaults to 12 columns in total. It also has a ``widgets`` field which +is an array of existing or newly added widgets to render in that column. + +.. _bootstrap grid layout: https://getbootstrap.com/docs/4.0/layout/grid/ + +.. _customization_localization: + +Customize Text in OnDemand +-------------------------- + +Using Rails support for Internationalization (i18n), we have internationalized many strings in the Dashboard and the Job Composer apps. + +Initial translation dictionary files with defaults that work well for OSC and using the English locale (``en``) have been added (``/var/www/ood/apps/sys/dashboard/config/locales/en.yml`` and ``/var/www/ood/apps/sys/myjobs/config/locales/en.yml``). Sites wishing to modify these strings in order to provide site specific replacements for English, or use a different locale altogether, should do the following: + +#. Copy the translation dictionary file (or create a new file with the same structure of the keys you want to modify) to ``/etc/ood/config/locales/en.yml`` and modify that copy. +#. If you want apps to look for these dictionary files in a different location than ``/etc/ood/config/locales/en.yml`` you can change the location by defining ``OOD_LOCALES_ROOT`` environment variable. +#. The default locale is "en". You can use a custom locale. For example, if you want the locale to be French, you can create a ``/etc/ood/config/locales/fr.yml`` and then configure the Dashboard to use this locale by setting the environment variable ``OOD_LOCALE=fr`` where the locale is just the name of the file without the extension. Do this in either the nginx_stage config or in the Dashboard and Job Composer env config file. + +In each default translation dictionary file the values that are most site-specific (and thus relevant for change) appear at the top. + +.. list-table:: OnDemand Locale Files + :header-rows: 1 + :stub-columns: 1 + + * - File path + - App + - Translation namespace + * - ``/var/www/ood/apps/sys/dashboard/config/locales/en.yml`` + - `Dashboard`_ + - ``dashboard`` + * - ``/var/www/ood/apps/sys/myjobs/config/locales/en.yml`` + - `Job Composer`_ + - ``jobcomposer`` + * - ``/etc/ood/config/locales/en.yml`` + - All localizable apps will check this path, unless ``OOD_LOCALES_ROOT`` is set. + - Any + +.. warning:: + + Translations have certain variables passed to them for example ``%{support_url}``. Those variables may be used or removed from the translation. Attempting to use a variable that is not available to the translation will crash the application. + +.. note:: + + Localization files are YAML documents; remember that YAML uses spaces for indentation NOT tabs per the `YAML spec`_. + +.. note:: + + OnDemand uses the convention that translations that accept HTML with be suffixed with ``_html``. Any other translation will be displayed as plain text. + +.. Links for the OnDemand 1.7.0 release versions of these apps +.. _Dashboard: https://github.com/OSC/ondemand/blob/master/apps/dashboard/config/locales/en.yml +.. _Job Composer: https://github.com/OSC/ondemand/blob/master/apps/myjobs/config/locales/en.yml + +.. _Yaml spec: https://yaml.org/spec/1.2/spec.html#id2777534 + +Change the Dashboard Tagline +............................ + +.. code-block:: yaml + + en: + dashboard: + welcome_html: | + %{logo_img_tag} +

OnDemand provides an integrated, single access point for all of your HPC resources.

+ motd_title: "Message of the Day" + +The ``welcome_html`` interpolates the variable ``logo_img_tag`` with the default +logo, or the logo specified by the environment variable ``OOD_DASHBOARD_LOGO``. + +You may omit this variable in the value you specify for ``welcome_html`` if you prefer. + +Change quota messages in the Dashboard +....................................... + +Two messages related to file system usage that sites may want to change: + + - ``quota_additional_message`` - gives the user advice on what to do if they see a quota warning + - ``quota_reload_message`` - tells the user that they should reload the page to see their quota usage change, and by default also tells users that the quota values are updated every 5 minutes + +Customize Text in the Job Composer's options form +................................................. + +The OSC-default value for ``options_account_help`` says that the account field is optional unless a user is a member of multiple projects. + +Items of note include what to call Accounts which might also be Charge Codes, or Projects. At OSC entering an account is optional unless a user is a member of multiple projects which is reflected in the default value for the string ``options_account_help``. + +Disk Quota Warnings on Dashboard +-------------------------------- + +You can display warnings to users on the Dashboard if their +disk quota is nearing its limit. This requires an auto-updated (it is +recommended to update this file every **5 minutes** with a cronjob) JSON file +that lists all user quotas. The JSON schema for version `1` is given as: + +.. code:: json + + { + "version": 1, + "timestamp": 1525361263, + "quotas": [ + {}, + {} + ] + } + +Where ``version`` defines the version of the JSON schema used, ``timestamp`` +defines when this file was generated, and ``quotas`` is a list of quota objects +(see below). + +You can configure the Dashboard to use this JSON file (or files) by setting the +environment variable ``OOD_QUOTA_PATH`` as a colon-delimited list of all JSON +file paths in the ``/etc/ood/config/apps/dashboard/env`` file. In addition to +pointing to files ``OOD_QUOTA_PATH`` may also contain HTTP(s) or FTP protocol +URLs. Colons used in URLs are correctly handled and are not treated as delimiters. + +.. warning:: + + Sites using HTTP(s) or FTP for their quota files may see slower dashboard load + times, depending on the responsiveness of the server providing the quota file(s). + +The default threshold for displaying the warning is at 95% (`0.95`), but this +can be changed with the environment variable ``OOD_QUOTA_THRESHOLD``. + +An example is given as: + +.. code:: sh + + # /etc/ood/config/apps/dashboard/env + + OOD_QUOTA_PATH="/path/to/quota1.json:https://example.com/quota2.json" + OOD_QUOTA_THRESHOLD="0.80" + + +Individual User Quota +..................... + +If the quota is defined as a ``user`` quota, then it applies to only disk +resources used by the user alone. This is the default type of quota object and +is given in the following format: + +.. code:: json + + { + "type": "user", + "block_limit": 5000000, + "file_limit": 1000000, + "path": "/path/to/volume2", + "total_block_usage": 400000, + "total_file_usage": 1000, + "user": "user1" + } + + +.. warning:: A block must be equal to 1 KiB for proper conversions. + + +Individual Fileset Quota +........................ + +If the quota is defined as a ``fileset`` quota, then it applies to all disk +resources used underneath a given volume. This requires the object to be +repeated for **each user** that uses disk resources under this given volume. +The format is given as: + +.. code:: json + + { + "type": "fileset", + "user": "user1", + "path": "/path/to/volume2", + "block_usage": 500, + "total_block_usage": 1000, + "block_limit": 2000, + "file_usage": 1, + "total_file_usage": 5, + "file_limit": 10 + } + +Where ``block_usage`` and ``file_usage`` are the disk resource usages attributed to +the specified user only. + +.. note:: For each user with resources under this fileset, the above object will be repeated with just ``user``, ``block_usage``, and ``file_usage`` changing. + + +.. _balance-warnings-on-dashboard: + +Balance Warnings on Dashboard +-------------------------------- + +You can display warnings to users on the Dashboard if their +resource balance is nearing its limit. This requires an auto-updated (it is +recommended to update this file daily with a cronjob) JSON file +that lists all user balances. The JSON schema for version `1` is given as: + +.. code:: json + + { + "version": 1, + "timestamp": 1525361263, + "config": { + "unit": "RU", + "project_type": "project" + }, + "balances": [ + {}, + {} + ] + } + +Where ``version`` defines the version of the JSON schema used, ``timestamp`` +defines when this file was generated, and ``balances`` is a list of quota objects +(see below). + +The value for ``config.unit`` defines the type of units for balances and +``config.project_type`` would be project, account, or group, etc. +Both values are used in locales and can be any string value. + +You can configure the Dashboard to use this JSON file (or files) by setting the +environment variable ``OOD_BALANCE_PATH`` as a colon-delimited list of all JSON +file paths. + +.. warning:: + + Sites using HTTP(s) or FTP for their balance files may see slower dashboard load + times, depending on the responsiveness of the server providing the quota file(s). + +The default threshold for displaying the warning is at ``0``, but this +can be changed with the environment variable ``OOD_BALANCE_THRESHOLD``. + +An example is given as: + +.. code:: sh + + # /etc/ood/config/apps/dashboard/env + + OOD_BALANCE_PATH="/path/to/balance1.json:/path/to/balance2.json" + OOD_BALANCE_THRESHOLD=1000 + +User Balance +............ + +If the balance is defined as a ``user`` balance, then it applies to only that user. Omit the ``project`` key: + +.. code:: json + + { + "user": "user1", + "value": 10 + } + +Project Balance +............... + +If the balance is defined as a ``project`` balance, then it applies to a project/account/group, whatever is defined for ``config.project_type``: + +.. code:: json + + { + "user": "user1", + "project": "project1", + "value": 10 + } + + +.. _maintenance-mode: + +Maintenance Mode +----------------- + + +As an administrator you may want to have some downtime of the Open OnDemand service for various reasons, +while still telling your customers that the downtime is expected. + +You can do this by setting Open OnDemand in 'Maintenance Mode'. +While Maintenance mode is active, Apache will not serve requests for paths outside the +``/public/maintenance/*`` wildcard. Instead, it will serve the ``/var/www/ood/public/maintenance/index.html`` +file, which you can change or brand to be your own. Changes to this file will persist through upgrades. +Any assets (e.g., images, stylesheets, or javascript) needed by the HTML file should be placed +in the ``/var/www/ood/public/maintenance/`` directory. You can also put symbolic links into the +``/var/www/ood/public/maintenance/`` directory, if you want to reuse assets located elsewhere in your +file system. + +While in maintenance mode, Apache returns the HTML file and a 503 response code to all users whose +IP does not match one of the configured allowlist regular expressions. +The allowlist is to allow staff, localhost or a subset of your users access while restricting others. + +In this example we allow access to anyone from ``192.168.1..*`` which is the 192.168.1.0/24 CIDR and +the single IP '10.0.0.1'. + +These are the settings you'll need for this functionality. + +.. code:: yaml + + # /etc/ood/config/ood_portal.yml + use_rewrites: true + use_maintenance: true + maintenance_ip_allowlist: + # examples only! Your ip regular expressions will be specific to your site. + - '192.168.1..*' + - '10.0.0.1' + +To start maintenance mode (and thus start serving this page) simply ``touch /etc/ood/maintenance.enable`` +to create the necessary file. When your downtime is complete just remove the file and all the +traffic will be served normally again. The existence of this file is what starts or stops maintenance +mode, not it's content, so you will not need to restart apache or modify it's config files for this to +take affect. + + +.. _grafana-support: + +Grafana support +--------------- + +It's possible to display Grafana graphs within the ActiveJobs app when a user expands a given job. + +Grafana must be configured to support embedded panels and at this time it is also required to have an anonymous organization. Below are configuration options are needed to support displaying Grafana panels in ActiveJobs. Adjust `org_name` to match whatever organization you wish to be anonymous. + +.. warning:: + + Changing a Grafana install to support anonymous access can cause unintended consequences for how authenticated users interact with Grafana. + It's recommended to test anonymous access on a non-production Grafana install if you do not already support anonymous access. + +.. code:: shell + + [auth.anonymous] + enabled = true + org_name = Public + org_role = Viewer + + [security] + allow_embedding = true + +The dashboard used by OSC is the `OnDemand Clusters `_ dashboard. + +Settings used to access Grafana are configured in the cluster config. The following is an example from OSC: + +.. code:: yaml + + custom: + grafana: + host: "https://grafana.osc.edu" + orgId: 3 + dashboard: + name: "ondemand-clusters" + uid: "aaba6Ahbauquag" + panels: + cpu: 20 + memory: 24 + labels: + cluster: "cluster" + host: "host" + jobid: "jobid" + cluster_override: "mysite" + +When viewing a dashboard in Grafana choose the panel you'd wish to display and select `Share`. +Then choose the `Embed` tab which will provide you with the iframe URL that will need to be generated within OnDemand. +The time ranges and values for labels (eg: `var-cluster=`) will be autofilled by OnDemand. + +* ``orgId`` is the ``orgId`` query parameter +* The dashboard ``name`` is the last segment of the URI before query parameters +* The ``uid`` is the UID portion of URL that is unique to every dashboard +* The ``panelId`` query parameter will be used as the value for either ``cpu`` or ``memory`` depending on the panel you have selected +* The values for ``labels`` are how OnDemand maps labels in Grafana to values expected in OnDemand. The ``jobid`` key is optional, the others are required. +* The ``cluster_override`` can override the cluster name used to make requests to Grafana if the Grafana cluster name varies from OnDemand cluster name. + +.. _disable-host-link-batch-connect: + +Disable Host Link in Batch Connect Session Card +----------------------------------------------- + +Batch connect session cards like this have links to the compute node on which the job is currently running (highlighted). + +.. figure:: /images/bc-card-w-hostlink.png + :align: center + +However, some sites may want to disable this feature because they do not allow ssh sessions on the compute +nodes. + +To disable this, simply set the environment variable in the dashboards' env file +``/etc/ood/config/apps/dashboard/env`` to a falsy value (0, false, off). + +.. code:: sh + + # don't show ssh link in batch connect card + OOD_BC_SSH_TO_COMPUTE_NODE=off + +If you wish to disable on a per-cluster basis, you can set the following in your :ref:`cluster YAML configuration `. + +.. code-block:: yaml + :emphasize-lines: 3- + + v2: + # ... + batch_connect: + ssh_allow: false + +.. _set-illegal-job-name-characters: + +Set Illegal Job Name Characters +------------------------------- + +If you encounter an issue in running batch connect applications complaining about invalid +job names like the error below. + +``Unable to read script file because of error: ERROR! argument to -N option must not contain /`` + +To resolve this set ``OOD_JOB_NAME_ILLEGAL_CHARS`` to ``/`` for all OOD applications in the +``pun_custom_env`` attribute of the ``/etc/ood/config/nginx_stage.yml`` file. + +.. code-block:: yaml + + # /etc/ood/config/nginx_stage.yml + pun_custom_env: + OOD_JOB_NAME_ILLEGAL_CHARS: "/" + +.. _customize_dex_theme: + +Customize Dex Theme +------------------- + +It's possible to use a customized theme when authenticating with Dex when using OnDemand's default authentication. +Refer to the upstream `Dex template docs`_ for additional information on templating Dex. + +The simplest approach is to copy the OnDemand theme and make changes. This is idea if you wish to make the following changes: + +- Change navigation or login page logos +- Change favicon +- Change CSS styles + +.. code-block:: sh + + cp -r /usr/share/ondemand-dex/web/themes/ondemand /usr/share/ondemand-dex/web/themes/mycenter + +To update the theme you must modify ``/etc/ood/config/ood_portal.yml`` and regenerate the Dex configuration: + +.. code-block:: yaml + :emphasize-lines: 3- + + dex: + # ... + frontend: + theme: mycenter + +The default ``ondemand`` theme can also be configured using the following configuration keys within ``/etc/ood/config/ood_portal.yml``: + +.. code-block:: yaml + :emphasize-lines: 4- + + dex: + # ... + frontend: + issuer: "MyCenter OnDemand" + extra: + navLogo: "/public/nav-logo.png" + loginLogo: "/public/logo.png" + loginTitle: "Log in with your Center username and password" + loginButtonText: "Log in with your Center account" + usernamePlaceholder: "center-username" + passwordPlaceholder: "center-password" + loginAlertMessage: "Login services will be down during center maintenance between 8:00 AM EST and 10:00 AM EST" + loginAlertType: "warning" + +Changes are applied by running ``update_ood_portal`` and restarting the ``ondemand-dex`` service. + +.. code-block:: sh + + sudo /opt/ood/ood-portal-generator/sbin/update_ood_portal + sudo systemctl restart ondemand-dex.service + +.. _dex template docs: https://dexidp.io/docs/templates/ + +.. _xdmod_integration: + +XDMoD Integration +----------------- + + +XDMoD Integration requires XDMoD 9+, OnDemand 1.8+, and the ability to facilitate single sign on between the two services. Currently this has been demonstrated to work using OpenID Connect via Keycloak as well as a modified instance of Dex Identity Provider to support sessions. + +.. figure:: /images/customization_xdmod.png + :align: center + + Example of XDMoD Job Efficiency reports in the OnDemand Dashboard. + +Steps to enable the XDMoD reports in the OnDemand Dashboard: + +#. Configure OnDemand with XDMoD host URL in PUN /etc/ood/config/nginx_stage.yml + + .. code-block:: yaml + + pun_custom_env: + OOD_XDMOD_HOST: "https://xdmod.osc.edu" + +#. Add OnDemand host as domain to XDMoD portal settings for CORS /etc/xdmod/portal_settings.ini + + .. code-block:: none + + domains = "https://ondemand.osc.edu" + +#. Configure identity provider to include OnDemand host in HTTP `Content-Security-Policy for frame-ancestors `_ since OnDemand uses iFrames to trigger SSO with XDMoD when a user logs in. Below is what we ensured Content-Security-Policy header for frame-ancestors was set to when configuring Keycloak: + + .. code-block:: none + + frame-ancestors https://*.osc.edu 'self' + +#. If you want the XDMoD links in the OnDemand Job Composer you also need to configure OnDemand with XDMoD resource id in each cluster config. For example, in the hpctoolset the resource_id for the hpc cluster is 1 in XDMoD, so we modify /etc/ood/config/clusters.d/hpc.yml to add a xdmod map to the custom map at the bottom of the file: + + .. code-block:: yaml + :emphasize-lines: 10- + + v2: + metadata: + title: "HPC Cluster" + login: + host: "frontend" + job: + adapter: "slurm" + cluster: "hpc" + bin: "/usr/bin" + custom: + xdmod: + resource_id: 1 + +#. In the Job Composer, Open XDMoD job links will include a warning message that the job may not appear in XDMoD for up to 24 hours after the job completed. The message is to address the gap of time between the job appearing as completed in the Job Composer and the job appearing in Open XDMoD after the ingest and aggregation script is run. This message appears from the time the Job Composer becomes aware of the job completion status, till an elapsed time specified in seconds by the locale key ``en.jobcomposer.xdmod_url_warning_message_seconds_after_job_completion`` which defaults to 24 hours (86400 seconds) with a text message specified by locale key ``en.jobcomposer.xdmod_url_warning_message``. To disable this message, set the value you your locale file under ``/etc/ood/config/locales``. For example, in the default locale we have these values: + + .. code-block:: yaml + + en: + jobcomposer: + xdmod_url_warning_message: "This job may not appear in Open XDMoD until 24 hours after the completion of the job." + xdmod_url_warning_message_seconds_after_job_completion: 86400 + + + Which results in these warning messages appearing in Job Composer: + + .. figure:: /images/customization_xdmod_jobcomposer_warning_1.png + :align: center + .. figure:: /images/customization_xdmod_jobcomposer_warning_2.png + :align: center + + +.. _remote-file-systems: + +Accessing Remote File Systems +----------------------------- + +Since 2.1 you can use ``rclone`` to interact with remote file systems. Since +every command in Open OnDemand is issued *as the user*, the user themselves +are required to setup their ``rclone`` remotes. + +You can refer to the `OSC's rclone documentation`_ on how to configure rclone +remotes. + +To enable this feature ensure that ``rclone`` is installed on the same machine +that Open OnDemand is installed. You also have to enable the feature through +the :ref:`configuration entry for enabling remote filesystems `. + +Cancel Interactive Sessions +--------------------------- + +We can now cancel an interactive session from the session panel without deleting the session card. +This functionality will allow users to remove the job from the scheduler and keep the information in the OnDemand interface. + +This feature is disabled behind a feature toggle. To enable it, set the configuration property ``cancel_session_enabled: true``. +For more information on how to configure properties, see :ref:`configuration documentation `. + +When enabled, the cancel button will appear for active sessions. +When the session is cancelled, the job will be cancelled in the scheduler, +the status will change to ``completed``, and the session card will be kept. +For completed sessions, the system will only show the delete button. + +.. figure:: /images/cancel_session.png + :align: center + +.. include:: customizations/custom-pages.inc +.. include:: customizations/support-ticket.inc + +.. _OSC's rclone documentation: https://www.osc.edu/resources/getting_started/howto/howto_use_rclone_to_upload_data +.. _2.0 documentation for controlling the navbar: https://osc.github.io/ood-documentation/release-2.0/customization.html#control-which-apps-appear-in-the-dashboard-navbar diff --git a/downcase-ids/_sources/enable-desktops.rst.txt b/downcase-ids/_sources/enable-desktops.rst.txt new file mode 100644 index 000000000..91e4ec872 --- /dev/null +++ b/downcase-ids/_sources/enable-desktops.rst.txt @@ -0,0 +1,28 @@ +.. _enable-desktops: + +Enable Interactive Desktop +========================== + +This installation guide will walk you through setting up an Interactive Desktop +app that your users will be able to use to launch a Gnome 2, Mate, or Xfce +desktop on a compute node within your HPC cluster. The user should then be able +to connect to a running session through their browser using the `noVNC`_ +client. + +.. danger:: + + Confirm that you have walked through the + :ref:`app-development-interactive-setup` instructions for interactive apps + before continuing on. + +.. toctree:: + :maxdepth: 2 + :numbered: 1 + :caption: Quick Start Guide + + enable-desktops/software-requirements + enable-desktops/add-cluster + enable-desktops/modify-form-attributes + enable-desktops/custom-job-submission + +.. _novnc: http://novnc.com/info.html diff --git a/downcase-ids/_sources/enable-desktops/add-cluster.rst.txt b/downcase-ids/_sources/enable-desktops/add-cluster.rst.txt new file mode 100644 index 000000000..e49d35344 --- /dev/null +++ b/downcase-ids/_sources/enable-desktops/add-cluster.rst.txt @@ -0,0 +1,53 @@ +.. _enable-desktops-add-cluster: + +Add a Cluster +============= + +We now need to add this cluster as a Desktop option in the Interactive Apps +list. All customization is done underneath the root directory +:file:`/etc/ood/config/apps/bc_desktop` which requires escalated privileges to +modify. + +#. Start by creating the working directory: + + .. code-block:: sh + + mkdir -p /etc/ood/config/apps/bc_desktop + +#. For *each* cluster that we want to launch a Desktop session on we will need + a corresponding :ref:`app-development-interactive-form` configuration file + in YAML format located in this directory. + + Assuming we want to launch a desktop on OSC's Oakley cluster we would have: + + .. code-block:: yaml + + # /etc/ood/config/apps/bc_desktop/oakley.yml + --- + title: "Oakley Desktop" + cluster: "oakley" + + .. warning:: + + The ``cluster`` attribute above **MUST** match a valid cluster + configuration file located underneath + :file:`/etc/ood/config/clusters.d/`. + +#. Navigate to your OnDemand site, in particular the Dashboard App, and you + should see in the top dropdown menu "Interactive Apps" ⇒ "Oakley Desktop" + (or whatever you set as the ``title``). + + After choosing "Oakley Desktop" from the menu, you should be presented with + a form to "Launch" a Desktop session to the given cluster. + + Most likely if you click "Launch" it will fail miserably. That is because we + will need to configure the submission parameters for cluster's resource + manager. + + .. note:: + + If by some chance when you click "Launch" and it actually successfully + submits a job to your cluster, it is **highly** recommended that you + click the link under "Session ID:". This will open the file browser + underneath the working directory of the batch job. This will allow you to + read all the logs generated to help debug any issues that may crop up. diff --git a/downcase-ids/_sources/enable-desktops/custom-job-submission.rst.txt b/downcase-ids/_sources/enable-desktops/custom-job-submission.rst.txt new file mode 100644 index 000000000..afd03b120 --- /dev/null +++ b/downcase-ids/_sources/enable-desktops/custom-job-submission.rst.txt @@ -0,0 +1,239 @@ +.. _enable-desktops-custom-job-submission: + +Custom Job Submission +===================== + +The :ref:`app-development-interactive-submit` configuration file describes how +the batch job should be submitted to your cluster. The location of this file +**must** be specified in the respective +:file:`/etc/ood/config/apps/bc_desktop/{my_cluster}.yml` form configuration +file, so that when a user submits the form, the specified submission +configuration is used when submitting the batch job. + +To customize job submission we will need to first edit our custom desktop app +:ref:`app-development-interactive-form` YAML file as such: + +.. code-block:: yaml + :emphasize-lines: 5- + + # /etc/ood/config/apps/bc_desktop/my_cluster.yml + --- + title: "My Cluster Desktop" + cluster: "my_cluster" + submit: "submit/my_submit.yml.erb" + +Notice we included the configuration option ``submit`` that points to our +custom :ref:`app-development-interactive-submit` YAML configuration file. This +can be an absolute file path or a relative file path with respect to the +:file:`/etc/ood/config/apps/bc_desktop/` directory. It is important to notice +you must use this or some other directory outside the app's root. + +.. note:: + + The ``*.erb`` file extension will cause the YAML configuration file to be + processed using the `eRuby (Embedded Ruby)`_ templating system. This allows + you to embed Ruby code into the YAML configuration file for flow control, + variable substitution, and more. + +.. danger:: + + Do not put the :ref:`app-development-interactive-submit` configuration file + directly underneath :file:`/etc/ood/config/apps/bc_desktop`. If you do, OOD will think + this a different app's ``form.yml``. Instead we typically + create the directory :file:`submit/` underneath the app's root directory and put our + :ref:`app-development-interactive-submit` configuration files underneath + that. + +.. _eruby (embedded ruby): https://en.wikipedia.org/wiki/ERuby + +We can now create and modify the :ref:`app-development-interactive-submit` +configuration file at:: + + /etc/ood/config/apps/bc_desktop/submit/my_submit.yml.erb + +Since it has the extension ``.erb`` we can take advantage of the Ruby language +to make the configuration file dynamic. In particular, you will now have access +to the user-submitted form arguments defined as: + +bc_num_hours + *Default:* ``"1"`` + + A Ruby ``String`` containing the number of hours a user requested for the + Desktop batch job to run. + +bc_num_slots + *Default:* ``"1"`` + + A Ruby ``String`` containing either the number of nodes or processors + (depending on the type of resource manager the cluster uses) a user + requested. + +bc_account + *Default:* ``""`` + + A Ruby ``String`` that holds the account the user supplied to charge the job + against. + +bc_queue + *Default:* ``""`` + + A Ruby ``String`` that holds the queue the user requested for the job to run + on. + +bc_email_on_started + *Default:* ``"0"`` + + A Ruby ``String`` that can either be ``"0"`` (do not send the user an email + when the job starts) or ``"1"`` (send an email to the user when the job + starts). + +node_type + *Default:* ``""`` + + A Ruby ``String`` that can be used for more advanced job submission. This is + an advanced option that is disabled by default and does nothing if you do + enable it, unless you add it to a custom job submission configuration file. + +Some examples on how to submit jobs using the above form attributes are given +in the following sections for the given resource manager. + +Slurm +----- + +For most cases of Slurm you will want to modify how the ``bc_num_slots`` +(number of nodes) is submitted to the batch server. + +This can be handled in your custom job submission configuration file as such: + +.. code-block:: yaml + + # /etc/ood/config/apps/bc_desktop/submit/my_submit.yml.erb + --- + script: + native: + - "-N" + - "<%= bc_num_slots.blank? ? 1 : bc_num_slots.to_i %>" + +All `batch script options`_ are underneath the ``script`` configuration option. +In particular since there is no option to modify number of nodes, we need to +directly interact with the ``native`` command line arguments. This is specified +as an array of :command:`sbatch` arguments. + +.. note:: + + It is recommended you use the corresponding `batch script options`_ before + using the ``native`` fallback. + +Torque +------ + +For most cases of Torque you will want to modify how the ``bc_num_slots`` +(number of nodes) is submitted to the batch server. + +This can be handled in your custom job submission configuration file as such: + +.. code-block:: yaml + + # /etc/ood/config/apps/bc_desktop/submit/my_submit.yml.erb + --- + script: + native: + resources: + nodes: "<%= bc_num_slots.blank? ? 1 : bc_num_slots.to_i %>:ppn=28" + +All `batch script options`_ are underneath the ``script`` configuration option. +In particular since there is no option to modify number of nodes, we need to +directly interact with the ``native`` command line arguments. + +For more information on the available options for the ``native`` attribute +when using Torque please see the `pbs-ruby documentation`_. + +.. note:: + + It is recommended you use the corresponding `batch script options`_ before + using the ``native`` fallback. + +PBS Professional +---------------- + +For most cases of PBS Professional you will want to modify how the +``bc_num_slots`` (number of CPUs on a single node) is submitted to the batch +server. + +This can be handled in your custom job submission configuration file as such: + +.. code-block:: yaml + + # /etc/ood/config/apps/bc_desktop/submit/my_submit.yml.erb + --- + script: + native: + - "-l" + - "select=1:ncpus=<%= bc_num_slots.blank? ? 1 : bc_num_slots.to_i %>" + +All `batch script options`_ are underneath the ``script`` configuration option. +In particular since there is no option to modify number of nodes/cpus, we need +to directly interact with the ``native`` command line arguments. This is +specified as an array of :command:`qsub` arguments. + +If you would like to mimic how Torque handles ``bc_num_slots`` (number of +**nodes**), then we will first need to change the form label of +``bc_num_slots`` that the user sees in the form. This can be done by modifying +our Desktop app local YAML configuration file: + +.. code-block:: yaml + :emphasize-lines: 5-7 + + # /etc/ood/config/apps/bc_desktop/submit/my_submit.yml.erb + --- + title: "Cluster1 Desktop" + cluster: "cluster1" + attributes: + bc_num_slots: + label: "Number of nodes" + submit: "submit/my_submit.yml.erb" + +Now when we go to the Desktop app form in our browser it will have the new +label "Number of nodes" instead of "Number of CPUs on a single node". + +Next we will need to handle how we submit the ``bc_num_slots`` since it means +something different now. So now modify the job submission configuration file as +such: + +.. code-block:: yaml + + # /etc/ood/config/apps/bc_desktop/submit/my_submit.yml.erb + --- + script: + native: + - "-l" + - "select=<%= bc_num_slots.blank? ? 1 : bc_num_slots.to_i %>:ncpus=28" + +You can also append ``mem=...gb`` to the ``select=...`` statement if you'd +like. + +.. note:: + + It is recommended you use the corresponding `batch script options`_ before + using the ``native`` fallback. + +.. _batch script options: http://www.rubydoc.info/gems/ood_core/OodCore/Job/Script +.. _pbs-ruby documentation: http://www.rubydoc.info/gems/pbs/PBS/Batch#submit_script-instance_method + +LinuxHost Adapter +-------------------- + +If you're using the :ref:`resource-manager-linuxhost` you actually don't *need* a specialized +submit.yml.erb. There is no need to specify resources like the other adapters above. + +You can however, use it to override the adapter's global fields for mount binding and specifying +which container use. + +.. code-block:: yaml + + # /etc/ood/config/apps/bc_desktop/submit/linuxhost_submit.yml.erb + --- + batch_connect: + native: + singularity_bindpath: /etc,/media,/mnt,/opt,/run,/srv,/usr,/var,/fs,/home + singularity_container: /usr/local/modules/netbeans/netbeans_2019.sif diff --git a/downcase-ids/_sources/enable-desktops/modify-form-attributes.rst.txt b/downcase-ids/_sources/enable-desktops/modify-form-attributes.rst.txt new file mode 100644 index 000000000..c878a40b1 --- /dev/null +++ b/downcase-ids/_sources/enable-desktops/modify-form-attributes.rst.txt @@ -0,0 +1,312 @@ +.. _enable-desktops-modify-form-attributes: + +Modify Form Attributes +====================== + +In some cases you may want to modify the form presented to the user as well as +any other configurable options. Some examples: + +- Use an Xfce desktop instead of Mate desktop. +- Remove the "Queue" form field as your scheduler will auto select the correct + queue. +- Hard-code the "Number of nodes" to just 1, so that users can't launch + desktops with multiple nodes. +- Change the label for the form field "Account" to "Project". +- Add help text to a given form field. +- Change default value for a form field. + +The :ref:`app-development-interactive-form` YAML configuration file is +responsible for defining these form attributes and how they are presented to +the user. + +For each configuration file underneath:: + + /etc/ood/config/apps/bc_desktop/ + +a separate desktop app will be presented as an option to the user from the +dashboard, with the simplest :ref:`app-development-interactive-form` +configuration file for an Interactive Desktop app given as: + +.. code-block:: yaml + + # /etc/ood/config/apps/bc_desktop/my_cluster.yml + --- + title: "My Cluster Desktop" + cluster: "my_cluster" + +Before we begin modifying form attributes. Let us first take a look at the +default form definition located in the source file +:file:`/var/www/ood/apps/sys/bc_desktop/form.yml` (**do not modify**): + +.. code-block:: yaml + + # /var/www/ood/apps/sys/bc_desktop/form.yml + --- + attributes: + desktop: "mate" + bc_vnc_idle: 0 + bc_vnc_resolution: + required: true + node_type: null + form: + - bc_vnc_idle + - desktop + - bc_num_hours + - bc_num_slots + - node_type + - bc_account + - bc_queue + - bc_vnc_resolution + - bc_email_on_started + +The ``attributes`` and ``form`` configuration options can all be overridden in +our global YAML configuration file. But typically you will only modify the +``attributes`` options. + +In the following sections you will find common examples on how to override the +above options. + +.. warning:: + + The ``form`` configuration option defines all the available attributes as + well as the order they appear in the form (it is an array). + + Caution must be taken if you decide to override the ``form`` configuration + option. As this is an array, you can't simply prepend or append, you will + need to completely redefine it with your included modifications. + +Change to Xfce Desktop +---------------------- + +The default installation has the ``desktop`` attribute hard-coded to the value +``"mate"``. If you would like to change this to use ``"xfce"`` you can make the +following edits to your custom YAML configuration file: + +.. code-block:: yaml + :emphasize-lines: 5- + + # /etc/ood/config/apps/bc_desktop/my_cluster.yml + --- + title: "My Cluster Desktop" + cluster: "my_cluster" + attributes: + desktop: "xfce" + +And all Desktops will attempt to launch the Xfce desktop. + +.. note:: + + Whenever you hard-code a form attribute to a value like ``"xfce"`` in the + above case, no input field will appear in the form for the user to fill in. + So in the above case, the user cannot specify the ``desktop`` attribute in + the form because we hard-coded it. + +Remove Form Field +----------------- + +To remove a form field such as "Queue" defined under the attribute ``bc_queue`` +from the Desktop form you can make the following edits to your custom YAML +configuration file: + +.. code-block:: yaml + :emphasize-lines: 5- + + # /etc/ood/config/apps/bc_desktop/my_cluster.yml + --- + title: "My Cluster Desktop" + cluster: "my_cluster" + attributes: + bc_queue: null + +After refreshing the form in your browser you should not see the "Queue" field +anymore. + +Basically we are hard-coding the value of ``bc_queue`` to be the YAML type +``null``. And as we discussed in the previous example whenever you hard-code an +attribute, it will not show up in the form. + +.. warning:: + + If you have any + :ref:`enable-desktops-custom-job-submission` configuration files that use + this attribute, they will receive empty strings ``""``, so you will need to + test if they are blank before handling them. + +Hard-code a Form Field +---------------------- + +If we want to remove a form field but define its value to something other than +a blank string, we can set the attribute's value directly. + +For example, if you don't want users to submit Desktops with more than 1 node +under the attribute ``bc_num_slots``, you can make the following edits to your +custom YAML configuration file: + +.. code-block:: yaml + :emphasize-lines: 5- + + # /etc/ood/config/apps/bc_desktop/my_cluster.yml + --- + title: "My Cluster Desktop" + cluster: "my_cluster" + attributes: + bc_num_slots: 1 + +As in the previous two examples, since we are hard-coding the value of the +attribute, the form field will not show up and the user is unable to change +this value. For the above case, the attribute ``bc_num_slots`` will always +return ``"1"``. + +.. warning:: + + If you have any :ref:`enable-desktops-custom-job-submission` configuration + files that use this attribute, care must be taken when handling the + attribute as it will always come back as a `Ruby String`_. + + So if you hard-coded an attribute to the integer ``1`` it will come back as + the string ``"1"`` and if you perform any arithmetic operations on this + attribute it will require you convert this back to an integer with the + method ``String#to_i``. + +Change a Label +-------------- + +You are able to modify the label for a corresponding attribute that appears +above the input field in the form. + +For example, if you want to change the label for the "Account" form field given +by the ``bc_account`` attribute to instead display "Project". This can be +modified with the following edits to your custom YAML configuration file: + +.. code-block:: yaml + :emphasize-lines: 5- + + # /etc/ood/config/apps/bc_desktop/my_cluster.yml + --- + title: "My Cluster Desktop" + cluster: "my_cluster" + attributes: + bc_account: + label: "Project" + +The key here is that we are defining a hash for the ``bc_account`` attribute +instead of hard-coding it to a specific value. This means we will only override +the equivalent option for this attribute (for the above example we are +overriding the ``label`` option for the ``bc_account`` attribute). + +Now when you refresh the form in your browser, you should now see an input +field with the label "Project". + +.. warning:: + + If you have any :ref:`enable-desktops-custom-job-submission` configuration + files that use this attribute, changing the label of the attribute will not + affect the value received by the user upon form submission. + + But care must be taken that if by changing the label of the attribute you + also change the *meaning* of the attribute, then you may have to handle it + differently. For example, changing a label of "Number of processors" to + "Number of nodes" will have consequences on how you submit the job. + +Add Help Message to Field +------------------------- + +You are also able to add a help message to any given form field through its +corresponding attribute. + +For example, if you would like to add a help message to the attribute +``bc_account`` you can make the following edits to your custom YAML +configuration file: + +.. code-block:: yaml + :emphasize-lines: 5- + + # /etc/ood/config/apps/bc_desktop/my_cluster.yml + --- + title: "My Cluster Desktop" + cluster: "my_cluster" + attributes: + bc_account: + help: "You can leave this blank if **not** in multiple projects." + +The key here is that we are defining a hash for the ``bc_account`` attribute +instead of hard-coding it to a specific value. This means we will only override +the equivalent option for this attribute (for the above example we are +overriding the ``help`` option for the ``bc_account`` attribute). + +Now when you refresh the form in your browser, you should see the help message +below the "Account" form input field. + +.. note:: + + Help messages can be written in Markdown_ format, but it is best not to get + carried away in the size of the help message. + +Change Field Default Value +-------------------------- + +You are able to modify the default value of a form field for a given attribute, +which should not be confused with hard-coding a value for an attribute. + +For example, if you would like the form field "Number of hours" given by +``bc_num_hours`` to be ``8`` hours by default, but still allow the user to +change it then you can make the following edits in your custom YAML +configuration file: + +.. code-block:: yaml + :emphasize-lines: 5- + + # /etc/ood/config/apps/bc_desktop/my_cluster.yml + --- + title: "My Cluster Desktop" + cluster: "my_cluster" + attributes: + bc_num_hours: + value: 8 + +The key here is that we are defining a hash for the ``bc_num_hours`` attribute +instead of hard-coding it to a number. This means we want to override the +equivalent option for this attribute (for the above example we are overriding +the ``value`` option for the ``bc_num_hours`` attribute). + +Now when you refresh the desktop form in your browser, you should see a default +value of ``8`` in the "Number of hours" form field. + +.. note:: + + There is a possibility you may see a number other than ``8`` in the above + example. That is because the Interactive Apps tool built into the Dashboard + **remembers** your last successful app launch for a corresponding app. So + when you go back to the form page for that given app, it will auto-fill in + the form with your previous values. + + +Minimal LinuxHost Form +---------------------- + +Because the :ref:`resource-manager-linuxhost` is not like a traditional scheduler, +there are very few form items you'll need to add or have your users choose from. + +This is a minimal configuration to launch an XFCE desktop environment on a cluster +we call owens_login. You'll notice a lot of entries are null because they don't +really have any meaning in the LinuxHost Adapter. + +.. code-block:: yaml + :emphasize-lines: 5- + + # /etc/ood/config/apps/bc_desktop/owens_login_desktop.yml + --- + title: Owens Login XFCE desktop + description: This launches a XFCE desktop on an Owens login nodes. + cluster: owens_login + form: + - desktop + - bc_num_hours + attributes: + bc_num_hours: + value: 1 + desktop: "xfce" + + +.. _ruby string: https://ruby-doc.org/core-2.2.0/String.html +.. _markdown: https://en.wikipedia.org/wiki/Markdown diff --git a/downcase-ids/_sources/enable-desktops/software-requirements.rst.txt b/downcase-ids/_sources/enable-desktops/software-requirements.rst.txt new file mode 100644 index 000000000..3aec1a68a --- /dev/null +++ b/downcase-ids/_sources/enable-desktops/software-requirements.rst.txt @@ -0,0 +1,23 @@ +.. _enable-desktops-software-requirements: + +Software Requirements +===================== + +The interactive Desktop app requires a Desktop Environment be installed on the +nodes that the batch job is meant to run on, **NOT** the OnDemand node. + +The following desktops are currently supported: + +- `Xfce Desktop`_ 4+ +- `Mate Desktop`_ 1+ (**default**) +- `Gnome Desktop`_ 2 (currently we do not support Gnome 3) + +.. warning:: + + Do **NOT** install the Desktop Environment on the **OnDemand node**. The + above Desktop Environments are **ONLY** for the **compute or login nodes** you intend + on launching Desktops on within batch jobs. + +.. _gnome desktop: https://www.gnome.org/ +.. _mate desktop: https://mate-desktop.org/ +.. _xfce desktop: https://xfce.org/ diff --git a/downcase-ids/_sources/glossary.rst.txt b/downcase-ids/_sources/glossary.rst.txt new file mode 100644 index 000000000..2a2180464 --- /dev/null +++ b/downcase-ids/_sources/glossary.rst.txt @@ -0,0 +1,20 @@ +.. _glossary: + +Glossary +======== + + Cluster + Physical machines with a resource scheduler that users can submit jobs to. + + Compute-node + The machine where a submitted job runs. Part of a cluster. + + Login-node + A server on the compute cluster that can be anything from the OOD dashboard itself, to a shell, to the file-browser app, etc. + Runs on the cluster but is not a compute-node. + + Web-node + A term used for when a site or institution has enough funds/personel to run the OOD login on a dedicated server and not on a login-node. + + PUN + The Per User Nginx. An Nginx instance running as the user. diff --git a/downcase-ids/_sources/how-tos/analytics/google-analytics.rst.txt b/downcase-ids/_sources/how-tos/analytics/google-analytics.rst.txt new file mode 100644 index 000000000..f33c25880 --- /dev/null +++ b/downcase-ids/_sources/how-tos/analytics/google-analytics.rst.txt @@ -0,0 +1,115 @@ +.. _google-analytics: + +Adding Google Analytics +======================== + +.. _Google service accounts: https://cloud.google.com/iam/docs/service-accounts +.. _Google IAM roles: https://cloud.google.com/iam/docs/understanding-roles +.. _Google Analytics: https://analytics.google.com/analytics/web + +If you wish you can setup your Open-OnDemand instance to send usage data to Google Analytics +(GA) that you can then query and report on, this page walks through how to do just that. + +.. note:: + + You'll need to have a `Google Analytics`_ account setup as a prerequisite to this. + + To query GA You'll need to have to have a service account setup with the appropriate permissions. See + info on `Google service accounts`_ and `Google IAM roles`_ for more details on that. + + +Configure Open OnDemand +-------------------------- + +Refer to the :ref:`ondemand.d configuration property google_analytics_tag_id ` +on how to configure this feature. + + +Querying Google Analytics +--------------------------- + +.. _GA client libraries: https://developers.google.com/analytics/devguides/reporting/core/v3/libraries + +.. warning:: + This documentation is for GA version 3. With the newer versions of GA this may not + work as intended. As OSC does not use GA we're unable to update these examples + ourselves, but would accept updates for the same. + +Now that you have Open-OnDemand sending information to GA and it's all configured correctly, +you can now query GA for this information, parse it and present it in any fashion you like. + +Here's a small portion of how we query GA in ruby, but there are many `GA client libraries`_ +available. + +This example is not complete and is only meant to illustrate how to query GA given the defined +metric set above. Let's go through each of these things. + +.. code-block:: ruby + + # Dimensions - here we want dimensions 1, 3 and something called pagePath which is the web + # page requested. pagePath is a google predefined dimension that we populated. Dimensions 1 + # and 3 were created above and are the username and timestamp (this is why the order in + # which they're defined is important). + DIMENSIONS = %w( + ga:dimension3 + ga:dimension1 + ga:pagePath + ) + + # we only want to report the hit metrics + METRICS = %w( + ga:hits + ) + + # First we specify the host so that we only get metrics from a specific host. Secondly, + # we filter only only 200 responses (dimension6 is status code) and we don't want to + # report on file editor edits. + FILTERS = %W( + ga:hostname==#{HOST};ga:dimension6==200;ga:pagePath!=/pun/sys/file-editor/edit + ) + + # now we can create our analytics object and make the query + analytics = Google::Apis::AnalyticsV3::AnalyticsService.new + + # Here we query for the data that we want. A lot of things are omitted in this example + # for brevity like START_DATE (dynamic query times like the first day of the month) + # or GA_PROFILE (part of our credentials). And the fact that this is in a loop paginating + # the results, updating 'start_index' and only requesting STEP_SIZE (10,000 in our case) + # results at a time. + results = analytics.get_ga_data( + "ga:#{GA_PROFILE}", + START_DATE, + END_DATE, + METRICS.join(','), + dimensions: DIMENSIONS.empty? ? nil : DIMENSIONS.join(','), + filters: FILTERS.empty? ? nil : FILTERS.join(','), + sort: SORT.empty? ? nil : SORT.join(','), + start_index: start_index, + max_results: STEP_SIZE + ) + + target = open('my-report', "w") + + # now we can write out the results in a format that I want for my reporting. + results.rows.each do |row| + begin + app = row[2] + row[2] = parse_uri(app, user: row[1]) + row << app + target.write "#{row.join('|')}\n" + end + + +More Info +----------- + +.. _GA measurement protocol: https://developers.google.com/analytics/devguides/collection/protocol/v1/reference +.. _analytics lua code: https://github.com/OSC/ondemand/blob/master/mod_ood_proxy/lib/analytics.lua + +For reference, here's more detailed information about implementations and protocols described +in this document. + +See our `analytics lua code`_ for the implementation of how we're extracting this information, +parsing it and sending it to Google. + +See the `GA measurement protocol`_ for more details on the format we're sending this data in. \ No newline at end of file diff --git a/downcase-ids/_sources/how-tos/app-development.rst.txt b/downcase-ids/_sources/how-tos/app-development.rst.txt new file mode 100644 index 000000000..c8be31e7c --- /dev/null +++ b/downcase-ids/_sources/how-tos/app-development.rst.txt @@ -0,0 +1,24 @@ +.. _app-development: + +App Development +=============== + +OnDemand is made up of the platform (Apache and NGINX), Passenger apps and +plugins. Passenger apps are rack based Ruby apps, wsgi based Python web apps, or +Node.js apps that follow a convention for the app's startup file. + +The Dashboard app, Shell app, and all other core apps in OnDemand are Passenger +apps that can be replaced by custom Passenger apps. Or you can create your own. + +OnDemand's Interactive Apps are plugins that contain configuration files and a job +template for running a VNC Server or Web Server application (such as Jupyter or MATLAB) +on a compute node. + + +.. toctree:: + :maxdepth: 2 + :caption: App Development Guide + + app-development/enabling-development-mode + app-development/interactive + app-development/app-sharing diff --git a/downcase-ids/_sources/how-tos/app-development/app-sharing.rst.txt b/downcase-ids/_sources/how-tos/app-development/app-sharing.rst.txt new file mode 100644 index 000000000..a82654a32 --- /dev/null +++ b/downcase-ids/_sources/how-tos/app-development/app-sharing.rst.txt @@ -0,0 +1,246 @@ +.. _app_sharing: + +App Sharing +============= + +Overview +-------- + +Apps may be shared via a variety of methods including: + +1. System installed by admin for everyone to launch (access controlled via file + permissions) +2. Source Code Sharing (share code, launch your own copy of an app). This is + like how the Job Composer works, or how many groups collaborate when using + HPC: they share their job scripts and code, and use their own copies to + submit jobs. +3. Executable Sharing (peer to peer) is where a user deploys an app in their + home directory that other users can launch. This is similar to adding to your + PATH the bin directory of another user. + +.. _app_sharing_system_installed_apps: + +System Installed Apps +--------------------- + +Admins may install apps to the system by copying the application directory to ``/var/www/ood/apps/sys``. Default directory permissions (``755``) will allow all users with access to OnDemand to see and run that app. Apps may have their access restricted by changing the permissions on individual application directories. For example if a site does not wish to show licensed software to un-licensed users they might do the following: + + .. code-block:: sh + + # Given: + # - an app named $NEW_APP + # - a group named $NEW_APP_GROUP + # - and a user named $NEW_APP_USER + + sudo cp -r "/path/to/$NEW_APP" /var/www/ood/apps/sys + sudo chmod 750 "/var/www/ood/apps/sys/$NEW_APP" + sudo chgrp "$NEW_APP_GROUP" "/var/www/ood/apps/sys/$NEW_APP" + + sudo usermod -a -G "$NEW_APP_GROUP" "$NEW_APP_USER" + +You can utilize file access lists (FACLs) to increase the granularity with whom you share the apps. For example, you could specify multiple groups and individual users to share with, or even exclude specific users or groups. + +This app authorization mechanism (using file permissions) can also be useful for canary deployments (sharing an app under development with a subset of users, or just your development team). + +Example Using File Permissions +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + + .. code-block:: sh + :emphasize-lines: 7 + + [root@ood sys]# pwd + /var/www/ood/apps/sys + [root@ood sys]# ls -l + total 20 + drwxr-xr-x. 14 root root 4096 Jul 26 15:20 activejobs + drwxr-xr-x. 3 root root 169 Jul 26 15:20 bc_desktop + drwxr-xr-x. 5 root root 157 Aug 1 18:19 bc_desktop_example_kde + drwxr-xr-x. 13 root root 4096 Jul 26 15:20 dashboard + drwxr-xr-x. 14 root root 4096 Jul 26 15:20 file-editor + drwxr-xr-x. 7 root root 4096 Jul 26 15:20 files + drwxr-xr-x. 14 root root 4096 Jul 26 15:20 myjobs + drwxr-xr-x. 7 root root 245 Jul 26 15:20 shell + + .. figure:: /images/app-sharing-permissions-before.png + :align: center + +To restrict usage to only members of the ``desktopers`` group: + + .. code-block:: sh + :emphasize-lines: 9 + + [root@ood sys]# pwd + /var/www/ood/apps/sys + [root@ood sys]# chmod 750 bc_desktop_example_kde/ + [root@ood sys]# chgrp desktopers bc_desktop_example_kde + [root@ood sys]# ls -l + total 20 + drwxr-xr-x. 14 root root 4096 Jul 26 15:20 activejobs + drwxr-xr-x. 3 root root 169 Jul 26 15:20 bc_desktop + drwxr-x---. 5 root desktopers 157 Aug 1 18:19 bc_desktop_example_kde + drwxr-xr-x. 13 root root 4096 Jul 26 15:20 dashboard + drwxr-xr-x. 14 root root 4096 Jul 26 15:20 file-editor + drwxr-xr-x. 7 root root 4096 Jul 26 15:20 files + drwxr-xr-x. 14 root root 4096 Jul 26 15:20 myjobs + drwxr-xr-x. 7 root root 245 Jul 26 15:20 shell + +Note that user ``ood`` is not a member of the ``desktopers`` supplemental group. + + .. figure:: /images/app-sharing-permissions-after.png + :align: center + +These changes take effect immediately, although when a user is added or removed from a group their PUN will need to be restarted for the change to take effect. + + +Code Sharing +------------ + +Code sharing is when an application's source code is shared between two or more users who run it as a personal development application. Models for this sharing can include using a web-based file repository such as Github, emailing Zip'd app directories, or a group readable directory symlinked to each user's ``~/ondemand/dev/`` directory. + +For an example of the later consider: + + .. code-block:: sh + + # As user mrodgers + owens-login01:mrodgers mrodgers$ pwd + # /fs/project/PZS0714/mrodgers + owens-login01:mrodgers mrodgers$ ls -l + # total 97856 + # drwxr-xr-x 7 mrodgers PZS0714 4096 Aug 1 16:03 blender-batch-render-app + owens-login01:mrodgers mrodgers$ cd ~/ondemand/dev + owens-login01:mrodgers mrodgers$ ln -s /fs/project/PZS0714/mrodgers/blender-batch-render-app + + # As user johrstrom + owens-login01:johrstrom johrstrom$ cd ~/ondemand/dev + owens-login01:johrstrom johrstrom$ ln -s /fs/project/PZS0714/mrodgers/blender-batch-render-app + +User ``johrstrom`` will now see ``blender-batch-render-app`` in their Sandbox Apps, but because they do not own the files they will not be able to edit the files, or update dependencies, etc resulting in a slightly broken experience. Better still would be peer to peer app sharing. + + +Peer to Peer Executable Sharing +------------------------------- + +By setting a few environment variables it is possible to enable a more polished peer to peer app sharing experience. There are two reasons why this mode is not always enabled: the first is that app permissions are the only thing that prevents all a site's OnDemand users from seeing a shared app, so it is important to get the permissions correct, and only to deploy apps that are production ready. The other reason to be careful with app sharing is that requires greater trust placed in app developers. + +.. warning:: Executable sharing means the app and all its code runs as the user + executing it, like everything else in OnDemand. User's might not + realize this. We currently do not provide an opt in screen warning + users that this app "will have permission to do everything on their + behalf and act as them". As a result, you should fully trust whoever + you enable to do share apps using executable sharing. + +Enabling The App Sharing Dashboard +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +#. To enable App Sharing in the Dashboard, set ``OOD_APP_SHARING=1`` in + ``/etc/ood/config/apps/dashboard/env``. +#. Set ``OOD_DASHBOARD_SUPPORT_EMAIL=your@email.edu`` to add a link to support + for finding an app. +#. Set ``OOD_APP_CATALOG_URL=https://link.to.online/app/catalog`` to link + externally to an advertised listing of apps available. +#. Pin usr apps to the dashboard and group them by category. + +.. code:: yaml + + # /etc/ood/config/ondemand.d/ondemand.yml + pinned_apps: + - 'usr/*' + + pinned_apps_group_by: 'category' + + +Enabling App Sharing in the dashboard serves two primary purposes: + +1. For shared app users, provide an interface to launch those apps +2. For app developers, provide an interface to help manage shared apps + +Currently this significantly changes the interface of the Dashboard. The MOTD +moves to the right of the screen and shared apps appear below the welcome logo +and text. + +Before: + +.. figure:: /images/app-sharing-mode-before.png + :align: center + + +After: + +.. figure:: /images/app-sharing-mode-after.png + :align: center + +Controlling Who Can Share and Access Apps +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Shared apps are deployed to +``/var/www/ood/apps/usr/$USER/gateway/$APPNAME``. We recommend ``gateway`` +be a symlink to the user's home directory at ``$HOME/ondemand/share`` and +by default set ``750`` permissions on ``/var/www/ood/apps/usr/$USER``. This +approach has these benefits (assuming users named ``efranz`` and ``an0047``: + +#. The admin as root controls who can share apps by creating root owned + directories like ``/var/www/ood/apps/usr/efranz`` and + ``/var/www/ood/apps/usr/an0047``. +#. The admin controls who can access that user's shared apps by setting + permissions on this directory. Thus by setting ``750`` on + ``/var/www/ood/apps/usr/an0047`` this ensures that an0047 can only share + apps with users in his primary group. At times we have created a \ + supplemental group (shinyusr) and chgrp the share directory to this group so + that the developer can share apps with every user in this group. +#. The developer who can share apps can modify permissions on the app + directories themselves i.e. + ``/var/www/ood/apps/usr/an0047/gateway/customapp`` + so that only a subset of the users he could share with have access. This can + be done through FACLs or using the same chgrp + 755 approach + +In summary, to enable a new user to create shared apps, run these commands: + +.. code:: sh + + # given a user efranz + sudo mkdir -p /var/www/ood/apps/usr/efranz + cd /var/www/ood/apps/usr/efranz + chmod 750 . + ln -s ~efranz/ondemand/share gateway + +Example of Executable Sharing +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +This is with two users Eric (efranz) and Bob (an0047). + +Eric has a dev app "MATLAB", and interactive plugin app. Eric can + +1. Launch MATLAB +2. View and Edit the code + +.. figure:: /images/app-sharing-1.png + :align: center + + + +Bob (an0047) cannot see this app because it is isolated in Eric's "Sandbox" +i.e. ``~efranz/ondemand/dev/matlab``: + +.. figure:: /images/app-sharing-2.png + :align: center + +If Eric shares the git repo path or URL with Bob, Bob can clone this into his +home directory if he is enabled as a developer. This is called "Source Code Sharing". + +Eric can share this app with Bob by selecting "My Shared Apps" and cloning the MATLAB +repo to deploy ``~efranz/ondemand/share/matlab`` + +.. figure:: /images/app-sharing-3.png + :align: center + +.. figure:: /images/app-sharing-4.png + :align: center + +.. figure:: /images/app-sharing-5.png + :align: center + +Now when Bob accesses the OnDemand home page he sees Eric's MATLAB app and can +launch it: + +.. figure:: /images/app-sharing-6.png + :align: center diff --git a/downcase-ids/_sources/how-tos/app-development/enabling-development-mode.rst.txt b/downcase-ids/_sources/how-tos/app-development/enabling-development-mode.rst.txt new file mode 100644 index 000000000..d5734b3c8 --- /dev/null +++ b/downcase-ids/_sources/how-tos/app-development/enabling-development-mode.rst.txt @@ -0,0 +1,118 @@ +.. _enabling-development-mode: + +Enabling App Development +======================== + +Enable in OnDemand v1.6+: +......................... + +Here are example steps to enable a user "efranz", assuming efranz's home directory is at ``/home/efranz``: + +#. Create a symlink so OnDemand finds efranz's apps: + + .. code:: sh + + sudo mkdir -p /var/www/ood/apps/dev/efranz + cd /var/www/ood/apps/dev/efranz + sudo ln -s /home/efranz/ondemand/dev gateway + + +#. Have efranz access the Dashboard, and efranz will see the Develop dropdown + + +Enable in OnDemand v1.4 & v1.5: +............................... + +Here are example steps to enable a user "efranz", assuming efranz's home directory is at ``/home/efranz``: + +#. Create a symlink so OnDemand finds efranz's apps: + + .. code:: sh + + sudo mkdir -p /var/www/ood/apps/dev/efranz + cd /var/www/ood/apps/dev/efranz + sudo ln -s /home/efranz/ondemand/dev gateway + +#. Create dev directory ``/home/efranz/ondemand/dev`` where efranz's dev apps will go (or ask efranz to do that) +#. Have efranz access the Dashboard, and efranz will see the Develop dropdown + + +Enable in OnDemand v1.3: +........................ + +Here are example steps to enable a user "efranz", assuming efranz's home directory is at ``/home/efranz``: + +#. Create dev directory ``/home/efranz/ondemand/dev`` where efranz's dev apps will go (or ask efranz to do that) +#. Have efranz access the Dashboard, and efranz will see the Develop dropdown. (if this doesn't happen, ) + +.. note:: + + The rest of the documentation below assumes you are working with OnDemand 1.4+. + + +Specify dedicated host for development (optional) +.................................................... + +The default host for the shell app is typically a login node. If this node does not contain a similar environment as the host OnDemand is installed on, including access to Software Collection (SCL) packages, it may be useful to provide developers a dedicated development host they can SSH to. This could even be the OnDemand host itself. If you configure the OnDemand dashboard to know about the dedicated development host, the dashboard will present links to open the shell app to this host. + +To specify a dedicated development host so OnDemand, set ``OOD_DEV_SSH_HOST`` environment variable for the dashboard in the file ``/etc/ood/config/apps/dashboard/env``. For example at OSC one of our OnDemand installs uses ondemand-test.osc.edu for the development host so we have this line: `OOD_DEV_SSH_HOST="ondemand-test.osc.edu" `_ + + +Make everyone a developer by default (optional) +............................................... + +To revert to the way developer enabling worked in OnDemand 1.3, change the nginx_stage app_root configuration for dev apps by modifying /etc/ood/config/nginx_stage.yml and replacing + +.. code-block:: yaml + :emphasize-lines: 2 + + app_root: + dev: '/var/www/ood/apps/dev/%{owner}/gateway/%{name}' + usr: '/var/www/ood/apps/usr/%{owner}/gateway/%{name}' + sys: '/var/www/ood/apps/sys/%{name}' + + +with + + +.. code-block:: yaml + :emphasize-lines: 2 + + app_root: + dev: '~%{owner}/%{portal}/dev/%{name}' + usr: '/var/www/ood/apps/usr/%{owner}/gateway/%{name}' + sys: '/var/www/ood/apps/sys/%{name}' + +Then users can just create the directory ``~/ondemand/dev`` and the Develop dropdown will appear. + +.. warning:: If you do this, it is recommended that you treat the node that OnDemand is running on as a login node, as you are effectively giving those users shell access by letting them run arbitrary code on the OnDemand node (of course the UID of the processes are still their regular unprivileged user UID). + +If you do this, you still might want to restrict who sees the Develop dropdown in the Dashboard. To do that you can explicitly show or hide the dropdown in the Dashboard by setting ``Configuration.app_development_enabled`` to true based on one or more Ruby statements in the initializer ``/etc/ood/config/apps/dashboard/initializers/ood.rb``. Code in the initializer runs as the user. This code also has access to the `ood_support library `__ in which we provide some helper classes to work with User's and Groups. For example: + + + .. code-block:: ruby + + Rails.application.config.after_initialize do + Configuration.app_development_enabled = OodSupport::Process.groups.include?( + OodSupport::Group.new("devgrp") + ) + end + + Or if you know the id of the group, this will avoid reading the ``/etc/group`` + file: + + .. code-block:: ruby + + Rails.application.config.after_initialize do + Configuration.app_development_enabled = Process.groups.include?(5014) + end + + Or a specific user list: + + .. code-block:: ruby + + Rails.application.config.after_initialize do + Configuration.app_development_enabled = %w( + bgohar efranz bmcmichael + ).include?(OodSupport::User.new.name) + end diff --git a/downcase-ids/_sources/how-tos/app-development/interactive.rst.txt b/downcase-ids/_sources/how-tos/app-development/interactive.rst.txt new file mode 100644 index 000000000..10f09e0e0 --- /dev/null +++ b/downcase-ids/_sources/how-tos/app-development/interactive.rst.txt @@ -0,0 +1,48 @@ +.. _app-development-interactive: + +Interactive Apps +================ + +Interactive apps can be developed and deployed using the same tools that are +currently provided for all Open OnDemand applications but requires further +:ref:`app-development-interactive-setup`. + +An Interactive App is a plugin that follows a custom +file/directory structure and API that can be described by the five stages: +:ref:`app-development-manifest`, +:ref:`app-development-interactive-form`, +:ref:`app-development-interactive-template`, +:ref:`app-development-interactive-submit` and +:ref:`app-development-interactive-view`. + +Additionally, there is :ref:`app-development-interactive-additional-info`. + +A typical file/directory structure for an Interactive App can look like:: + + my_app/ + ├── form.yml + ├── manifest.yml + ├── submit.yml.erb + ├── template + │ ├── before.sh.erb + │ └── script.sh.erb + ├── view.html.erb + ├── info.{md,html}.erb + └── completed.{md,html}.erb + +Each of these files/directories are described below in their respective stage. + +.. toctree:: + :maxdepth: 3 + :caption: Stages of an Interactive App + + interactive/manifest + interactive/form + interactive/form-widgets + interactive/dynamic-form-widgets + interactive/template + interactive/submit + interactive/view + interactive/sub-apps + interactive/conn-params + interactive/additional-info diff --git a/downcase-ids/_sources/how-tos/app-development/interactive/additional-info.rst.txt b/downcase-ids/_sources/how-tos/app-development/interactive/additional-info.rst.txt new file mode 100644 index 000000000..94a4d60fb --- /dev/null +++ b/downcase-ids/_sources/how-tos/app-development/interactive/additional-info.rst.txt @@ -0,0 +1,47 @@ +.. _app-development-interactive-additional-info: + +Adding Additional Information to the session cards +================================================== + +.. _bc_info_html_md_erb: + +info.{md,html}.erb +------------------ + +It's possible for you to add additional information to this session's card. + +You can do so by creating a Markdown file ``info.md.erb`` or an html file +``info.html.erb`` in the applications folder. Markdown files get generated +into html with # turning into an

and ## turning into an

and so on. + +Again, they're `eRuby (Embedded Ruby)`_ files so you can add some dynamic behavior +to them. Along with any library you may choose to use you can also access these +variables directly. + +id + The session UUID of the job +cluster_id + The cluster the job was submitted to +job_id + The job id from the scheduler +created_at + The time the session was created + + +.. _bc_completed_html_md_erb: + +completed.{md,html}.erb +------------------------ + +:ref:`bc_info_html_md_erb` above will display on the session's card +regardless of the state of the job - it will always be displayed. + +``completed.{md,html}.erb`` on the other hand, will only display +once the job has reached the ``completed`` state. + +You may want to add this to the session's card to display information +to the user when the job is completed. Again, as it's `eRuby (Embedded Ruby)`_ +files so you can add some dynamic behavior to them. + + +.. _eruby (embedded ruby): https://en.wikipedia.org/wiki/ERuby diff --git a/downcase-ids/_sources/how-tos/app-development/interactive/conn-params.rst.txt b/downcase-ids/_sources/how-tos/app-development/interactive/conn-params.rst.txt new file mode 100644 index 000000000..aa3e6f410 --- /dev/null +++ b/downcase-ids/_sources/how-tos/app-development/interactive/conn-params.rst.txt @@ -0,0 +1,70 @@ +.. _app-development-interactive-conn-params: + +Connection Parameters ``conn_params`` +===================================== + +App developers can use ``conn_params`` in the ``submit.yml.erb`` to pass runtime data generated +in the ``before.sh.erb`` back to the ``view.html.erb``. + +This is helpful for: + +* Data that is only known **after** the job submits and starts running. +* Data that needs to be used to connect to the application. + +This technique will generate a file in the jobs working directory called ``connection.yml`` +when the app launches which will contain the defined variables and their associated values. + + +Jupyter Notebook Example +------------------------ + +Here's an example using a Jupyter application which needs +needs to know the exact API to connect to. We can either connect to +JuypterLab at ``/lab`` or Juypter Notebook at ``/tree``, but this +information is not known until the job has been submitted. + +So once the job is submitted, we need to export the ``jupyter_api`` +environment variable that can then be written to ``connection.yml`` +which OnDemand will consume and use in the ``view.html.erb``. + +.. warning:: + + The environment variables in ``before.sh.erb`` *must* be lowercase and + exported through the *export* function. + +.. code:: shell + + # within template/before.sh.erb + + JUPYTER_API="<%= context.jupyterlab_switch == "1" ? "lab" : "tree" %>" + + export jupyter_api="$JUPYTER_API" + + +Now with that variable exported, you need to add it to ``conn_params`` in +``submit.html.erb`` to ensure that OnDemand makes use of it. + +.. code::yaml + + batch_connect: + template: "basic" + conn_params: + - jupyter_api + +In the ``view.html.erb``, which renders after the submission in the interactive apps page, +you can access the value of this variable with:: + + <%- + ... + next_url = "#{base_url}/#{jupyter_api}" + + full_url="#{login_url}?next=#{CGI.escape(next_url)}" + ... + %> + + ... + +
+ +This configuration uses the value from ``jupyter_api`` to populate +the ``action`` attribute in the rendered page in the interactive sessions. diff --git a/downcase-ids/_sources/how-tos/app-development/interactive/dynamic-form-widgets.rst.txt b/downcase-ids/_sources/how-tos/app-development/interactive/dynamic-form-widgets.rst.txt new file mode 100644 index 000000000..b797f849d --- /dev/null +++ b/downcase-ids/_sources/how-tos/app-development/interactive/dynamic-form-widgets.rst.txt @@ -0,0 +1,233 @@ +.. _dynamic-bc-apps: + +Dynamic Form Widgets +==================== + +.. note:: + This feature was added in version 2.0. + +Prior to version 2.0, sites would have to make their own custom `form.js` to +add dynamic behavior for batch connect forms. While sites can still have their +own custom javascript, 2.0 added out of the box support for common use cases based +on configuration. + +.. warning:: + Form fields use underscores (``_``) as a word separator and these directives use hyphens (``-``). + So when referencing the element ``node_type`` you will need to reference it as ``node-type``. + +To enable any of the dynamic batch connect capabilities described in this page, +set the :ref:`dynamic batch connect setting `. + + +Your own ``form.js`` +******************** + +If we don't support what you need for your application to be dynamic, then you can add your +own ``form.js`` in the root of the project. It is free form javascript, so most anything is +allowed. `jQuery`_ is available to interact with elements. + + +Hiding select options +********************* + +The ``data-option-for`` directive allows you to hide select choices of an element based on +the current value of a different select element. + +Let's use this example: We have two clusters, ``oakley`` and ``ruby``. The ``oakley`` cluster +has GPUs but ``ruby`` does not. + +So, we'd like to give the choice of ``node_type`` to show ``gpu`` when the user has +the ``oakley`` cluster selected, but not when ``ruby`` is selected. + +Everything is shown by default so we need to configure the app such that +``gpu`` option is hidden when the ``ruby`` cluster is chosen. + +Open OnDemand provides ``data-option-for`` directive for this use case. + +With this one configuration we're telling the OnDemand system that ``gpu`` is not an option +for the ``ruby`` cluster. It will then hide it every time the cluster ``ruby`` is chosen. + +.. code-block:: yaml + :emphasize-lines: 8 + + attributes: + node_type: + widget: select + options: + - 'standard' + - [ + 'gpu', 'gpu', + data-option-for-cluster-ruby: false + ] + +.. tip:: + This example shows toggling options based on the cluster, but this feature + generically support any field and value. + + +Hiding entire elements +********************** + +The ``data-hide`` directive allows you to hide another element based on +the current value of a select element. + +Let's continue examples involving GPUs. We'd like to provide users +with options for `CUDA`_ versions. + +But using Nvidia's `CUDA`_ libraries only makes sense when the user is requesting GPUs. +So, we want to hide the ``cuda_version`` element when a users chooses standard ``node_type``. + +Here's the example YAML for this app with two select widgets. This +instructs the webpage to hide the ``cuda_version`` when the ``standard`` +``node_type`` is selected. + +.. warning:: + In addition to hiding form fields like this example shows, one should + also use a ``data-set`` directive to set the value because the field + is no longer visible to the user. While it's hidden, it will still retain + the current value, if any has been supplied. + + By forcing a value after hiding it you can ensure that the correct values + are being passed to the server. + +.. code-block:: yaml + :emphasize-lines: 7 + + attributes: + node_type: + widget: select + options: + - [ + 'standard', 'standard', + data-hide-cuda-version: true, + data-set-cuda-version: 'none' + ] + - 'gpu' + + +Additionally, you can use ``check_box`` widgets to hide elements. +Here we have a checkbox ``enable_cuda_version`` that will show +``cuda_version`` when checked and hide it when it's not checked. + +.. tip:: + Checkboxes respond to ``when-checked: true`` and ``when-un-checked: true`` + for hiding elements when checked or unchecked. + +.. code-block:: yaml + :emphasize-lines: 6 + + attributes: + enable_cuda_version: + widget: 'check_box' + html_options: + data: + hide-cuda-version-when-un-checked: true + +Dynamic Element Labels +********************** + +The ``data-label-*`` directive allows you to change the label of another +form element based on the selected option in a select widget. + +.. code-block:: yaml + + attributes: + node_type: + widget: select + options: + - [ 'small', 'small', data-label-cores: 'Number of Cores (1-4)' ] + - [ 'medium', 'medium', data-label-cores: 'Number of Cores (1-8)' ] + - [ 'large', 'large', data-label-cores: 'Number of Cores (1-16)' ] + + cores: + widget: "number_field" + required: true + value: 1 + +In this case, selecting Node Type 'small' will change the label of Cores to +'Number of Cores (1-4)'. + +Dynamic Min and Maxes +********************* + +The ``data-min`` and ``data-max`` directives allow you to set the minimum and +maximum values of another element based on the current value of a select element. + +Sites have node types of all shapes and sizes. Some sites even have +heterogenous clusters where there are different node types in the cluster. + +This feature allows for setting the minimum and maximum values for input +fields like the number of cores to request. + +Let's see an example. We have `standard`` nodes in both clusters, but they're +different sizes. In the ``oakley`` cluster nodes have a total 28 cores and in the +``ruby`` cluster they have 40. + +In this example ``data-max-num-cores-for-cluster-oakley`` is attached to the standard +node type. This config is saying, when the ``node_type`` is ``standard`` +and the ``cluster`` is ``oakley`` set maximum ``num_cores`` to 28. + +.. code-block:: yaml + + attributes: + node_type: + widget: select + options: + - [ + 'standard', 'standard', + data-max-num-cores-for-cluster-oakley: 28, + data-max-num-cores-for-cluster-ruby: 40, + ] + - [ + 'gpu', 'gpu', + data-max-num-cores: 1, + data-min-num-cores: 1, + ] + +This example also illustrates a simpler variant of this directive attached to ``gpu``. +This configuration doesn't have a for clause, so it will set the minimum and maximum +values for ``num_cores`` when ``gpu`` is selected, regardless of which cluster is selected. + + +Setting values based on other elements +************************************** + +The ``data-set`` directive allows you to set a value on a different element based +on the current value of a select element. + +Let's use charge-back accounts as an example. Let's imagine we want to set the charge-back +account automatically based on the selection of node type. + +In this example, when ``standard`` ``node_type`` is chosen, the ``charge_account`` element +will be automatically set to ``standard-charge-code``. + +In addition to setting strings, option choices can also set check boxes. +You will see in this example that when you change the ``node_type`` selection +the ``enable_gpu`` check box will either be checked or unchecked. + +.. code-block:: yaml + + form: + - enable_gpu + - charge_account + - node_type + attributes: + enable_gpu: + widget: check_box + node_type: + widget: select + options: + - [ + 'standard', 'standard', + data-set-charge-account: 'standard-charge-code', + data-set-enable-gpu: 0 + ] + - [ + 'gpu', 'gpu', + data-set-charge-account: 'gpu-charge-code', + data-set-enable-gpu: 1 + ] + + +.. _CUDA: https://developer.nvidia.com/cuda-toolkit +.. _jQuery: https://jquery.com/ diff --git a/downcase-ids/_sources/how-tos/app-development/interactive/form-widgets.rst.txt b/downcase-ids/_sources/how-tos/app-development/interactive/form-widgets.rst.txt new file mode 100644 index 000000000..0fed933a9 --- /dev/null +++ b/downcase-ids/_sources/how-tos/app-development/interactive/form-widgets.rst.txt @@ -0,0 +1,164 @@ +.. _form-widgets: + +Form Widgets +============ + Checkbox (check_box) + A checkbox. Note that you can change the checked and unchecked values. For example changing + them from ``1`` and ``0`` to ``yes`` and ``no``. + + .. code-block:: yaml + + test_checkbox: + widget: check_box + checked_value: 1 + unchecked_value: 0 + label: "Test Checkbox" + help: | + Your help message + +================================================================== + + Hidden Field (hidden_field) + A hidden field that will not be shown, but will still be in the HTML. + + .. code-block:: yaml + + test_hidden_field: + widget: "hidden_field" + value: "Test Hidden Field Value" + +================================================================== + + Number Field (number_field) + A number field. + + .. code-block:: yaml + + num_cores: + widget: "number_field" + label: "Number of cores" + value: 1 + help: | + Your help message + min: 1 + max: 28 + step: 1 + +================================================================== + + Radio Button (radio_button) + Note that in the options below, the text to display is on the left of the comma, and the select value is on the right of the comma. + The value: key represents the default selection. + + .. code-block:: yaml + + mode: + widget: "radio_button" + value: "1" + help: | + Your help message + options: + - ["Jupyter Lab", "1"] + - ["Jupyter Notebook", "0"] + +================================================================== + + Resolution Field (resolution_field) + Change the resolution for interactive applications that use VNC. + + .. code-block:: yaml + + test_resolution_field: + widget: "resolution_field" + label: "Test Resolution Field" + required: true + help: | + Your help message + +================================================================== + + Select Field (select) + Note that in the options below, the text to display is on the left of the comma, and the select value is on the right of the comma. + + .. code-block:: yaml + + version: + widget: "select" + label: "JupyterLab Version" + options: + - [ "3.0", "app_jupyter/3.0.17" ] + - [ "2.3", "app_jupyter/2.3.2" ] + - [ "2.2", "app_jupyter/2.2.10" ] + - [ "1.2", "app_jupyter/1.2.21" ] + help: | + Your help message + +================================================================== + + TextArea Field (text_area) + A text area. This allows for multiple lines of text input. + + .. code-block:: yaml + + test_text_area: + widget: "text_area" + label: "Test Text Area" + value: "Test Text Area Value" + help: | + Your help message + +================================================================== + + Text Field (text_field) + A text field. This only allows for a single line of text input. + + .. code-block:: yaml + + test_text_field: + widget: "text_field" + label: "Test Text Field" + value: "Test Text Value" + help: | + Your help message + +.. _path_selector: + +================================================================== + + Path Selector (path_selector) + A Path Selector. This is a special OnDemand feature that is not + directly an HTML input type. It builds a ``text_field`` input + type, but also provides a button that will provide a modal that + allows users to navigate through directories to select a path. + + This is useful in forms where a path must be selected and you + want to allow your users to choose an arbirary path. + + ``directory`` is the initial directory the path selector will open + to when the users opens the modal. This defaults to the users' HOME. + + ``show_hidden`` is a boolean flag to show hidden files or not. This + defaults to false - it will not show hidden files. + + ``show_files`` is a boolean flag to show files or not. This defaults + to true - it will show files. + + ``favorites`` allows you to override the :ref:`favorite paths you've added + in files menu `. Provide an array of new favorites + or set to ``false`` to disable showing favorites altogether. + + .. code-block:: yaml + + path: + widget: "path_selector" + directory: "/fs/ess/project" + show_hidden: true + show_files: false + favorites: + - /fs/ess + - /fs/scratch + +================================================================== + + +.. _markdown: https://en.wikipedia.org/wiki/Markdown diff --git a/downcase-ids/_sources/how-tos/app-development/interactive/form.rst.txt b/downcase-ids/_sources/how-tos/app-development/interactive/form.rst.txt new file mode 100644 index 000000000..b970f76ec --- /dev/null +++ b/downcase-ids/_sources/how-tos/app-development/interactive/form.rst.txt @@ -0,0 +1,815 @@ +.. _app-development-interactive-form: + +User Form (form.yml.erb) +======================== + +The configuration file ``form.yml`` creates the `html form`_ your customers will use +to start the interactive application. + + +Let's look at a simple example of how the main components ``form`` and ``attributes`` work. +``form`` is a list of form choices for the application. ``attributes`` is then specifying +*what* those items in ``form`` are, whether they're number fields or choices and so on. + +.. code-block:: yaml + + # ${HOME}/ondemand/dev/my_app/form.yml + --- + + # the cluster(s) this app can submit jobs to. + cluster: "owens" + + # 'form' is a list of form choices for this app. Here we're allowing users to set + # the account and the number of cores they want to request for this job. + form: + - account + - cores + + # By default, everything defined in 'form' above is a text_field. Let's leave account alone + # We do however, want cores to be a number_field and with some min & max values. + attributes: + cores: + widget: 'number_field' + min: 1 + max: 40 + +It is located in the root of the application directory. + +Assuming we already have a sandbox Interactive App deployed under:: + + ${HOME}/ondemand/dev/my_app + +The ``form.yml`` configuration file can be found at:: + + ${HOME}/ondemand/dev/my_app/form.yml + +Then in the browser, navigate to the dashboard and choose in the top +right menu: *Develop* → *My Sandbox Apps (Development)*. Finally click *Launch +My App* from the list of sandbox apps. + + +You should now see the HTML form used to gather the user-defined attributes for +building and launching the ``my_app`` Interactive App session. + +.. warning:: + + Since 4.0 HTML IDs of the form items are always lowercase. The examples above + show lowercase configurations of ``account``. Specifying ``Account``, + or ``ACCOUNT`` or any variation of uppercase and lowercase will result in + the same behhavior as specifying ``account`` (all lower case). + + If you write your own ``form.js`` take care to note that HTML IDs of these + form items will **always** be lowercase regardless of how they're defined in + the YAML file. + +.. tip:: + + You can include dynamically generated content in the form by renaming the + form file to ``form.yml.erb`` and incorporating ERB syntax. + +Configuration +------------- + +This is the full list of items with details, you may supply to this yaml file to configure this application. + +.. describe:: cluster (Array or String) + + the cluster ids that the Interactive App session is submitted to. + + .. note:: + + Prior to 1.8, this configuration was a single string (a single cluster). + We now support one application submitting to multiple clusters. See the + section below on `configuring which cluster to submit to`_ for more information. + + .. warning:: + + The cluster id must correspond to a cluster configuration file located + under:: + + /etc/ood/config/clusters.d + +.. describe:: form (Array) + + a list of **all** the attributes that will be used as the context for + describing an Interactive App session + + .. note:: + + The attributes that appear as HTML form elements will appear to the + user in the order they are listed in this configuration option. + +.. describe:: attributes (Hash) + + the object defining the hard-coded value or HTML form element used for the + various custom attributes + +.. describe:: cacheable (Boolean) + + whether or not the application is cacheable or not. Defaults to true. + +Attributes +---------- + +Attributes are *variables* whose values can be set either by the user from +within an HTML form or hard-coded to a specific value in the form +configuration. These attributes and their corresponding values are then made +available to the Interactive App through its intermediate steps: +:ref:`app-development-interactive-template` and +:ref:`app-development-interactive-submit`. + +.. _app-development-interactive-form-predefined-attributes: + +Predefined Attributes +`````````````````````` + +The Dashboard that supports these plugins provides the plugins with some useful +predefined attributes that can be included in the ``form:`` configuration list +with very little or no modification on the part of the developer. + +So a very simple ``form.yml`` that requests the user input a queue followed by +an account to submit the batch job (interactive session) to, and then +subsequently submits the job to that queue without any customization on the +part of the app developer can look like: + +.. code-block:: yaml + + # ${HOME}/ondemand/dev/my_app/form.yml + --- + cluster: "owens" + form: + - bc_queue + - bc_account + +The most commonly used predefined attributes are given as: + +.. _bc_account: + +bc_account + This adds a ``text_field`` to the HTML form that will be used as the charged + account for the submitted job. + + This attribute gets directly set on `OodCore::Job::Script#accounting_id`_. + +.. _bc_queue: + +bc_queue + This adds a ``text_field`` to the HTML form that will supply the name of the + queue that the batch job is submitted to. + + This attribute gets directly set on `OodCore::Job::Script#queue_name`_. + +.. _bc_num_hours: + +bc_num_hours + This adds a ``number_field`` to the HTML form that describes the maximum + amount of hours the submitted batch job may run. + + This attribute gets converted to seconds and then set on + `OodCore::Job::Script#wall_time`_. + +bc_num_slots + This adds a ``number_field`` to the HTML form that describes the number of + processors, CPUs on a single node, or nodes that the submitted job may use + (depends on the resource manager used, e.g., Torque, Slurm, ...). + + This attribute manipulates the brittle `OodCore::Job::Script#native`_ field + with a value that depends on the given resource manager for the cluster. + + .. warning:: + + This predefined attribute is very resource manager specific, and is the + most brittle of all the other predefined attributes. May require + customization (see + :ref:`interactive-development-form-customizing-attributes`) to work at + your center. + +.. _bc_email_on_started: + +bc_email_on_started + This adds a ``check_box`` to the HTML form that determines whether the user + should be notified by email when the batch job starts. + + This attribute sets value of `OodCore::Job::Script#email_on_started`_ + depending on whether the user checked the box or not. + + +.. _auto-bc-form-options: + +Automatic Predefined Attributes +```````````````````````````````` + +Automatic attributes automatically generate lists or values. For example +``bc_queue`` above will generate a ``text_field`` where the user has to +input the queue name themselves. + +``auto_queues`` on the other hand will automatically build a list of queues +for the user to choose from without intervention from the administrator +or the user. + +They must be added to the `form` list. + +EX: + +.. code-block:: yaml + + form: + - auto_modules_ + +auto_primary_group + This will automatically set the `OodCore::Job::Script#accounting_id`_ to the + primary group of the user. No choice will be given to the user. + +auto_modules_ + This will generate a list of modules in a ``select`` widget. + For example ``auto_modules_matlab`` will automatically populate a dropdown + list of every single ``matlab`` version available, including the default + version. + + To disable the default version, use the ``attributes`` field like so: + + .. code-block:: yaml + + attributes: + auto_modules_matlab: + default: false + + See :ref:`the module directory configuration ` on how to enable + the cluster module files that need to be read. + + If you have dynamic form widgets enabled, the option list will be cluster aware. + Meaning only versions appropriate to a given cluster will be shown when that + cluster is chosen. + + .. note:: + + The form configuration is case sensitive. So there is a difference between + ``auto_modules_R`` and ``auto_modules_r``. + + Hyphens cause issues in templating the script files. For example, + a form configuration like ``auto_modules_netcdf-serial`` would need to be + referenced in the ``script.sh.erb`` as ``<%= auto_modules_netcdf_serial %>`` + replacing any hyphens (``-``) with underscores ``_``. + +.. _auto_groups: + +auto_groups + This will automatically generate a ``select`` widget populated with a list of the Unix + groups the user is currently in. Administrators can configure :ref:`filter for autogroups ` + to limit the groups shown. + +.. _auto_queues: + +auto_queues + This will generate a ``select`` widget list of all the queues available to the user. + These queues will be cluster if you have :ref:`dynamic options ` + enabled. That is, they'll show or hide relevant lists given the currently selected + ``cluster``. + + .. warning:: + We only have support for Slurm queues (partitions) at this time. + +.. _auto_accounts: + +auto_accounts + This will generate a ``select`` widget list of all the accounts available to the user. + + ``auto_accounts`` will generate cluster aware lists if you have :ref:`dynamic options ` + enabled. This means it will only show a list of accounts available for the ``cluster`` that's + currently selected. If this setting is not enabled, it will generate a list of all accounts + available on all clusters and will not hide any of them. + + If, however, your site has a simpler accounting scheme where all accounts are available on + all clusters, you can set the :ref:`bc_simple_auto_accounts ` setting + for some optimizations. + + By default, we return exactly what the scheduler returns. These accounts may be lowercase + when you need uppercase accounts. To enable uppercase accounts set the environment variable + ``OOD_UPCASE_ACCOUNTS`` to anything. If the environment variable is set to *anything* the + system will uppercase the accounts (set it to ``yes`` if you don't know what value to give). + + .. warning:: + We only have support for Slurm accounts at this time. + +auto_qos + This will automatically generate a ``select`` widget populated with a list of all the QoS + (Quality of Service) values available. These are cluster aware if you have + :ref:`dynamic options ` enabled. + +.. _`oodcore::job::script#accounting_id`: http://www.rubydoc.info/gems/ood_core/OodCore%2FJob%2FScript:accounting_id +.. _`oodcore::job::script#queue_name`: http://www.rubydoc.info/gems/ood_core/OodCore%2FJob%2FScript:queue_name +.. _`oodcore::job::script#wall_time`: http://www.rubydoc.info/gems/ood_core/OodCore%2FJob%2FScript:wall_time +.. _`oodcore::job::script#email_on_started`: http://www.rubydoc.info/gems/ood_core/OodCore%2FJob%2FScript:email_on_started +.. _`oodcore::job::script#native`: http://www.rubydoc.info/gems/ood_core/OodCore%2FJob%2FScript:native + +.. _interactive-development-form-customizing-attributes: + +Customizing Attributes +`````````````````````` + +For each defined attribute in the ``form:`` configuration option, you can +modify/override any component that makes up the HTML form element in the +``attributes:`` configuration option, e.g.: + +.. code-block:: yaml + + attributes: + my_custom_attribute: + label: "My custom label" + ... + +The available configuration options that can be modified for a given attribute +are: + +.. describe:: widget (String, null) + + For more detailed information with examples regarding the widget field, see :ref:`form-widgets`. + + the type of HTML form element to use for input + + - ``check_box`` + - ``hidden_field`` + - ``number_field`` - can use ``min``, ``max``, ``step`` + - ``radio_button`` + - ``resolution_field`` (used for specifying resolution dimensions necessary for VNC) + - ``select`` - can use ``options`` + - ``text_area`` + - ``text_field`` + + Some other fields that have varying support across different browsers + include: ``range_field``, ``date_field``, ``search_field``, + ``email_field``, ``telephone_field``, ``url_field``, ``password_field``. + + Default + Accepts any text + + .. code-block:: yaml + + widget: "text_field" + + Example + Accepts only numbers + + .. code-block:: yaml + + widget: "number_field" + +.. describe:: value (String, null) + + the default value used for the input field + + Default + None + + .. code-block:: yaml + + value: "" + + Example + Set default value of 5 + + .. code-block:: yaml + + value: "5" + + .. warning:: + + Values get cached so that users do not need to repeat previous session + submissions. So this default value will only appear for the user if + they have no cached value. + +.. describe:: label (String, null) + + the label displayed above the input field + + Default + Uses the name of the attribute + + Example + Better describe the attribute + + .. code-block:: yaml + + label: "Number of nodes" + +.. describe:: required (Boolean, null) + + whether this field must be filled out before submitting the form + + Default + Not required by default + + .. code-block:: yaml + + required: false + + Example + Make field required + + .. code-block:: yaml + + required: true + +.. describe:: help (String, null) + + help text that appears below the field (can be written in Markdown_) + + Default + No help text appears below input field + + .. code-block:: yaml + + help: null + + Example + Leave a long descriptive help message using Markdown_ + + .. code-block:: yaml + + help: | + Please fill in this field with **one** of the following + options: + + - `red` + - `blue` + - `green` + +.. describe:: pattern (String, null) + + a regular expression that the control's value is checked against (only + applies to widgets: ``text_field``, ``search_field``, ``telephone_field``, + ``url_field``, ``email_field``, ``password_field``) + + Default + No pattern + + .. code-block:: yaml + + pattern: null + + Example + Only accept three letter country codes + + .. code-block:: yaml + + pattern: "[A-Za-z]{3}" + +.. describe:: min (Integer, null) + + specifies minimum value for this item, which must not be greater than its + maximum value (only applies to widget: ``number_field``) + + Default + No minimum value + + .. code-block:: yaml + + min: null + + Example + Set minimum value of 5 + + .. code-block:: yaml + + min: 5 + +.. describe:: max (Integer, null) + + specifies maximum value for this item, which must not be less than its + minimum value (only applies to widget: ``number_field``) + + Default + No maximum value + + .. code-block:: yaml + + max: null + + Example + Set maximum value of 15 + + .. code-block:: yaml + + max: 15 + +.. describe:: step (Integer, null) + + works with ``min`` and ``max`` options to limit the increments at which a + value can be set, it can be the string ``"any"`` or positive floating + point number (only applies to widget: ``number_field``) + + Default + No step size + + .. code-block:: yaml + + step: null + + Example + Only accept integer values + + .. code-block:: yaml + + step: 1 + +.. describe:: options (Array>, null) + + A list of options for a ``select`` widget. In the simplest form + you can have a list of strings like ``chevy`` in example below. + They can also be in the form ``[