Skip to content

Commit

Permalink
Merge pull request #9959 from IQSS/9590-faster-redeploy
Browse files Browse the repository at this point in the history
Faster redeployment for developers, skip deploy, remove version from war in dev
  • Loading branch information
pdurbin authored Feb 23, 2024
2 parents 82585f9 + fe166f7 commit 1752b46
Show file tree
Hide file tree
Showing 61 changed files with 324 additions and 211 deletions.
1 change: 1 addition & 0 deletions .env
Original file line number Diff line number Diff line change
Expand Up @@ -2,3 +2,4 @@ APP_IMAGE=gdcc/dataverse:unstable
POSTGRES_VERSION=13
DATAVERSE_DB_USER=dataverse
SOLR_VERSION=9.3.0
SKIP_DEPLOY=0
12 changes: 12 additions & 0 deletions conf/proxy/Caddyfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
# This configuration is intended to be used with Caddy, a very small high perf proxy.
# It will serve the application containers Payara Admin GUI via HTTP instead of HTTPS,
# avoiding the trouble of self signed certificates for local development.

:4848 {
reverse_proxy https://dataverse:4848 {
transport http {
tls_insecure_skip_verify
}
header_down Location "^https://" "http://"
}
}
5 changes: 5 additions & 0 deletions doc/release-notes/9590-faster-redeploy.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
In the Container Guide, documentation for developers on how to quickly redeploy code has been added for Netbeans and improved for IntelliJ.

Also in the context of containers, a new option to skip deployment has been added and the war file is now consistently named "dataverse.war" rather than having a version in the filename, such as "dataverse-6.1.war". This predictability makes tooling easier.

Finally, an option to create tabs in the guides using [Sphinx Tabs](https://sphinx-tabs.readthedocs.io) has been added. (You can see the tabs in action in the "dev usage" page of the Container Guide.) To continue building the guides, you will need to install this new dependency by re-running `pip install -r requirements.txt`.
3 changes: 3 additions & 0 deletions doc/sphinx-guides/requirements.txt
Original file line number Diff line number Diff line change
Expand Up @@ -5,3 +5,6 @@ sphinx-icon==0.1.2

# Markdown support
myst-parser==2.0.0

# tabs
sphinx-tabs==3.4.5
8 changes: 8 additions & 0 deletions doc/sphinx-guides/source/_static/docsdataverse_org.css
Original file line number Diff line number Diff line change
Expand Up @@ -182,3 +182,11 @@ div.form-group .glyphicon.glyphicon-asterisk {font-size: .5em; vertical-align: t
pre {
white-space: pre-wrap;
}

div.sphinx-tabs {
width: 100%;
}

li div.sphinx-tabs {
padding-left: 0;
}
2 changes: 1 addition & 1 deletion doc/sphinx-guides/source/admin/metadatacustomization.rst
Original file line number Diff line number Diff line change
Expand Up @@ -413,7 +413,7 @@ Setting Up a Dev Environment for Testing

You have several options for setting up a dev environment for testing metadata block changes:

- Docker: See :doc:`/container/index`.
- Docker: See :doc:`/container/running/metadata-blocks` in the Container Guide.
- AWS deployment: See the :doc:`/developers/deployment` section of the Developer Guide.
- Full dev environment: See the :doc:`/developers/dev-environment` section of the Developer Guide.

Expand Down
1 change: 1 addition & 0 deletions doc/sphinx-guides/source/conf.py
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,7 @@
'sphinx.ext.graphviz',
'sphinxcontrib.icon',
'myst_parser',
'sphinx_tabs.tabs',
]

# Add any paths that contain templates here, relative to this directory.
Expand Down
20 changes: 15 additions & 5 deletions doc/sphinx-guides/source/container/base-image.rst
Original file line number Diff line number Diff line change
Expand Up @@ -217,7 +217,14 @@ provides. These are mostly based on environment variables (very common with cont
- ``0``
- Bool, ``0|1``
- Enable the dynamic "hot" reloads of files when changed in a deployment. Useful for development,
when new artifacts are copied into the running domain.
when new artifacts are copied into the running domain. Also, export Dataverse specific environment variables
``DATAVERSE_JSF_PROJECT_STAGE=Development`` and ``DATAVERSE_JSF_REFRESH_PERIOD=0`` to enable dynamic JSF page
reloads.
* - ``SKIP_DEPLOY``
- ``0``
- Bool, ``0|1`` or ``false|true``
- When active, do not deploy applications from ``DEPLOY_DIR`` (see below), just start the application server.
Will still execute any provided init scripts and only skip deployments within the default init scripts.
* - ``DATAVERSE_HTTP_TIMEOUT``
- ``900``
- Seconds
Expand Down Expand Up @@ -272,7 +279,8 @@ building upon it. You can also use these for references in scripts, etc.
(Might be reused for Dataverse one day)
* - ``DEPLOY_DIR``
- ``${HOME_DIR}/deployments``
- Any EAR or WAR file, exploded WAR directory etc are autodeployed on start
- Any EAR or WAR file, exploded WAR directory etc are autodeployed on start.
See also ``SKIP_DEPLOY`` above.
* - ``DOMAIN_DIR``
- ``${PAYARA_DIR}/glassfish`` ``/domains/${DOMAIN_NAME}``
- Path to root of the Payara domain applications will be deployed into. Usually ``${DOMAIN_NAME}`` will be ``domain1``.
Expand All @@ -299,9 +307,9 @@ named Docker volume in these places to avoid data loss, gain performance and/or
- Description
* - ``STORAGE_DIR``
- ``/dv``
- This place is writeable by the Payara user, making it usable as a place to store research data, customizations
or other. Images inheriting the base image should create distinct folders here, backed by different
mounted volumes.
- This place is writeable by the Payara user, making it usable as a place to store research data, customizations or other.
Images inheriting the base image should create distinct folders here, backed by different mounted volumes.
Enforce correct filesystem permissions on the mounted volume using ``fix-fs-perms.sh`` from :doc:`configbaker-image` or similar scripts.
* - ``SECRETS_DIR``
- ``/secrets``
- Mount secrets or other here, being picked up automatically by
Expand Down Expand Up @@ -353,6 +361,8 @@ Other Hints

By default, ``domain1`` is enabled to use the ``G1GC`` garbage collector.

To access the Payara Admin Console or use the ``asadmin`` command, use username ``admin`` and password ``admin``.

For running a Java application within a Linux based container, the support for CGroups is essential. It has been
included and activated by default since Java 8u192, Java 11 LTS and later. If you are interested in more details,
you can read about those in a few places like https://developers.redhat.com/articles/2022/04/19/java-17-whats-new-openjdks-container-awareness,
Expand Down
211 changes: 185 additions & 26 deletions doc/sphinx-guides/source/container/dev-usage.rst
Original file line number Diff line number Diff line change
Expand Up @@ -144,43 +144,202 @@ Alternatives:
Redeploying
-----------

Rebuild and Running Images
^^^^^^^^^^^^^^^^^^^^^^^^^^
The safest and most reliable way to redeploy code is to stop the running containers (with Ctrl-c if you started them in the foreground) and then build and run them again with ``mvn -Pct clean package docker:run``.
Safe, but also slowing down the development cycle a lot.

The safest way to redeploy code is to stop the running containers (with Ctrl-c if you started them in the foreground) and then build and run them again with ``mvn -Pct clean package docker:run``.
Triggering redeployment of changes using an IDE can greatly improve your feedback loop when changing code.
You have at least two options:

IntelliJ IDEA Ultimate and Payara Platform Tools
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
#. Use builtin features of IDEs or `IDE plugins from Payara <https://docs.payara.fish/community/docs/documentation/ecosystem/ecosystem.html>`_.
#. Use a paid product like `JRebel <https://www.jrebel.com/>`_.

If you have IntelliJ IDEA Ultimate (note that `free educational licenses <https://www.jetbrains.com/community/education/>`_ are available), you can install `Payara Platform Tools <https://plugins.jetbrains.com/plugin/15114-payara-platform-tools>`_ which can dramatically improve your feedback loop when iterating on code.
The main differences between the first and the second options are support for hot deploys of non-class files and limitations in what the JVM HotswapAgent can do for you.
Find more details in a `blog article by JRebel <https://www.jrebel.com/blog/java-hotswap-guide>`_.

The following steps are suggested:
.. _ide-trigger-code-deploy:

- Go to the Payara admin console (either at https://localhost:4848 or http://localhost:4849) and undeploy the dataverse application under "Applications".
- Install Payara Platform Tools.
- Under "Server":
IDE Triggered Code Re-Deployments
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

- Click "Run" then "Edit Configurations".
- Click the plus sign and scroll down to Payara Server and click "Remote".
- For "Name" put "Payara in Docker" or something reasonable.
- Under "Application server" select a local directory that has the same version of Payara used in the container. This should match the version of Payara mentioned in the Installation Guide under :ref:`payara`.
- Change "Admin Server Port" to 4849.
- For username, put "admin".
- For password, put "admin".
To make use of builtin features or Payara IDE Tools (option 1), please follow steps below.
Note that using this method, you may redeploy a complete WAR or single methods.
Redeploying WARs supports swapping and adding classes and non-code materials, but is slower (still faster than rebuilding containers).
Hotswapping methods requires using JDWP (Debug Mode), but does not allow switching non-code material or adding classes.

- Under "Deployment":
#. | Download the version of Payara shown in :ref:`install-payara-dev` and unzip it to a reasonable location such as ``/usr/local/payara6``.
| - Note that Payara can also be downloaded from `Maven Central <https://mvnrepository.com/artifact/fish.payara.distributions/payara>`_.
| - Note that another way to check the expected version of Payara is to run this command:
| ``mvn help:evaluate -Dexpression=payara.version -q -DforceStdout``
- Click the plus button and clien "Artifact" then "dataverse:war".
#. Install Payara Tools plugin in your IDE:

- Under "Startup/Connection":
.. tabs::
.. group-tab:: Netbeans

- Click "Debug" and change the port to 9009.
This step is not necessary for Netbeans. The feature is builtin.

- Click "Run" and then "Debug Payara in Docker". This initial deployment will take some time.
- Go to http://localhost:8080/api/info/version and make sure the API is responding.
- Edit ``Info.java`` and make a small change to the ``/api/info/version`` code.
- Click "Run" then "Debugging Actions" then "Reload Changed Classes". The deployment should only take a few seconds.
- Go to http://localhost:8080/api/info/version and verify the change you made.
.. group-tab:: IntelliJ

**Requires IntelliJ Ultimate!**
(Note that `free educational licenses <https://www.jetbrains.com/community/education/>`_ are available)

.. image:: img/intellij-payara-plugin-install.png

#. Configure a connection to Payara:

.. tabs::
.. group-tab:: Netbeans

Launch Netbeans and click "Tools" and then "Servers". Click "Add Server" and select "Payara Server" and set the installation location to ``/usr/local/payara6`` (or wherever you unzipped Payara). Choose "Remote Domain". Use the settings in the screenshot below. Most of the defaults are fine.

Under "Common", the username and password should be "admin". Make sure "Enable Hot Deploy" is checked.

.. image:: img/netbeans-servers-common.png

Under "Java", change the debug port to 9009.

.. image:: img/netbeans-servers-java.png

Open the project properties (under "File"), navigate to "Compile" and make sure "Compile on Save" is checked.

.. image:: img/netbeans-compile.png

Under "Run", under "Server", select "Payara Server". Make sure "Deploy on Save" is checked.

.. image:: img/netbeans-run.png

.. group-tab:: IntelliJ
Create a new running configuration with a "Remote Payara".
(Open dialog by clicking "Run", then "Edit Configurations")

.. image:: img/intellij-payara-add-new-config.png

Click on "Configure" next to "Application Server".
Add an application server and select unzipped local directory.

.. image:: img/intellij-payara-config-add-server.png

Add admin password "admin" and add "building artifact" before launch.
Make sure to select the WAR, *not* exploded!

.. image:: img/intellij-payara-config-server.png

Go to "Deployment" tab and add the Dataverse WAR, *not* exploded!.

.. image:: img/intellij-payara-config-deployment.png

Go to "Startup/Connection" tab, select "Debug" and change port to ``9009``.

.. image:: img/intellij-payara-config-startup.png

You might want to tweak the hot deploy behavior in the "Server" tab now.
"Update action" can be found in the run window (see below).
"Frame deactivation" means switching from IntelliJ window to something else, e.g. your browser.
*Note: static resources like properties, XHTML etc will only update when redeploying!*

.. image:: img/intellij-payara-config-server-behaviour.png

#. Start all the containers, but take care to skip application deployment.

.. tabs::
.. group-tab:: Maven
``mvn -Pct docker:run -Dapp.skipDeploy``

Run above command in your terminal to start containers in foreground and skip deployment.
See cheat sheet above for more options.
Note that this command either assumes you built the :doc:`app-image` first or will download it from Docker Hub.
.. group-tab:: Compose
``SKIP_DEPLOY=1 docker compose -f docker-compose-dev.yml up``

Run above command in your terminal to start containers in foreground and skip deployment.
See cheat sheet above for more options.
Note that this command either assumes you built the :doc:`app-image` first or will download it from Docker Hub.
.. group-tab:: IntelliJ
You can create a service configuration to automatically start services for you.

**IMPORTANT**: This requires installation of the `Docker plugin <https://plugins.jetbrains.com/plugin/7724-docker>`_.

**NOTE**: You might need to change the Docker Compose executable in your IDE settings to ``docker`` if you have no ``docker-compose`` bin (*File > Settings > Build > Docker > Tools*).

.. image:: img/intellij-compose-add-new-config.png

Give your configuration a meaningful name, select the compose file to use (in this case the default one), add the environment variable ``SKIP_DEPLOY=1``, and optionally select the services to start.
You might also want to change other options like attaching to containers to view the logs within the "Services" tab.

.. image:: img/intellij-compose-setup.png

Now run the configuration to prepare for deployment and watch it unfold in the "Services" tab.

.. image:: img/intellij-compose-run.png
.. image:: img/intellij-compose-services.png

Note: the Admin Console can be reached at http://localhost:4848 or https://localhost:4949

#. To deploy the application to the running server, use the configured tools to deploy.
Using the "Run" configuration only deploys and enables redeploys, while running "Debug" enables hot swapping of classes via JDWP.

.. tabs::
.. group-tab:: Netbeans

Click "Debug" then "Debug Project". After some time, Dataverse will be deployed.

Try making a code change, perhaps to ``Info.java``.

Click "Debug" and then "Apply Code Changes". If the change was correctly applied, you should see output similar to this:

.. code-block::
Classes to reload:
edu.harvard.iq.dataverse.api.Info
Code updated
Check to make sure the change is live by visiting, for example, http://localhost:8080/api/info/version

See below for a `video <https://www.youtube.com/watch?v=yo3aKOg96f0>`_ demonstrating the steps above but please note that the ports used have changed and now that we have the concept of "skip deploy" the undeployment step shown is no longer necessary.

.. raw:: html

<iframe width="560" height="315" src="https://www.youtube.com/embed/yo3aKOg96f0?si=2OCDj-_fmQFBMOLc" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" allowfullscreen></iframe>

.. group-tab:: IntelliJ
Choose "Run" or "Debug" in the toolbar.

.. image:: img/intellij-payara-run-toolbar.png

Watch the WAR build and the deployment unfold.
Note the "Update" action button (see config to change its behavior).

.. image:: img/intellij-payara-run-output.png

Manually hotswap classes in "Debug" mode via "Run" > "Debugging Actions" > "Reload Changed Classes".

.. image:: img/intellij-payara-run-menu-reload.png

Note: in the background, the bootstrap job will wait for Dataverse to be deployed and responsive.
When your IDE automatically opens the URL a newly deployed, not bootstrapped Dataverse application, it might take some more time and page refreshes until the job finishes.

IDE Triggered Non-Code Re-Deployments
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Either redeploy the WAR (see above), use JRebel or look into copying files into the exploded WAR within the running container.
The steps below describe options to enable the later in different IDEs.

.. tabs::
.. group-tab:: IntelliJ

This imitates the Netbeans builtin function to copy changes to files under ``src/main/webapp`` into a destination folder.
It is different in the way that it will copy the files into the running container deployment without using a bind mount.

1. Install the `File Watchers plugin <https://plugins.jetbrains.com/plugin/7177-file-watchers>`_
2. Import the :download:`watchers.xml <../../../../docker/util/intellij/watchers.xml>` file at *File > Settings > Tools > File Watchers*
3. Once you have the deployment running (see above), editing files under ``src/main/webapp`` will be copied into the container after saving the edited file.
Note: by default, IDE auto-saves will not trigger the copy.
4. Changes are visible once you reload the browser window.

**IMPORTANT**: This tool assumes you are using the :ref:`ide-trigger-code-deploy` method to run Dataverse.

**IMPORTANT**: This tool uses a Bash shell script and is thus limited to Mac and Linux OS.

Using a Debugger
----------------
Expand Down
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
6 changes: 2 additions & 4 deletions doc/sphinx-guides/source/developers/classic-dev-env.rst
Original file line number Diff line number Diff line change
Expand Up @@ -88,6 +88,8 @@ On Mac, run this command:

On Linux, install ``jq`` from your package manager or download a binary from https://stedolan.github.io/jq/

.. _install-payara-dev:

Install Payara
~~~~~~~~~~~~~~

Expand Down Expand Up @@ -260,7 +262,3 @@ Next Steps
If you can log in to the Dataverse installation, great! If not, please see the :doc:`troubleshooting` section. For further assistance, please see "Getting Help" in the :doc:`intro` section.

You're almost ready to start hacking on code. Now that the installer script has you up and running, you need to continue on to the :doc:`tips` section to get set up to deploy code from your IDE or the command line.

----

Previous: :doc:`intro` | Next: :doc:`tips`
6 changes: 1 addition & 5 deletions doc/sphinx-guides/source/developers/coding-style.rst
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,7 @@ Place curly braces according to the style below, which is an example you can see
Format Code You Changed with Netbeans
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

As you probably gathered from the :doc:`dev-environment` section, IQSS has standardized on Netbeans. It is much appreciated when you format your code (but only the code you touched!) using the out-of-the-box Netbeans configuration. If you have created an entirely new Java class, you can just click Source -> Format. If you are adjusting code in an existing class, highlight the code you changed and then click Source -> Format. Keeping the "diff" in your pull requests small makes them easier to code review.
IQSS has standardized on Netbeans. It is much appreciated when you format your code (but only the code you touched!) using the out-of-the-box Netbeans configuration. If you have created an entirely new Java class, you can just click Source -> Format. If you are adjusting code in an existing class, highlight the code you changed and then click Source -> Format. Keeping the "diff" in your pull requests small makes them easier to code review.

Checking Your Formatting With Checkstyle
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Expand Down Expand Up @@ -131,7 +131,3 @@ Bike Shedding
What color should the `bike shed <https://en.wiktionary.org/wiki/bikeshedding>`_ be? :)

Come debate with us about coding style in this Google doc that has public comments enabled: https://docs.google.com/document/d/1KTd3FpM1BI3HlBofaZjMmBiQEJtFf11jiiGpQeJzy7A/edit?usp=sharing

----

Previous: :doc:`debugging` | Next: :doc:`deployment`
4 changes: 0 additions & 4 deletions doc/sphinx-guides/source/developers/containers.rst
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,3 @@ Using Containers for Reproducible Research
------------------------------------------

Please see :ref:`research-code` in the User Guide for this related topic.

----

Previous: :doc:`deployment` | Next: :doc:`making-releases`
Loading

0 comments on commit 1752b46

Please sign in to comment.