YALB-1550: Single Content Sync

[dblanken-yale]

YALB-1550: Single Content Sync

Description of work

• Add single content sync and supporting functionality around it for successful exporting of content/media/taxnonomy via views
• Patch ZipArchive for single content sync
• Create export code for media and taxonomy terms
• Creates new views for nodes, media, and taxonomy off of /config/system to assist in graphical exports

Functional testing steps:

Via drush

• Bring this repo local and run composer update single_content_sync
• In comand line:
  • Exporting
    • Run the following command: lando drush content:export nodes ./exports --translate --assets
    • Note the directory location of the zip, specifically after /app/web/
    • Note that in a real world example, you'd copy this file out and copy it to a new site under /web somewhere, noting the location to use in the drush import command.
    • Go onto the website and delete a node--make note of the node you deleted so you can verify that it was re-imported later.
  • Importing
    • In command line, type lando drush content:import ./exports/<name_of_the_zip_file_that_was_made>
    • Verify that after a little bit, you see [notice] Message: Successfully imported the content. and received no errors.
    • Visit the site and look for the node you deleted.
  • Do the same for media: lando drush content:export media ./export --assets --translate
  • Do the same for taxonomy terms: lando drush content:export taxonomy_term ./export --assets --translate

The reason the --all-content flag is not used is that it includes block_content as well. If we'd like that, then --all-content could be passed to the first node version and it would return them all. Technically since the UUIDs would be the same, it should be ok, but I'd be more explicit of what we're targeting.

Via site

• Log in as the administrator
• Visit Configuration->System->Export Node
• Verify that Export content is in the Action drop down at the bottom of the listing
• Visit Configuration->System->Export Node->Export Media
• Verify that Export media is in the Action drop down at the bottom of the listing
• Visit Configuration->System->Export Node->Export Taxonomy
• Verify that Export taxonomy is in the Action drop down at the bottom of the listing
• Feel free to export any of those three to verify that they work.
  • Click on any of the above views
  • Click multiple check boxes in the list to select those items to export
  • Ensure the Export X is selected for the bulk action and click the Apply to selected items button
  • A link should be generated in the message back to the user. Click the link to download the zip file of what you selected.
• In an incognito browser, visit the site and log in as a CAS user
• Verify that you cannot see any mention of imports or export links

Importing

Due to the size of the content zips, importing should be done via Drush

[github-actions]

Created multidev environment pr-409 for yalesites-platform.

[dblanken-yale]

I have 2 worries:

1. In order to not have the new Taxonomy view present, I hung the permissions on an already existing permission: export configuration
2. Including ys_starterkit as part of the config split for starterkit; since we need it enabled by default, I almost feel we should temporarily break the specifics into a temporary module that we would then migrate into ys_starterkit once we're certain it can be retired. Right now it's included or else the export media and taxonomy are available to non-admins. :(

[vinmassaro]

There's work being done for drush import/export in this issue. Do we need our own Drush commands?

[dblanken-yale]

Oh yeah you're right! They're controlling the entity type so that was the only thing I thought of as to why we'd need to keep mine, but they've already thought of it. Ya I'll take it out.

[codechefmarc]

Most of this worked great, but a few issues:

• The Manage All Taxonomy view was still there after I disabled the Starterkit config config split, I think because that config is in /config/sync and not /config/starterkit_config
• I was able to import smaller exports no problem (taxonomy terms) but even just a few nodes zipped into a 126MB zip caused my local and Pantheon to not accept the file. I got an Ajax error in the console:

Local:

An AJAX HTTP error occurred.
HTTP Result Code: 413
Debugging information follows.
Path: /admin/content/import?element_parents=upload_fid&ajax_form=1
StatusText: error
ResponseText: 
413 Request Entity Too Large
413 Request Entity Too Large
nginx
"

Pantheon:

"An AJAX HTTP request terminated abnormally.
Debugging information follows.
Path: /admin/content/import?element_parents=upload_fid&ajax_form=1
StatusText: error
ReadyState: 0"

[dblanken-yale]

@codechefmarc For the first point, I am so glad you said that. My biggest worry was hanging the permissions of the "Manage All Taxonomy" on another permission set in order to attempt to hide it, but I can just include it in the config split! I never even thought of the fact that it has a yaml config! DUH!

As for the other, I've not had that issue, but did mostly do my testing locally, and I bet for the multi devs I did small sets of files. ☹️ I'll have to dive into that more as well.

Thank you so much for this info. I'll get to work on it.

[dblanken-yale]

It looks as if we have some limits on the file size uploads both in nginx and in PHP. Locally using lando, nginx looks to be limiting to 80M, while PHP is limiting to 100M. I'm very confused as that means my uploads must have been less than 80M? I could have swore I went higher.

In any event, the uploads are indeed being limited by nginx, resulting in a 413: Request Entity Too Large. I'm looking into if we can fix this with a .htaccess modification or if we have to look into overloading the php.ini/nginx.conf.

[codechefmarc]

If this is the file size limit for uploads, perhaps we can get around this by doing the hosting of the zips on S3 or via another repo and importing from those places instead of a direct upload? That may be some custom code, but, would be fun to figure that one out. (Well for fun for some!)

[dblanken-yale]

@vinmassaro contacted me and wanted me to check out the status of the drush commands they're working on. If it's good enough, we might be able to get around it for now using drush. But I bet that's why they implemented the S3 feature. haha

It looks like their drush command works similarly to mine where it exports to a directory and we have to go get it. The import needs to be accessible by the web root to import. I am still going to look more into it. I've not dealt with S3 so it'd be neat to try if this doesn't get us there.

[dblanken-yale]

Great news! I was able to get drush debugging done, and noticed that it was erroring on the local files directory (permissions maybe?). In any event, I wiped completely and did a fresh pull with the patch and it worked! I'm rebuilding now both a yalesites dev and a dev version of yalesites.yale.edu so I can get real content from them to test it one more time. I think it's working! I'll provide instructions in the PR above when I am certain.

[codechefmarc]

The drush command worked beautifully to import! I tested locally only for now (didn't want to mess up the multidev) but also question: I had to move the zip file into the web folder for it to work (I tried using a relative path outside of web but it didn't find it). So, would this take an external URL so we could do S3 / GitHub repo to store these zip files? I assume you looked into that.

[dblanken-yale]

Ya that is a limitation currently--when this command was worked on, the S3 implementation hadn't been done yet, and the branch is behind that commit. My guess is that it'll be added, but for now it is only zip files within the web directory. ☹️

Currently there is a "getRealDirectory" method that is called that returns the path, but it currently doesn't account for the S3 situation.

[dblanken-yale]

And now that I'm looking at the implementation, it's not that the zip file can reside in S3, it's that image assets residing in S3 that are referenced by a node properly get imported (no save is attempted). So we might still need some work pointing to a URL to download a zip.

https://git.drupalcode.org/project/single_content_sync/-/merge_requests/63/diffs#161d21d70da7441b94306f7cff37f4a82306c7da_364_381

[dblanken-yale]

Placing this here in case we need it ever, there was talk of a drush dl command that was dropped from Drush 9+. Some reference this example as a way to attempt to recreate this in a custom drush command if ever needed:

https://github.com/drush-ops/drush/blob/master/examples/Commands/SyncViaHttpCommands.php#L60

[dblanken-yale]

@vinmassaro The changes we talked about are in place.

• config split for this specifically has been removed
• all user-facing views have removed the bulk export option
• new views have been created inside of Configuration->System specifically to export nodes, media, and taxonomy
• removed the patch as the drush command is now in the main release
• these new views have permissions related to being able to export configs (ideally UID 1)

When you can, give it another try, and thanks for talking this through with me yesterday.

[vinmassaro]

Looks great. I was able to export starterkit content and import it into a clean local install with no issue. Approved!