[1.1.0] Compatibility with AutoPkg 2.x (Python 3)

[homebysix]

Summary

A major update to the AutoPkg command line tool is just around the corner. AutoPkg 2.0 is due in early February 2020, and it will run in Python 3 only. In order for JSSImporter to work with AutoPkg 2.0, a few moderate tweaks will be necessary to both JSSImporter and python-jss. I'm opening this issue as a place to track those tweaks and link to pull requests that might resolve them.

Proposed changes

Here's the high-level list of the changes I think that will need to be made. I already have development branches that incorporate many of these changes, but it would be good to have a second or third pair of eyes.

For reference, the pull requests that contain these changes are here:

• AutoPkg 2.x (Python 3) compatibility adjustments #174
• AutoPkg 2.x (Python 3) compatibility adjustments python-jss#95

Imports

Most imports work fine in both Python 2 and Python 3, but python-jss's import uapiobjects as uapi fails. This should be addressed by homebysix/python-jss@980b627.

Print statements

Simple matter of changing print foo to print(foo) in most cases. Should be addressed by homebysix/python-jss@5a88ff1.

xml.etree.ElementTree

This is a bit out of my wheelhouse, but as I understand it there are two implementations of ElementTree, and we prefer the Python implmentation. A regular from xml.etree import ElementTree results in the C implementation, which Shea was working around using Element = ElementTree._Element_Py. That no longer seems to work the same in Python 3.7.x, but if we pop the standard xml.etree.ElementTree, we can then use ElementTree = importlib.import_module('xml.etree.ElementTree') to load the Python module. I've made this adjustment to python-jss in homebysix/python-jss@2fdc4b7 and to JSSImporter in homebysix@fc6bfbb. I'm not sure of the underlying mechanics here, but it seems to work in Python 3.7.x while retaining compatibility with Python 2.7.x.

String types

JSSImporter and python-jss include a few references to unicode and basestring, which are only used in Python 2. While it's possible to use six.string_types to resolve this, I've chosen instead to map unicode and basestring to Python 3's str, which includes both plain text and unicode. This is similar to the approach taken by Munki. I've made the change for JSSImporter in homebysix@ec8077d and for python-jss in homebysix/python-jss@8904c92 and homebysix/python-jss@b3e2c16. I've also changed unicode() calls to str.encode("utf-8") in homebysix/python-jss@0d0452f.

dict.iteritems()

Calls to dict.iteritems() should be replaced by the equivalent use of dict.items() or dict.keys(). I've adjusted this in homebysix/python-jss@daf7b4d for python-jss and homebysix@5240ab7 and homebysix@5860666 for JSSImporter.

Encoding and decoding

In Python 3, some functions produce bytes output instead of strings. Specifically, calls to requests.response.content and subprocess.check_output() that worked in Python 2 need to be explicitly decoded or cast as strings before they'll work in Python 3. I've done this in homebysix/python-jss@1f70b24 and homebysix/python-jss@f0c359d.

FoundationPlist and GURL

These Munki modules have both been updated upstream, including some adjustments for Python 3 compatibility. I've brought in those changes in homebysix/python-jss@18d1093 and homebysix/python-jss@c68eb81.

Testing

I would love for least one or two others to validate that my branches allow them to successfully run their JSS recipes/overrides on both AutoPkg 1.4.1 and AutoPkg 2.0 RC1.

I've found the simplest way to put the test code in place is to overwrite the folder at /Library/Application Support/JSSImporter/jss and the file at /Library/AutoPkg/autopkglib/JSSImporter.py with the corresponding paths from my branches of the python-jss and JSSImporter projects. You'll need to update your override trust info as well.

Next steps

If the testing goes well, feel free to merge the changes. It may be wise to post minimum viable releases for Python 3 compatibility soon so that Mac admins can be ready for the early February AutoPkg 2.0 release.

There are a few non-urgent issues raised by pylint that can be addressed in the future, and I also have ideas for how to incorporate pre-commit, but those items don't need to hold up the process at this point.

Thanks for your consideration!

[grahampugh]

Hi @homebysix - this is amazing work, thank you so much for your efforts.

In my initial tests, I have found that my usual build method (using the makefile which utilises munkipkg) to create the installation package is no longer incorporating requests and boto into the package. To make JSSImporter work, I therefore have to manually install them afterwards using pip3 install requests boto --user. Any ideas what might have changed to cause this?

This is my normal quick build-and-install method while testing:

make clean; make; sudo installer -pkg pkg/jssimporter/build/jssimporter-1.1.0.pkg -tgt /

[grahampugh]

I believe I have now solved this - I don't think the bundled-in requests were ever really being used. I will pull your branches into the jssimporter organisation so that I can merge in the subsequent changes.