Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Upgrade from python3.8 to python3.10 #5311

Closed
wants to merge 18 commits into from
Closed

Upgrade from python3.8 to python3.10 #5311

wants to merge 18 commits into from

Conversation

jumaffre
Copy link
Contributor

@jumaffre jumaffre commented May 30, 2023

This is part of the upgrade to Ubuntu 22.04 for SNP/virtual builds, where python3.10 is available via pip by default.

The couple of Ubuntu hosted CI executors are also upgraded to Ubuntu 22.04 so that python3.10 is available by default.

@ghost
Copy link

ghost commented May 30, 2023

python_3_10@71279 aka 20230602.15 vs main ewma over 20 builds from 70865 to 71259

Click to see table

main

build_id build_number Commit latency factor ls_virtual_cft^ pi_ls_virtual_cft^ ls_jwt_virtual_cft^ pi_ls_jwt_virtual_cft^ ls_sgx_cft^ ls_sgx_cft_mem ls_js_virtual_cft^ pi_ls_sgx_cft^ pi_ls_sgx_cft_mem ls_jwt_sgx_cft^ ls_jwt_sgx_cft_mem ls_full_js_virtual_cft^ pi_ls_jwt_sgx_cft^ pi_ls_jwt_sgx_cft_mem ls_js_jwt_virtual_cft^ ls_js_sgx_cft^ ls_js_sgx_cft_mem hist_sgx_cft^ ls_full_js_sgx_cft^ ls_full_js_sgx_cft_mem ls_js_jwt_sgx_cft^ ls_js_jwt_sgx_cft_mem RB put (/s)^ CHAMP put (/s)^ RB get (/s)^ CHAMP get (/s)^ pi_basic_virtual_cft^ pi_basic_sgx_cft^ pi_basic_sgx_cft_mem
70865 20230525.9 0.786008 45675 48739.4 12579 13109.1 20214.6 1.88908e+07 4453.97 20464.8 1.25993e+07 6358.05 1.67936e+07 3476.35 6502.5 6.30784e+06 3219.82 1710.88 1.05021e+07 46328.9 1418.61 1.05021e+07 1406.75 1.05021e+07 833914 1.1784e+06 8.17027e+06 3.08415e+07 nan nan nan
70877 20230525.13 0.794755 45934.3 48428.5 12570.5 13170.2 20359 1.88908e+07 4461.48 20465.8 1.25993e+07 6676.38 1.67936e+07 3475.11 6549.6 6.30784e+06 3255.95 1712.51 1.05021e+07 50575.2 1417.26 1.05021e+07 1409.9 1.05021e+07 838216 1.17488e+06 8.16681e+06 3.0809e+07 nan nan nan
70890 20230525.18 0.824966 45752 45085.9 12519.6 12887.1 20256.8 1.88908e+07 4466.87 20475.5 1.25993e+07 6687.4 1.67936e+07 3494.19 6511.7 6.30784e+06 3294.38 1713.61 1.05021e+07 48388.8 1412.75 1.05021e+07 1405.36 1.05021e+07 833584 1.17758e+06 8.15433e+06 3.07489e+07 nan nan nan
70913 20230526.1 0.775 45465.9 48431.1 12670.9 12951.5 20281.4 1.88908e+07 4459.58 20525.5 1.25993e+07 6712.33 1.67936e+07 3557.35 6555.2 6.30784e+06 3239.73 1711.13 1.05021e+07 46459.8 1430.04 1.05021e+07 1402.67 1.05021e+07 818517 1.17913e+06 8.15436e+06 2.98056e+07 nan nan nan
70936 20230526.10 0.821101 45729.2 48747.5 12462.9 13129.4 20373.4 1.88908e+07 4432.86 20532.5 1.25993e+07 6720.57 1.67936e+07 3450.04 6519.8 6.30784e+06 3229.12 1731.27 1.05021e+07 48927.4 1418.71 1.05021e+07 1419.35 1.05021e+07 823856 1.18116e+06 8.15235e+06 2.99661e+07 56514.4 23089.8 1.25993e+07
70948 20230529.3 0.811405 46102.1 48458.8 12669.6 12849.8 20392.1 1.88908e+07 4416.14 20488.2 1.25993e+07 6698.89 1.67936e+07 3577.84 6554.7 6.30784e+06 3225.76 1714.14 1.05021e+07 50943.7 1422.96 1.05021e+07 1422.19 1.05021e+07 822933 1.18082e+06 8.17415e+06 3.06518e+07 56351.3 23001.5 1.25993e+07
70961 20230530.3 0.813727 45788.8 48571 12528.1 12996.4 20242.1 1.88908e+07 4419.43 20481.9 1.25993e+07 6679.21 1.67936e+07 3566.12 6510.7 6.30784e+06 3252.82 1715.43 1.05021e+07 50885.1 1420.64 1.05021e+07 1422.18 1.05021e+07 838192 1.18066e+06 8.15543e+06 3.08722e+07 55477.3 23050 1.25993e+07
70978 20230530.9 0.805699 45408.6 48763.2 12628.7 13046.8 20312.6 1.88908e+07 4441.63 20464.3 1.25993e+07 6431.76 1.67936e+07 3585.68 6547.6 6.30784e+06 3267.74 1714.26 1.05021e+07 48725.5 1418.42 1.05021e+07 1402.54 1.05021e+07 831314 1.18099e+06 8.14875e+06 3.1361e+07 48746.3 22985.9 1.25993e+07
71023 20230531.2 0.818541 45617.4 48613.1 12592.3 12826.3 19998.2 1.88908e+07 4434.62 20406.2 1.05021e+07 6671.18 1.67936e+07 3590.06 6587.9 6.30784e+06 3270.73 1718.4 1.05021e+07 49664.6 1418.54 1.05021e+07 1402.26 1.05021e+07 837564 1.17128e+06 8.14729e+06 3.06023e+07 56406.1 22866.2 1.25993e+07
71030 20230531.4 0.838945 45488 47653.5 12502.6 12913.2 20242.8 1.88908e+07 4439.96 20568 1.25993e+07 6733.13 1.67936e+07 3567.53 6599.7 6.30784e+06 3253.23 1715.08 1.05021e+07 51037.4 1417.79 1.05021e+07 1405.46 1.05021e+07 835195 1.17716e+06 8.15264e+06 3.10233e+07 56582.8 23039.4 1.25993e+07
71037 20230531.5 0.814365 45704.2 48420.5 12516.1 12968.8 20339.4 1.88908e+07 4450.7 20543.3 1.25993e+07 6723.42 1.67936e+07 3551.59 6516.7 6.30784e+06 3279.19 1710.31 1.05021e+07 44922.4 1429.89 1.05021e+07 1401.84 1.05021e+07 840604 1.17575e+06 8.15076e+06 3.09043e+07 57517.5 22979.7 1.25993e+07
71125 20230531.27 0.81868 45570.5 48645.2 12465.9 13128.9 20199 1.88908e+07 4447.5 20352.1 1.25993e+07 6315.41 1.67936e+07 3482.56 6462.9 6.30784e+06 3250.28 1707.91 1.05021e+07 47716.3 1412.87 1.05021e+07 1387.23 1.05021e+07 832399 1.17845e+06 8.15468e+06 3.07415e+07 56160.5 22830 1.25993e+07
71140 20230601.2 0.809393 45687.6 48718.2 12450.3 12785.8 20314.7 1.88908e+07 4412.75 20572 1.25993e+07 6764.59 1.67936e+07 3562.44 6549.8 6.30784e+06 3361.72 1714.95 1.05021e+07 48747.6 1424.05 1.05021e+07 1406.67 1.05021e+07 839936 1.18409e+06 8.15491e+06 3.12648e+07 57075.4 22948.3 1.25993e+07
71165 20230601.11 0.824953 45549.3 48092.6 12389.2 13144.1 20107.2 1.88908e+07 4468.28 20512.8 1.05021e+07 6277.63 1.67936e+07 3413.21 6464.3 6.30784e+06 3276.2 1709.15 1.05021e+07 43889.3 1418.32 1.05021e+07 1401.13 1.05021e+07 843175 1.17999e+06 8.15559e+06 3.06293e+07 54929.6 22842.1 1.25993e+07
71178 20230601.14 0.78119 45910.4 48142 12399.3 13330.1 20337.4 1.88908e+07 4456.72 20489.8 1.25993e+07 6355.63 1.67936e+07 3471.95 6506.6 6.30784e+06 3224.21 1704.04 1.05021e+07 48027.5 1418.68 1.05021e+07 1401.86 1.05021e+07 828307 1.18457e+06 8.15319e+06 3.10242e+07 57267.2 22931.1 1.25993e+07
71201 20230601.19 0.808667 45895.1 48715.9 12293.2 13167.6 20363.9 1.88908e+07 4467.32 20478.9 1.25993e+07 6406.29 1.67936e+07 3475.76 6502.8 6.30784e+06 3280.43 1713.76 1.05021e+07 47964.2 1418.43 1.05021e+07 1421.34 1.05021e+07 829567 1.18099e+06 8.15316e+06 3.09165e+07 56759.9 22950.9 1.25993e+07
71211 20230601.22 0.796293 45714.4 48392.7 12630 13086.3 20216.6 1.88908e+07 4443.59 20483.1 1.25993e+07 6736.26 1.67936e+07 3484.48 6563.4 6.30784e+06 3265.72 1733.73 1.05021e+07 50265.7 1422.61 1.05021e+07 1422.76 1.05021e+07 833223 1.18487e+06 8.15442e+06 3.13073e+07 56736.6 22930.1 1.25993e+07
71221 20230602.2 0.829951 45570.2 47549.7 12380.8 12828.6 20418.8 1.88908e+07 4455.82 20526.8 1.25993e+07 6737.26 1.88908e+07 3496.43 6607.2 6.30784e+06 3234.86 1711.7 1.05021e+07 44041.5 1418.74 1.05021e+07 1420.2 1.05021e+07 836887 1.17647e+06 8.15225e+06 3.05726e+07 55697.9 22968.3 1.25993e+07
71242 20230602.9 0.792046 45520.7 47575.6 12483.8 12895.7 20310.9 1.88908e+07 4431.07 20439.1 1.25993e+07 6717.2 1.67936e+07 3503.38 6600.6 6.30784e+06 3289.73 1732.85 1.05021e+07 47652 1417.24 1.05021e+07 1418.2 1.05021e+07 838767 1.1824e+06 8.1542e+06 3.08109e+07 55973.5 23053.5 1.25993e+07
71259 20230602.12 0.789445 47627.5 48617.1 12501.4 12914.4 20414.9 1.88908e+07 4420.18 20684.4 1.25993e+07 6766.88 1.67936e+07 3491.81 6549 6.30784e+06 3262.77 1732.46 1.05021e+07 48087.6 1419.76 1.05021e+07 1404.96 1.05021e+07 834594 1.1753e+06 8.14657e+06 3.08053e+07 56217.4 23054 1.25993e+07

python_3_10

build_id build_number Commit latency factor ls_sgx_cft^ ls_sgx_cft_mem pi_ls_sgx_cft^ pi_ls_sgx_cft_mem ls_virtual_cft^ pi_basic_sgx_cft^ pi_basic_sgx_cft_mem pi_ls_virtual_cft^ pi_basic_virtual_cft^ ls_jwt_sgx_cft^ ls_jwt_sgx_cft_mem pi_ls_jwt_sgx_cft^ pi_ls_jwt_sgx_cft_mem ls_jwt_virtual_cft^ pi_ls_jwt_virtual_cft^ ls_js_virtual_cft^ ls_js_sgx_cft^ ls_js_sgx_cft_mem ls_full_js_virtual_cft^ ls_js_jwt_virtual_cft^ ls_full_js_sgx_cft^ ls_full_js_sgx_cft_mem hist_sgx_cft^ ls_js_jwt_sgx_cft^ ls_js_jwt_sgx_cft_mem RB put (/s)^ CHAMP put (/s)^ RB get (/s)^ CHAMP get (/s)^
71190 20230601.17 0.78311 20320.5 1.88908e+07 20532.9 1.25993e+07 45816.2 22910.3 1.05021e+07 48752.9 56144.1 6695.75 1.67936e+07 6574.1 6.30784e+06 12412.8 13133.2 4435.55 1731.59 1.05021e+07 3469 3244.96 1423.85 1.05021e+07 46550.9 1420.59 1.05021e+07 832951 1.17681e+06 8.15459e+06 3.09496e+07
71232 20230602.7 0.799672 20373.1 1.88908e+07 20613.9 1.25993e+07 47797.4 22826.2 1.05021e+07 49158 48898.1 6710.72 1.67936e+07 6548.6 6.30784e+06 12497 13175.6 4458 1709.32 1.05021e+07 3552.83 3358.58 1412.58 1.05021e+07 50971.7 1405.19 1.05021e+07 835801 1.17217e+06 8.1379e+06 3.07817e+07
71248 20230602.10 0.787298 20081.5 1.88908e+07 20593.5 1.25993e+07 45741 22927 1.05021e+07 48404.8 55810.1 6656.8 1.88908e+07 6532.9 6.30784e+06 12432 13154.4 4437.57 1715.87 1.05021e+07 3495.85 3279.72 1411.58 1.05021e+07 45620.6 1407.38 1.05021e+07 834826 1.17717e+06 8.15199e+06 3.07065e+07
71272 20230602.14 0.798748 20337.5 1.88908e+07 20555.4 1.25993e+07 45719.7 22765.4 1.25993e+07 48565.1 55500.4 6715.27 1.67936e+07 6552.1 6.30784e+06 12462.3 12803.4 4438.96 1737.42 1.05021e+07 3469.27 3271.7 1419.22 1.05021e+07 46296.9 1422.68 1.05021e+07 839683 1.17599e+06 8.15413e+06 3.0639e+07
71279 20230602.15 0.786687 20436.8 1.88908e+07 20658.5 1.25993e+07 45720.6 22898.7 1.25993e+07 49143.9 55713.8 6738.16 1.67936e+07 6552.1 6.30784e+06 12286.2 13002.7 4423.23 1734.42 1.05021e+07 3557.69 3259.89 1432.06 1.05021e+07 50343 1423.41 1.05021e+07 819628 1.17872e+06 8.16138e+06 3.08327e+07

images

@jumaffre jumaffre marked this pull request as ready for review May 31, 2023 11:21
@jumaffre jumaffre requested a review from a team May 31, 2023 11:21
@jumaffre
Copy link
Contributor Author

The SNP pipeline is failing as expected as it is building the image off main. It will be fixed once the PR is merged.

@@ -185,7 +185,7 @@ jobs:
- job: cleanup_aci
displayName: "Cleanup ACI"
pool:
vmImage: ubuntu-20.04
vmImage: ubuntu-22.04
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is also updating some of the dependencies to 22.04, which feels like a bigger change than Python 3.10. Can we describe that in the PR title, and changelog if necessary?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should have only been done on CI jobs that used Python (3.8 but now 3.10) but do not make use of any of our containers. In other words, it's not a user-facing change, but simply for us to use 22.04 for a few CI agents. I'll reflect that in the PR description.

@jumaffre
Copy link
Contributor Author

jumaffre commented Jun 2, 2023

This is currently blocked by the GitHub Pages job as Sphinx doesn't work with Python 3.10 because of python-babel/babel#990.

I see two options:

  1. Wait for the fix (Fix for #990 python-babel/babel#1006) to be merged upstream (may take a while).
  2. Temporarily relax the required version condition in setup.py so that ccf can be installed with python3.8 (as there are no compatibility issues between 3.8 and 3.10).

@jumaffre jumaffre marked this pull request as draft June 2, 2023 14:00
@@ -9,5 +9,5 @@ jobs:
clean: true
fetchDepth: 1

- script: python3.8 scripts/extract-release-notes.py --target-git-version
- script: python3.10 scripts/extract-release-notes.py --target-git-version
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we ought to switch to calling python3 everywhere, and communicate the version requirement through https://packaging.python.org/en/latest/guides/distributing-packages-using-setuptools/#python-requires

Wherever we can, using a shebang line or setuptools entry points would be nicer too, to avoid having to specify which python, now that 2.x is thoroughly obsolete.

@@ -1,3 +1,10 @@
- name: Add Python repository
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am confused by this, doesn't 22.04 ship with 3.10? If so, why do we need the ppa?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR doesn't update the base container image to 22.04, and only updates Python to 3.10. Updating our container images to 22.04 will be a bigger change (in a separate PR) which cannot be completed just yet because of the lack of Open Enclave packages on 22.04.

@jumaffre
Copy link
Contributor Author

As per the update on #5291, we will not do this as there are no tangible benefits and we will remain on Ubuntu 20.04 for now.

@jumaffre jumaffre closed this Jun 16, 2023
@jumaffre jumaffre deleted the python_3_10 branch August 7, 2023 08:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants