From 40c9d24c9879c82764d4163bd5779b3084fa8553 Mon Sep 17 00:00:00 2001 From: Felix Schaumann Date: Mon, 13 Nov 2023 18:14:07 +0100 Subject: [PATCH 1/3] Add hparam notebook --- docs/images/hparam_tuning.png | Bin 0 -> 48040 bytes .../extras/01_hyperparameter_tuning.ipynb | 433 ++++++++++++++++++ 2 files changed, 433 insertions(+) create mode 100644 docs/images/hparam_tuning.png create mode 100644 docs/notebooks/extras/01_hyperparameter_tuning.ipynb diff --git a/docs/images/hparam_tuning.png b/docs/images/hparam_tuning.png new file mode 100644 index 0000000000000000000000000000000000000000..d0fab0447b64f584a4dec77e0e58fea9a130d925 GIT binary patch literal 48040 zcma&ObzD?Y|1L^Mi$fzh3L+g6(gR2dNOz-1gGkp5tsn>^NK1!wgGe}lh(mWH(p>{X z41G7gzjMw#_ndp~{fE!mdp2vYwZ6U9^Xv~=8p@=^^u$M0jXaMucs8xOs0^?0_`QyDK5=*PNbU`z0-IIQ{S%B)4ow)G+Vg*Hj89MkKj`>{RMyocxO;dMBM=r& zD$~bms;V6P{DWhz_Vz<2Iw2>~(b4KxmzU7l>w2fjx+FbqZEe}BpFf}Jwq|EPcmVlI zO(Y}P5G*mshr$6v`=V_&y1|f>xjy8RvCpkgGhF|lBG!{g;9r%QhCiEwz5ezQNcKSm zlr#-s>0o|T`&)?&J>9$@QG-Aim-31Dp6>4IU3PYMU32pc;A&vtUQt$JA{&fVphHtd z#o=34YU;Ezm>StP*wvLiHl}BeGC3X{8M%9PbmVz@iPCT_VpBo zX^-!%`VUl2REo+ZxzxcN19xx{m5FKdv2bg+NA~)1I)|m^^`TqLCK04!Ztc8?5~Zzt z^mM40s*>7s%QF^o5il|z1aTG=KdV!7~L zJeO@?2CA34WV5~z&tgHoy~XIgH$=#iD05hy^woR=eA#JovYC^+{tkF{9}hyK#?5Ob zCBsU5$fGo>Pwt_sVk12#EfH}~S-*QwRIruCcjo6SWAZS#4F|7Y9hnZEk5TEWiM%v* z!uW_~#A_myepLORsmmEJ2JE=bLpFSHubN4uuiS66kd-IyyGNLrSiRhr)#7q-1+E{PD{sh zS^R>hj;do#aB72H{jbN5wT(RQnv=~^vR_w}mAHp#%6(Zx3dYl7`VLV$=Uu~3(6H5E zycv)ubCz+Yc{VIh3W7(t(*E*BIYhKY!%$SKwCMaVN-I!k^H_z4R7vbt$}09!ra zwA4hEuWY8CDf#)LV5>kw%ocI`w(m(|@IhH`7$dY9L7tHFZ6iQQLF{QH3iYF@$%Kn) zxcyz}OFiumA(2Vk&Z23H8>hOnaHmd6C5h?JBlCr+G?)gisN09EtzuwZAY9q3+%8XtidYlJ(3Pmif8{d6Zh3x(7 zg;!VBb-&xk8%AwV%~mcH#+)1R;9E*2vmcQ$aiU&Q(PDl07*lY>a|QX5^tB}vr-`1L zmq5Jgit)Jmx%$Vbm!7nZ@23K(j{%oq@&q$eEtRuQcYu9A z@}#Gy$Alg8%iH=wf2^=A;^!+liv^+Hj@oSk3HYEYvjqoig{)mPO6s3|YC$J|Wk+2L~$V;}0=)3mxL^jcU)If6}Jp zdvK|`tkmYCUiPhKfnI;@L%eIHUZl;~hGj<+4n`wGQnR|c+WkV`!!QJNvpmHLSKI#` zygW36#_V9kApemkV#bR(i~J}%X|Hr3h_(K)B?-irYefErd2dIQ7{Nte#<*oDAJAI* z6*jk%C~c`fn#gu8$|0Ro9Bl?sFrNw(X)0}txYD}{R0_58)D52rg*KF9f2sCsKW3TWhvMl5B2tYhx`83vBrM1y$u81KNPsLT7%GTF%X z$yl2^kR!A?fu&w;yN@H3RAgO_&nESxEkaw;xs`s}231{m+tFr-<|^uCa0hsygQL>v zT%?D{=@VXFy{clYQv$3$#GRGl^)G@%n<-rIH-_0t0tgMsAx@StbLOq}4Mw?o#-*KY zd}`P>?FRIQL6HeCs71iC24{5-6Ojb655}?KUw^utuw&9AwjGvB^=p3Zdu9YUFLghu zugmeEcDd0Ku11^9yYOK?4Y;2#DY}cLH!a9l$0KRG){l+5M@j1Txj`VWjtbYL2)jwq z-ccd6I9yZN*3Z}bDk&ZraaIrLjE@*26&TFV|0XERb=2wh9gf71wzo*-YwY=Mk97_S z2@?ZmUJOcR{ebSGi1MFuh+XpnT282{6>#e2J{whu2mZ=>Lpp@=(WS+uahu(;;e6qk z&M_4i*c7t#Z9;b`)4#+Cx?<_kUU-Oc_dt8HLU!2K56%YiocIy9wyx~IK%xhpkLkLH z8C!Tyjxpl7l|cakM5JWapuO-UbsW*T;DjW8j5*9@*tje+F&2p8X2Vc6{IJkyZuuVT zh+<}ag{sEk2Ta{SpnY9RPm@8?uzTdSo{uN5i*Kh)UejGjlK!m$AGOgIK@ek~kL&WB z>UmTMD#G@6>P%*?7(L?Q;bdzLum8 z5W%Dw7&#fW&#ejC5dfS$;5*Zib_-ut7u!X9^1}L{fQ5hJzYXWg_vqUS2rM+gXs7n| zQz-<|`Q)e6p4v1Jq*e;TC~4^j6?+=ew~UT(O^6_vH_XgTg-a}Qg9a<$%Y%MUe8>Ik zn3cuNO(iHVMqn<>4~kSC*YZ0NDe3xA9hz0Si^2T96$IJ%h9Y!o5bz-pE>9>f+7EvG zGfLd7)^iNXsD#P9)PBEVTvYD_W~d&3sQBvglv!=n>1|kN=zZv}k^cf3S+o}mSNt{E ziN{wN@zI3SwUPN%)8;XkglPZ&-PmxT<7^6rzsEs5Odyvz`Tr-6j9&zjhW$tLnxKT(`8@u>rFKlzFfZq{N)XKko?4*_F!6 zsCyKh4A@n(A2cT+Q3{f73apL6Z8);^^Y(`BWTm<$*x5}QA}9L*?FWRAf{ZYu1)c%4 zxDSvbv`a)#l}G>`C;u9`E(I}UwsgV+|1ItHgZ87f*%ugA7mbhq*`%#sAgcYobg`85 zZ8-I$as<=D$rBz1otvOuHt(z*ywfo_#GrgQ;H@(>tkqz~c@k*Z39B@!{v8Bj8OFdv z1H=~O2rgK5c;z@guABhc$h@#Tsa6A|5P+}2hj{8J|Na#x#H%jp6s_m){nR`rDyhvD zb(>=R`~B!xTvvxf>ghNAHWmp;2t_=jdovxB$MQc;2n2)b>+A1Dg5VB6gXNOipBt-b z%e^Ew|4#8(`Dm!Mu&t0;wG9U4(ew+s4NciFvlgYmY&2%*g&_Jr!CV`8t1;bx=6%s- z1}m|~Z3Qm|QXjkw(A62;xJ)b+G|y)%F$&)MIWC~;F2Tvksq*!Ol_xzG0jA1wN`@-k zw9QOYjOO3UBMx_V&ZlIW5!!&CtBkEobxpFPwZC1pd?;&G+Un-?asS~_AGFjDn%NH- zj2z@#h)YS~$a^eOLt-THOB~mX;^}ddqn7_Ju?n{CN3iuI5*!DJ3KPwJ`ufe5Wfh`C5&~oB_(Jyhud@GNWY0{Z});L zbNiQfxH3}9(WL1Qq&{QHcJNwyCwWF0dg66S{V3-3B(e#2^Go@BpQve|=|2Oq98rQ5 z-RDZyWD_ba0+Cbi{f>-`@64Eem@A*~h}^|5Nn!YpFxNe03iB9^d{cjmeCi8pPLa;j z)B0Y6?O*5Eube^ONM<8;_09tPuLAtF=}TdNjSK+s=s1C9d@>@CTtU}f3G1o-UbadE z4w*lVFxy8}TP2CkM_-4t0wRm{zkhbGg#}qQJJ)NJfoF5hcdB5H!iH>o0Z%=oznt*U z0BI8tHzVW6HY!5FY4G6as7opANC?|t?n1s{O|2m3M4VjECY{_D&3vuEiv4}3dlwURWM@z5?4(T4?ywC??ZXyF~#W+VLxeQKWZ#;2Tnpq5}<=acaN^PBzd2^A9HvM zM~aKuFwHYITvNr@QqBb5lof#vMrvK*Imi*9-WtCH&17vU>E0%i(Zuq9rbP{%&~RF! zrPk4S^Uy0TUBOhtPTROUrXTLb(xiJ+nl6neO}*)NvU{70-D+}X1{0XJG!-t{Y!0Sv z&g_Hg70I3#jiF+=9^U#=Z&;iBHaACh5%yg?5T4BVj$#vm4uxJ5$$%(vEO!)T80cOE z!;6{t^qu_gz63wSx>aTe)^u#nj{Tm1);b02(0+WD<(Al*BCgs)Qom2O6$3D)BJkt< z6EIRs^BU#07Xx+9qMqmZ=XcB4g}}(!3b=>e8jvajS@Dgc@qvDVBX-3z;yjQ1J7fZntZ=;BV3=d035xd7D&5o0w zyCC$Zj=!G?{w~7o#c{p*x?xm2mAcy4m!u%A76u zmY9PS1k|N~+hr_2JD1=m11Blzv7WKHa)#aNa7d(T^<&9i$0qd|7u(w$-kMI8go2gu z@AF3Q@{kh8t<7=cGtUP##7+n212I&)-<}*9830ci$J20_3>ooanR~h5{*WjPzNe%w z>SU~ht}-%M#RCGZR5Qfg6K9iEnFx}LEqz=3EKVQeW*`fb3)eY=&ixgy(&Q!06GKCx zJ$(at1GUHP5q{NoccK0WgB~E*014gdwDG~MmY4|{K!N@wAzMlwurU?PZT+E-V91k ztdjYKm2hk2?KorU)(@snwUv*(OYYG5|0v|wEw`Vb$? zy-`x!{CmOro!(uD!B8jAF!`SwFld=pFA20Or2TFEa95Q1KdZP+a{C(MBPMmO7`i(~ zKby}B!8N&Gp!=3bLqU)=(SZGOAUH(WAsgcDCP#pRR|jkc8vXFLn@VYACVXjEf(cv_x1s*|lU&QMPKf?Oai7BGeXH>sNur<~78`KiAeo+1F zG4mg*|K|k%*D3q`XUc#uZ6WS&|7CC9Fm6A8Z$Q=(fbM+!SAlC~!+_+(;C5R7ToPjd zD|r1^6c7{bkibUJ8jxa4A8WG%!K*_E4QLm^zw}~D3StRN4_F;!A(T~M`dA%}5fti5 zjekQ330gUzTb;hGyKNCYwL<9Q*K5mZ)k{D}?f;f5PXp5floJixQV7`#vBsSSXQ0cO zZ@05_B!YYcbb-epyEBZTOe?w_N!$b!JpZQ#AXHmFe_KEaAwu)2w^^5PjWv+b0&g(w z9<5#4E-;nt3K2rb9HJgb0{dSp1Q7UOG8`s)d#4YYPWexzn;{kELh~HAo>Ic0^Isuy zn*&^aT0MA}{JRfXRh#**PX1>Zz&erLE|x7q<`DBq4al0sbr8_oRRlyU^+E`+t)+{e zWnDz(0x*PsuigQ6s{R!w=}1~xF#BXIcw@Bzf1YsWlMK)k)L^Y)*?NAqxza~h?>~~h z)Vi1oh|MyKzAWr^| z{mvv>3{D0zf-%bqqr@s{i!jd2Fb9iU+)DO8L$3kKMh@^OkjxtYD@SijQ89K~$iR<( z)*K6OdYJ&!KY1GX9sj+O6chNbRbQ8cgV6s1pOT+%itVl8uS+4;fC6XoCydoz?wmnWIX2&cB^-mtRSSl4ft|Ds4Iaou+~oZ1l>u^cV15&J(Ho1q}42D9&XawAJg zXz%}c+7KLzY-}sMx<|`*&86gs1KE9H_8n7;<)0`*S|{2PbuGaY@q3qx)s7Dzr=_*` zU9?QJ0aM*$cYA?!c#idp4#G$~xqD>q!=4D29cz6Gdp&8mJ?U=%UD?2N80N{$gyy*C zGu-JxY9Zp_%1!bfuzbOdGX>IwPv&YFcI%^+0d~EIy_v>Gr!O1pE?+lV zWnnCSaBGtE4h>W$xppF-(w|NFu!50&mWT=TksGPKbkNy%?|!J+vIz*!>n9kpRWuCqS}QigWJPih*4VI?IcX~7`&bCWP`?2*N{T3%lDy&ZT% zMhnY7(Yto3=Ng5&lSn(~@2)bs=WnY1X2U6$M`;NRR>hwpksq}EV%1xI$H!&9*;r;w zIqS04^^5n19QNrkTUT8=O>S|v*2QLKBx%a*XR5dEepYYY-sWnhz`BgVv>fZa`+jcb zXKVESPV`r+R3q~KFHB&coU+qCqkCJ{9gUEad;9gRaKYcR#5x9rb~ ziRr%=hyUK;C}7p5)-%^vKlZBEv(T0U_GR@93?8c$b9Az?>2c)#9MPGybE#fpExDUw z`DbI>cjD?*goV72}?pNbx@tmb@+xcpNjG#4g|KNdDJ-@m0gooR)&j{$f+LBS}T$ z4ReCLinxA_ctJ+8+FLaTeer_C*9|lA#%U_-uY@*S!-O`H)4uRlh}`jKs-u0_6Li$` zHFgO*r$AX*2Sxb7TW9T?F9o{5;E+%@_iPbxnJK2BrE3}e`szmUIimY~-y&v-}8qHj%> z&C)QQ6q;tXRU?3`a_VHqT>Ct~$F?SbF}l|Ee<=++vwm`g2)cJwlFVJO8bWkd$Ne#9 zGJcGv?X>&QHh7;lSLLwjyQJZyyKeHT-3K*d-VaQMB^yT0q}2NNWmLkb>Fd2wt0X91 z$qcodmspTGF^S+P$6#Daf%5}o$h4=rNLVFbN|7L8m36FLSqQIV; zMWZ+E_2|5cN{h2=7w<{D70*cbJJvHD>^G$;l`iEC2H~N4O9u)f)1_uSI8$TI+_j92 z@kGW9v?C4=-J{eZ3+s-Acdu)W%Djud;Xw4=OVJ#h{c;B?M0&>B8i|RCW7snLk+E=g zJFYNz=c{^Yl#TwHwpmozt0l@JGgL)xZZ0|0?Jk5SdR501&$12+>>FY5CSqhnCn_r1 z8i-;%#ZO5fkUdNnNR=vv_0@g=-Ne5_EQ^J{aQfh@rgNZ&M7J$m0B1dHROm3`?-o!Cp5Hx zi?SddNfv>-q)o>I0UHPca8AHpP~_hC^}zGXQrDFWg|aPFT%Is=#lFdRvu1+0WWr_Y zc~JA})3m47NfVA0okenwN9W{HILu8VuE?<{;_mW z;-_GKPQyP1q6tI&xb7>sWP!7j$9XR=uTa0BGoIC|N};@RerIQ6XEA)9`?#RdvS~Dj zxVWNTmC@F8Gx!SqP10d(%B`ugWOQ_=5xueB@K^53Q`d15x$*OuFB{-jl3xeY%jQT(37_h|^pXtWI! zGbdZTRkhj;IdU5}33@iOIy_cckp~}Tc9&nG^6eCYUHN(PNixpO1Eft;jgG)p`Ub_GX zz96CDb%36Ad$~+E#k{}EAXn{H@tDD)Sk(6NP9v#mLw?Fva2799=w zR=TvKEZ9L}Fx0Y<)YqDtiiTM(EP}L9zjU(820B|_Xc+YJjYX!}elAPAdW^xMjU=^JULqK(XVqpBX;|X$h!`EKU7W0{)MLlCWW|6 z^N;-U_kGNlRj2z2T5bjvCh~LsCrRtfbA)2}st%i*HDB(m6U{bzPPItcZ-co8HmG{aMM+0EQOfK!+COQ3xt7| ztZmN`o#(|I)8?5*<YWeID}@q1^##AknhLg z`4`Vw^HZX><4;#`QXBy=?2vBGYaW;4yM0OCq04-uS|~`MP16gj!1PUPv=W?&pIa#D zViB1YwFXt*I{Zs;^s~1zYW1%gW#Nm&*?OP(z$W>f*(;EYl8+Dc6LY}1cq2Sh)m?0p zygDN?0UWSp?7uyCY9or>b7A}@I zjB1R!j2)7{p8P)0tK2@qgP8T2%Up%-QOtg@J^-dP+`9K&`6;~mI3-xgtO3j3^0Ys{pf_tlUiSoQdmmN}6S(p_Up|^uVcKl{Dy0yV zaxO3@wM%Y%EpSYJ{v+k2S><%J|CN98@%%9Px&z&nL-J(0yEL3Xje>$=gK|y0dqOu9}zdiYg8 zL5v8?dj2|WWnD%0Yd0}mm5Gj;E$_2(7F0=znk-kGCnz!!3&1zn;3+DEJ2*|ce43XF z4~_a*gs<+|qDnSAE92T6{rE*X4@=L!4I7h~Z&6Wr1)=T1h>bfPCA%i|g|_k;%OP|< z48GDtF3ltX4c{Y|4LBPYecfu~XhJ_eF8V`UJiYrU9BZkTAMqlY&oa(O?~ha!FWB=+ zPYHGbVobo2dla#{Bx$XoZZ$(#o-7X{63|Q%?h1iOTFiiD z3&&!#*TmRFm)C<(nBfjwS0VlhaEfWMLlO&O(PEsG?Ci(seMb|u!HC3N$6Y6lqkm%$ zha7yAVi#k^K;Arve5Ce^l>EW3{*|c%O3&+`cz$TPx(KI&s;cJ--O?*h*$VURzkRpq zv0UIT;`}Oouyk%blp4qy>{E;N2do&rUWygoN{w}i+_7^|oU(0g8_Ynmn}`h$^I^_N zq0N;Bz3XRJ^ZRKC-9;sm+}4{5m&eD(md`}~wh$IIk#zl;J9sEDLx?DS`SR$O(BLSm zrWEf@&|X2ZO?aK{oWHjCUcp|j4wIOtk%6PI7dhKUMrnfqN!NB3A_T8#LqNq(-al95 zaSwiW_w>+c|B+;ofSOl0O&@}7?t~IovOJR|jr%iR`f|@EUCd9_;Ae<@M7QcmF=hZ4 zE+zNjgUa51b|XnPDC`TC z6%hv=(E=`q3=MhgHrv2=2R$uz?E8!jKSQ6W@AjVaJ9E~Snqc~CySu!U*{p|oN&JPg`HB7 zp5((plX=O1P8$z>87YvmBC2D?E+H+{aL6YpK}F6jeMB=*c; z7R0g}*OL5A#x+CzM`9L;aEvR%hses%D6UYZee#(5B1%6=wCTeiGqEJw$XsreXyuAW zT!W9MAEwE%JqwS_P4mXZ2Yp84Ji?k^=ORnO5V^X^$CTw8JBtkk z#dnhzWNx@^(EASw^oEi4cOV-_sX`TB!o_H(_6;opK7V`TF$bl>V2R>wxMOcjcB-!% zS6dX0aKKpU&cPG25e}Bo#j0MZ6fTJIu!d^A+H|@(!%d`XxTI^1m zl>#iI*W0zljI<3C8lvzYS-(#K*KiWx9X477fV}ELO zSSh*pHO~5X&%l|U+m4{Z>yWQnav`OZ3>nK<^Aj$Te|m?Buk4DfBA*WFDXc2ju@@2% zW(N`WKOt1N2tJNd{^hf>@nrhusjRShb&S0QHk63y!`{A?DK?*Qy#Q87pooI!Y6J_`#6-YnKaG_;_TB1PS9}vXOfM=y6B^+5uXi1%KZY(hI>_sdlY1- zcLig^?7dMN?(U-l0SeCd$A8#}{dIMaDj^Z@Hr9G>_c?7~zpK2^)qrL+U%Da+e77LF z=Q_6t>lA{DbS_y6RhFT{3j4~!MS6b3GAwh7TCS-iRM*ryDcZPC&&Q&K$Mo?iCpVAC z%wecF<1rWYdFNissXushyy`7>SfLG?>dH@5CQ9|=HUC|*m37)K)f4kmi<-0V6;GZi z>KcNe_y44b_+&=@+HtvfRLzd-P%0?9b8L+AUG5D(6^h@RJf)*3^4(G)>pBbtVXJXrFR& ztWzyn*z}K}4FNRrgwOU}?_)lEfx5RsT*UsAv4e4({5D?H!Wx$`*zlXw>u|=MHqYJj zvV`E^eN|Q7tFH~*uNYby6Ld&~ndbeD$9#Tp!$EOO!G-D;)_KNgx|gYT*ht z{qz~4(ntFdx81paeO)>0a4-U15k9*5ac~kKSTgl}(?95bE_r^iRRU$zo!3<{Kyn+$ zPO;!?DT|XxL2X7veV}=R7CkL`l!lBfsWP|2yBpPcrp$)_2UyQ{_#A)t&p(7;BX9{e zD{orH=X{FUgF)o|S4&1#WLU7o(?2&=Qja%q-w9|1$Gvt2=eE&W?C~_*WSq*xi{2+J ze!3qxq_3g;D|&wsU%>MhhfjEvEzdU)uB^!-t>}Kx(TAgI@P_rTdHYanSwc~Eg&?uy z`!Ia0)#vpofFSW2KeD)`KOx?cxZ&?w#rnAaVRh|;J^0FVidkVs%tBMB^?g(t!$jP4U8;^F=CQI&GzgMD??x9}u$6aG;%voIzm zhhK4=cd9X3iRgKeFo;dh;)o#EA-W4B z?k4tu>>FwS>oz*=VhS$d5xINJ5=)uioKBhP)LH1mtM2Ac-enXtGq95Vk^N-<+%osb$qtDLKn#l}@W_(5l4Z^|Jl66n{Ci2_ z-uLeo7rVvCE~iu)HJT`8*}vZ+nvGeGMNc%`$flY`lD{Rqz_}lEFy3%DwRry?gLvA2 z<@c%AbceoW?FaWC50r2FX4932dqh8$gff~Q;RgJj8e*zZP#BpCEW?HUn)!oF;*ngC z*E_lXYN6hCYN^Ydz{z8>;P=e6B{UOjy{312lYbOkI4`R7af2P|Fm6X-HFIptiH(=# z*wYl8sIEyAyh~J>B&%EEN=7aH^P-`Ne6wOrFzG-IE*Ga20{5~$+*x+&)(KaP+S&C^yUz%gK+-P|%dUwD}KwHt%|E944I%aN>cpDG1eBs5^G03~67^LsM1G$}z9%#kASC zh^RfLWcJTVLFGdHVU@>(xV-j(npxs%qQ=bygn1<}g1ZUolM^SBt5zkz2=b_~ztb87 zvg8n0%{8Dz&UEYJn*_02h0lGRg3kN%FJ9*pY~(oXkoXfreFlS4D_$!;Wq<|_E~D`v z`U_XBZ&GVENnT zz${mW=dAF)xALwZKpVwh_wv-{bAIKntciNz&LQpehK}?XxGTOZfnIb8T^Ta~+gikk zG!h6^u=B4thofedNy=pDMMFJP)lHjeocM};9DfXHCHavS)-Xu)IO!_*Ru67BvuwB| z7jV;0*fX+a)B`@gISEc?zx-4YI)7jil6hx7|H7dd^Ts2n;dZ=6Uu$q7vX6i8L|d3% z=zlKKY~LR^%h(9UT0Z1v;FSXv3xN0n!(I``r;YdN&#w5OUj8}(UEvTQp2%rfokRcy zhN>IUz1@{yLK$vuKECQPWBC43LorVh|1>t3n5EXGnqZ?@nU0x-4Cy&h7x0Q@sgQP55XwAxf=T!MAA(~^jO{9o*r*D zQe4?EWsO{FC$sMM5FJGuwsMo!MErYMvbYMVVlYTSzC}=QNFj=_gxcxpXf`>eLtE8B zo%@_^N7it)kv9NgdAwgGue%b4Unl!{aH$p*Su+1(=eeab!=q%o_2NG4?&VUa2f%i$ zd}SYB!ULhuRDD!&Vcj2y&JFDYUhU7eJlWJv<}wHZj8bfthLh-)Ol*(kOqC6-*S2JO z#d~EHlm^YLKwuAJN&9Cjk(oLd&@r9VdlnY!ji0LmPv!+?5KgTpPCQ8oGHnsIoM|m{ zKJWz8s8S%1uta7R|7fmeXQN`%)mV)wtny6?UOC{@pES?Yx8o{a#{4d<*?1&;fs^I0 zO5%Qv@4}TM-0vC~mJ^-avDw$C=6N}0eExD3U*aw1j=q6`wy_`H?sQOiyfjNmDfzJH zLYK-+{l364SVH6H-o}T~L$?=styyxt24#hmG8vzH#eI+R&0+!c2(!yyL~;@l2wk*6 zvMtV$L`RwYeAWk?4K{Smk2d@{ht9a%lrl%qKfL*UnPN!PzwH_TDp-pD+Df2kF*h{) z*#Vguw&!5#vzPbqm5G;EuliHlcN*7NU{ww3g=F=-*nw+0lupy=DSr00IP|pl zv3u4J>n`nd$89&gk~@#!nx&UD`1Gx%AFn0qHY=Y%LqN=k)9WRcrhLC1H35;JO|J8n zS*$b(Cvsk6L=s|CB%FrFlo!k>VE^m1POt5*W07k%`SRD1K0h^w<%zj5pDrV!WEYmG zlH%gTINAroUMDPR%^S++&v-m$D{$Ta{%OG>%6LL)H*z@sWu--Eoj1<}yJ#ofZ0oPj zonvNUlL1k;8L9K8bt|_(GZ0gXNDy7QiO*F#9vRCES+C6~FX5M4W|ckx*{RW9jM9Nj z@mSE)Kvn?hzUsqT70MqJ0>X;}|IKn6^D}{15QCW8DkV^8kzg$v^0|kLbc2fluGXL# z${;-@zka5^p`_rW13J#fwXp~b;}@dGlvb2+>CBkG;STCmX%V(8ZwsQU*$FIhBKU7h zaJqy?2!djvyQ_~TZtfxZbWHLCxzQTatd{@|hP(Cj2i90reFa21*#0P-jv{L7Lv!%` zv|LQ;7^w8=5UN^tRTej4v&$x>FTw6rOWQ{crZnpfM02=y!?zVYe8Qpl%+UpjGVO&?+tl%XhdH?< z1C_b1J~5xvjT0)p?-q#LC69KGp%aWVmrq5@@u;Dy%C8(ZR;TffTmb`L*Qdpx^pcvD z)_X*&`ejVyR>yhsz^>Qvi8UP5R8%S`Gm3uoHc)U>R7_z9QlmQrXIue~2V400@j>~l z=apRItoL4_+h;qqw+{u!PHq|zt)GdbqiihA^j@T=$L59FqD#EaD~s+J@kc0*Ai|I< zr!+TJN)T+tQv2$D>2r9(HXR#cGGM`;UeK{-FRumwa0G)CuWE2v>+r@q4~2+jBap_6 zF3$Zhoajml3Y6T&&BfwtwhWK&=Lu2VmBTyos|jU(GwwwNcwh^vQO#h`RUjEw#M)j$ zYV=BENjx)V!uk7k*IBvb$*?ar0);uWfyP1{n|6A9DDWXVvp>slg%us67T$|EVxoG7$4@g{J7uy zw6nnOb+_%-ncB;mVkbPPy57xhfn4%=8ZvKPU5NKQZ~)B{cI>mJt8ZJtk?k zJPBdDIMKp7hwV4m9S#&moDQ4rjiYNkU!@6RqaLq65+@8r$gkj(P*D>uFg&#ty!t*f z9`ya~G}lbk^s#p(uChiKp=&)Bk5c;ZO(a4=!L9KO&mHxMhh^pm=|*Kb#O}k6yxBMR zACm!r#yq#DH@~G27>w`D%0WHb!=;V~cSfWILYo46`6`dd6g(KdHCe=$hZngF*NoS> z@6wc%RD8_{0F6wO82-6lEsYc0RTSZULKvb)E$fwMltllUgOl^KRmb521sBOvD`VhD z?QRvL)5kF^#TUw>(1}a1{p4Tyy#zXEA*CSwd}1c?K$ErWm$JV(=wDZ7e7jamVx^pZl3O(_5Kn4?T7}T7Qk5;`|Y8WqtE|z;rMxBS?cznOU;EG znvO)is)m;EhQiSRBs)p7-UcPIxg1SH86U4(cQN=1?}e|89PWBZ!*gILonn-1>uJIK z!;AS{Vv|A(fCV@Py%zB@H+W>JjrwzhlNOjAdf1y1%&X!;3iXXgC#G<8H|l7jY;5dC z0dmUamYjiX{f6&5XlL7C!0YaYjVt9!p0me4G-`=C3LpJ~&)YVRC$%N(r>Z%08hbvG zGIPIo6|lOx%7>z+rq+J#oX&Gh3dOFutnB5iKf1UUr|_K--M*5+ZUqjIS1AjPa}2-W zSSm8VLn5{GoXb`iec9UyaA;_0b>F<D;iRnpxB5SMi?}&`(2u=H{|ZM%|e|JUrwuPzbyi5hn3qM;=3f0-Tzb zhQ>YdcwG*b$4W}Uv=tVKtC!RVt;*s-tHiW?Nj>8z?lBKPimP0^SlS~k6vbK4-Tquw zyN=Bz{r$7XJI%u7+r0H=RM=@2awr-)N;Qr4Ckp-xY5rM!s)~wuUTrcZjU<$d4-o0x zSSg8#Ds*&o37MH14h|0Ls*a9az#CKzl(?8$Qj|nb2T%NmrFsCV=~gDDCZuMnI!K^B z7IO{jUf>kAala+G#6gm+hBz%Sh((B!Cn`@$zSlf6;FjU+LBH=v3iWQlX?+0hBB19{-)~;2;#Xx&@w0T^Kc%^uw-q z(Q`Tm>HK^nP=8w}i;F~}%DBxE4lkLgQO3UeqC2Z%y9PFhQFjP;p6UoS@#`K&xgDhN zi&kJHI#&X>nv5#^quKW<&dyH7gfE5)8IL6$W~UiTIF`4gqOWrA7o^}VguGlxwaX{? zyXX?Zw6(RhLhwRC?tW@Q!V2@17MUJiMr7AJZB20$4{CVR9;qBl%FGa!F?`P94U>u{ zK@f-3cSdEUq}-E(Zx+Q zWgp-GU^4HlAL{%i6%KO-IEap&NbxcM_Cq%pVSsZO0++}?elw z4?14mU%=X-LJbuXd|MP+!&{aJ4kQlb*m&@(y)#sgs)$8Sgw_4!Te2tpwU`d19KhK8 zOq(>JT)n-V-+^o&@izb{YOLVIpHY>2N;6Bn%3b%s?1rvqDDBNfhJM#c0^dHU*VV=u zfY9#z^;oB{*r9G)hkShedj&65Ul=gpaj63y1E@2@$wM+oY$HDyz5Hj9(Lp*0W7O*Z z>Fb2kQ3^>v2UD#4^woVao7K-n+I#Q57zoBg_IQG z`8|!Pe+N5(odHN9Jqu%?KIqE%Vb~impl;%3UzTi&>R+21Nm}j5gaSvmpj~%|KYGx>pD(rNUuNnbf~lzy(#;M898#) zY>TcPL{M8+`u31Q*mjc}-yYbxg}G{AGktWIzJ77R#AkMCe6)fGS^4<@guiR^ycug{ zgu~Jm&lXZs+7G=vsc3`c_*W>zj!rwDqit0WB`1-QK& zysTd&q9Zj)AegQ7=QDn;zmQKI0f3g+i5P*&z7lNk$^dWjBQ9%_4uYA3;X}8ba*V$q+O6N{;6H3!^Jd6aB+x4Wjz!=!8=wH>m}Q6|w?JI$X-$A|q--t&CG^S1v zEse$;-LYI<+*CPJ+I+wT=hs32a7Ox6k~(+}G?Ac?y%aIrmDJNg@(%9c_nbZ64|=}I zvqiq-9}2f2m=RSi6hOOa^g#!uq{^S|mP$z{Di`xTsSLX0PeZP5JEzY>@uVt>~rn*gH9W#1GGo@K!q#2rsnGO^l?lZ zT*7q?z=YPO6|VOp&)u#8W-?`op+mxy>l~y->wJ>lB#=3ac?`H__z$}I|2J+jo*sh& zIC}u{b)M!BvtWIstoaY}`+f^|mjM930!ZU*+4GbMfIIni2cTC?9}Z{ogc3@UkigNIvzC^Xw{a$hw{w! ziK;fBB3#)9`GyeMP51zk761a@5^Mm`rLJxQTxDbCtXly3?M>?Wf3R3Wvws-gS}94) zcQM@6+5Y9)?b|Nv zui7p+lnx_Y+2Mmdhp;o-Ub|MauH@`G~SqO6C((l3johX5jgv=F$MSZR~_6ZJet2Cg95)SnrAGFRc6@Fv9L z)!Oo(2*5s1wQd;+w~*_=u0u?f&PvmLfayI?XeP!Ha^$8{j#07_LjNO>0~t90zyF6j zk7?Th0n3&Fa1Q}UH%_7cJmE*Kf4E-=Q~e#wN*zw)gZ?DfwJ*H$0CNwR=_NqOYi62v zKC?wk11S91Ti!qS(BU|c$x1;L{AGODs;=<;h%|$9{N6Um(+C zZf49ebK0uLiuZFwGbf#=cWNtnO|ZBKOKv3XN$=t<5A%tLh%|Bk(h|Ac+HySg)Gx!_ zR~1D+St*;STn~7YHKEKEY!2Q=4mD)L^$3E_GBdeQaqdHAVEJRzlcUPxvU9FMoo(ZT zIE~z6270p#$g(WmRf*x#tAIhCbMtxUbNzkibK?dce}6|v!P;b$QCT90IO>6BM57Ox_OkBxbs@Bk^2X0R`;VZ-dgY1 zpQ-ltaOjpvaf-jANFUhlT+H{42wRGeUzzvs{Fe24=}Z5uMzvC7IA%80VYlv0bPkcA z$9`=3Sq?zPl1fvxp|Uz^Hg#SQvq`k91GGF{8O}Du{_*mkdVkCGFa^7sH`-{&!iGCC zDpbA;2DaY!fDb6xFHAjfHNseiYSKN(Wo1j`d#|je;G7%8@RYy|9iDY3EByh7k-542up5!52v?eFZ|@u|kS9q-*d|E)%GAej zKJ(jUlh4yR2v@RdX`M&)vz!?xuLSeN`4Tk)8|DIPyJ4VujrySZ35vT;bt&=SNFJN6 zQbDOZkdUT#K1#JRPX^Qz7sgIpeyPF0TQx!-hS12e`nEYRi zy>(PnU-r9=Qz0cnJ?0xUgb=@?P?g55a0R+C@cX>suVl~_6ULlEw7PmLX+D=mz9V|Eu?UTQinn`Ktg>+axV<8oe3e3&k%P5t5 zJm_+^@3~3g`?fTH7anH+!HtV+-C2!OTKvx;Dfve|IwWy_5K43|tL|-4xQ{?P6*hYE zF?Nlq$m%k?2j{!Z-jeu+(XW@kULVo#MmaOTSAvQPWvZkSOrbRKR7XT?yj7k$6bE7k zVK!0lsCVvaKdh#hai-~KeoC90zvy8G0AtSNclN|kHiCgBfV@RL71q?$>|Zsov`KsX zY1$&106rTH;IofQZ@K(~|F#V#LpxbeMUM!M_SCes0g+Sn2L~X~5a>w^Ehs1;eb)E* z4EHasBU#*$BPJjh0eDwZVq#-Qm^dJ1BYRi!A6=_ptUgZ4@?pzK@bSsA zl6g)fLq01rb7(?B!t+k@|GtqM2v-0;3DwbSYbH{c24kHwV_*CWgX{ePZmM9Q=t+@= zoLr&k+d%D7!3#$DxB_7FeEe53G~fe${6wnK)<`Dd{I_rOyvPob=zYv*P?mL@;-N26 z&tVmbHIbi|Cj{&57E}nu?@7gu|C~M|r-SX;Ub!UaV8P%#H00B;pFf?Nx9-EX%N$$O zrG1A)xa}1h<1XPk+s8xhr|CK`>W7>n|OZ_J*u$KHTeO+3Da*YCc${1A1 z4&3}xz6SC<#&`AgrGN#khj`?6R<&2%iQ5aUsrrm3mUq3%k3iwyPl#4kjY8$3;}z9B zLh&ml1GlQSt9q!8BpoR&tt27j8r^+w?vrixZFzY3+vbP(zLJL=`j12Py~^L2s*+6m zRIMs3a0$_uVW#BD8N5j@F0Q{kBuL*LZE8MKGgijl)Or>bxQ1clqt zW+MOCWy{dEf7r&`*;S(T-PLcOyQXFf7iRwMJ2`G3u0-SHQPHEWUh|#>Hgco&els#`1L*V# z2Df}HZT^za#hX-BYypgN*M*Brx2r*|tDo{uR9o6yUMq2P{?oGH(=dA>r|qBn|I`MDhEldBR^0~I+k8eL zm%Q}l4{HTOXZDv_Zhc}j*3%hHH6a0aBA%;Xj<}oRuLd}z_?Ax)n4ikWbp z_~O7!`6tx~1%(>QTjhqV!zjv*#yuF875uC5$th5?JIh$Kg>jIp3|jwdq2`wLpc*z2 z08!ATtV2c80tD>t+3w!i8b{Rt0=3T3&8vY3BtE`mGQVZ$EB!j=PBgv2?%Vjzk-90< zTUwu8O7D`>y|ay4o_=>zpJu;nAs{f_I8!SrQ#VV?`9OP2!@tvDx5V5wq*FCmcsEo9 zqi?`?M)B;TmZv>=V&(IV{w{JX-=!tX?@$(H3+-?dHFUK7Eb^@S@CM_96JB){E5h?G znAU5}+WW4;bDPJ%J)}@2TjA;oW3o*Le7w}R37st*acq^g$jE_^)_G|C~-WENxy}jArr_t>3Q7old&) zs6~TF9Si|3Jb7ZXy6M5#8#ck=w*Q^OWx@aG>tgTBig!Bg(>-4andt2lt>5_n&bblc z<`5OaNOO3L?=@j!2lQV^%68Jsa^hXzQ;9J5_R~)Jlham=s13hj+s&+4!ht>W4tVaN zErvih7OmGG{2>W|u^l2CLpcNe;%Oi(^#9`>>X3E|y7PE1f312fKIq7@#B<-Ceb2j& zy;}pLI*1l$%FYeT(>HL9^l)ONgNsn>GlF@sbLdMazNl2u5u?*R2=k{fBRGn_JQa)4 z33NpEkMr7e=nLra#~AqDae2LBu!$fa$KqbBYTp}2hZcWt{6HUoi1j%nr}GJ_cUYj$ zf2Il2qHCx!m;3C_7`Fkkyh33{G7A@{fMu!Cq(--izZ(g7~xF9(3vLQ+mA3?)y*;~-a~NadXony~nt z+ebM|gzs53I#-8)^9jGO&;1wlVf!@;UKt6z@rCXIjm`TrnwM`Kct$Xc9@d7caQYn! z-1h|DHZHrb1f3-`3fYm2FBT4e+E%DuWQpDGqcwox0ffUlRJE@ z93tu3Rk|$c&Dnm(&-97apT}~=&Mct{B47{k(KZ71%k1=%x*49D2|;0m@5a_{jc&+z z?j$vHh-U^!e!D-VcYrMUoG0Dg}imgC!|d^`S@q6+OK0XJQf^4oC}v{PIHxP)}K z6Ls>%TUq0Y9}+p!+A$1<+Q!#XM~JDY_Tu}tu#Q$Uall+$2~W|^>YQrd`z<}`xar** z?>$YL{7gcarZ(qvDJygONRM-@{g!76!KHmhOa58Yt*#&(QQvZy56BXlQ^d?>pJ1!< zo7TC!cF#`y`E509ESFK9sZlOGoFI+YJ~aAByF!)?5WMT%OBiwqF;fqc6<4 zZZQcmFff|45qoOzkIKiT(_MnU27(~ewHyqkno{4WmDi&GD9w4EPwY&b6@w7Ky_Ll` z19u-$E_Awh8cod?d+)v_c0$2Q{~R^(DcirOLLt!Y7eVguM17vGBRNIc{t8lfVkFz^ z(i#nhj7GUAF#<$x3P!XtAA>>(QpU-1p!-N?Y-5ShnRku7vDvH*lt;lJNcjTi+RY4& z=R{;u*_6pf4Eaq&`tu(b)vp5y$3J@OlDV&lDT1o3`yYDJ6&^^6`V*ks|5LWO7sqzJdmt(@;G>o4C( zpyK-qC~hy*qC!9zL=nUcgaL^D>h(iw#nL!J&kNg_{N_H@tb~OxPcVgf@|gHL)vlg; zEz!3c=Y5d~h_$_6Hc8hz)zQ)#dW%LU@j#uY-mzOXi5L=mi}Q%VFS&bravLVFwwd z2>=}rQOVrwzpCZ&NQnQ6o*fW{3hu}Wwn7?uE;53CEr{XAJN9AekE{J?L^1m0?Dw>~ zNPVZB-L};xHYUUUlMwqQD=9(#}(l@6z?I zg|m zz|ka*&rW&%Hse#7pW6NY!B)Agi3Z`%r04Ws+hB7m3%9|HdP@<6$D(@2l=9du-=M4j zQnIBz*%e?thM`xy($Pr_)xjWr*?1Zrt^%OC5C8tSCIJ$kv6@ZsL2@NDT2Qvl`iKMVql`_Ngug0$G%QbVxYFotpkt!dd$!@sb_wmVAJnnd_y!A_!lR%3)Bmt1#k?Jy_E(X@Sc|bYMRwvD{8WiXisboSOqp2PSU5p` zUf$+Pcx!o}V=Fq@wG};Q{J?k@AtPaT)$>GHB^Aos9w``OrU3nq6xbubv zlsl6QW+kNS69+F#?tN76+>(omHYb+D3!S80PPb_aPPBr%8ZhruNEEFca(2SPRk3N- zcwi^&OQ#*uZo5SjfTW2npID+C5zxtRz3VntCq1DWNJ!4d4v|w&Qey+vbj93>XQ3ir zHb+QeV8HVGRNsEiJ2k|<)3B;pdNCbSSWDb-iJTZ6#F@(X6c=&|fK)6X4&RF3HcI(n z<$o;>4!;h?1Jg@BDbAIm55X9c&}c0BzCr?hloDlN=FQC`S5$n}X$M2gm05uVN-apGlco#l8W9=*?x{=}c?+X(#!D)BYNf4M$z;YRN zMkS#v`!u^@iH&493AE^g@c+)O*UBVrN`+z){d66Z-+0ZeNi3X`KCFxFhO@fOs%C3r zBb*iTk>D%n-#W8WVjm%6V|b;ji;KQKu8Dtdl=;=)Xx-Y);Y~n0^ELla&`;3dJNaD$ zpUlMl=FIZf8G<~|Cq@rpvqirE;dTn2q8`Wh!nEe_4QGeS(2xPwAczEr)*7;n>a{yZ zZf3{|n%!I?l$p4Dcd6VYRD%Y)-IbcB^-+`YCR|Y?(UAmVTITFTW4J>RxhiGi`44F) zPnY<2P_B;$nJE=)W-JTP(V^jg@5jW`{kR!g#0mQ2a|T&m&V8HOzkPhCpYrS-93{YN zdKn0E{EI05nfK4z(zyWXb5)|4x)}YIXb2CuhY=}&Xut6TMx5mz*s#FYQQN5FB*ImO zUU$m6tK*3~OP&GcOySgY86B|BMRI{Zj2Fx&e=c8&?7IO(O!&AI2&rV9#MG%e~yI2lN5e+Zu2 z)z*QfW(40LLgBf+yV6;CZ(@>gO8|?KjNTG)Pu@GYl_MX|pk-%lMaTvRoBMO8B~U~R z9|5;m0aSRzrZN9>kk3sU;l}05mx1w4GQscAbCN)kKfcjrzsJ7IC`Rf$XZ&kJgwLgW z5u`^@k2Qo~V_<%UVv^nKE3Qs(3un&5)N|61?v6&R&%!?(iaV)g5pJ-o-B7?!3Xs7v z-k$=nPW|sU1zestza7(w-ppfL0T!Xyp3&^m?^3u@R#q0jYojZt2`kGLl)48{v5&+I z()0{(&XEfbK;OC5!h56ar6Ftyh-z?F{)hkoT0ZKY{BC*&U}6}r@$8V#7ssS{L|Kxv z49B$!8(AM}E6|uUf9=)2N5zDt-?U&dPr|#GOi!$ORF~TaXL%S}RQPay2jXuQ_qm`Hj z+sAK7-Nq1EzqqXstDH4>9ar>|hOMN8IB))Zjc+=NpLX=>r(U!oOVN*T`%4J<{GOER zTi)L49{1c}`|f(W6g67HYWrQm0bk;ojKR@hC+e3K^~ZuO7uA2-Dk{AygDTYcOT^Nf zZRUoxTsy-gV*nMXxA}0&p#?WXeapXgCC@*5Ob>5@3}hTQ+ZxxtF~eJ}PlC?62@x7D z3f{GtLUln@epuP{lnd6$msPx?H>kPc{6s{YJjr*^`Ej^WP*nj=5{8$jnD=?ooq^SJ zV2L8_#+W|n2kl3ZIO^=n9<-80%selzF985vUnd1wwzIPn!tFvf!ol&m8#0(&U&82E zC6e*1N{EL)5S(*)wAQe#l!oga-{d^0aDi_SMmvZof-ohYmRjA*J(8ST=hhRtw+Vek zSdh^`+*|hig_8as2~7R4iDnfkqCzeFyE&b4{od9vC#fUM0Z zTd5@SE}Gz0*p}Vw86PTx*c0#QW36<#6@^pqScD32cwn`1q_ zopCthCK9z5mP^n$BrmcO*GU=+W~rfxtqi`6om?SQm-rD;$d!CB*J}359)QI`(Pbq! zA@UlRlDlo5<+u3U@CG4oo=2$n44x+-nnSKyRer5R?oA9uO{5`j<7WK5Z}EkPRaw97 z#1*Al-?$Qzuoh`M=?ta4ljlJofbE&`vC<~EG+Q}aAx}M_hcq5z%LCi}A>(QaSN883 zF$-}TRR~Ltj@+xOh)J(=^3XA9$us2dK~*Nv^wYEC073_=&Q0C1{Us7V8YrQX5dcYX zse1sPqf4^nGyDUq8b^C=p3%Wz7#gUTl<9AoEsS*RG!qdPMu8_JSh87A3&cOm>lsDh zUGT1{3@f3?D3}lr@_>r*26Y|h+xGb{!<*={4X6eSrUN4gl@k$vX*Ba7}As1xRPV zisP1L-ID?Kk4|$KE2GVpap&+-(EKc zGkvV~;N#M(PrCpMANonfX8q;kwXP8Ci|Y2gq!qPcYd?}RBZr1xACc6u_Iw+)Xs$Oh+K2@xaA@z8Ma;ia;;)@Ct$#7E&2)+Gnmy}5smB_+4(QO`V@;aA&GF;e zLa%@dFIF*G~MYiwvBs97#E%E;2WkPDH&gmUQ5nd#9>@n zL|wbj#>Wv>IPy_ONN9Q#oS2cO;)@kmFu+GPcA<}DbyTdtZ}Z!Yo$cJAQx88b&%rlz z{PqQh^LKE3{+a5+eB2yyMLHyty`jZ{04ne_a? zuR6Ygv0bOe$MM#ylkav~{;=jw=3l^%T8aMDB^qs6Rh5C1VGRj2Ts{uChN|_|-dDY} zx*|QP*RxWXzR!z!@H77XO^?4@LL!V1anzF6~ z!HMf5amA7)b>$fhrrobpRqT27*?5JMG4ayr(yO%5kK7RgQrbAIVvIv~;TKBPcLDVE zRPprn;~L0VX*>D4S%=za|66a+&|4R#p_K`59P#+brwf@3?5&m|T&$c4?IvIQn3)Zz zS#$^({FaD0@-mn#BX4f{CA%+77Eoemkw2K*7u8l2NcZAmm90*ONMxFI=KzT)FdVDf z>E-D#1)mZ#7P2r2mDC?o8$USP9hFg1i|-p6!1e^!L^Fk5vtT4@*9tN#o12y32UDt_ zOGIhw5eIU?gcToJA~0hzST9KwRTPSNeDLz#F(k~`44nKr`Bxn5e*RoA(#4&j8-OX* z+v(RA7hH}N8$~Z0dv9O(0~Q}aL^mX=V{NzwNy^$Bsy0nt^Nc4~z72o8VyETo8i1WA zF63=O$9~D*LEAd`2aF}*zc(eE7I3ten~d8!D2dKTtIsCQPViL4e~-^3yXK-f4U~R&GfAHQsYZ4s!Hj zg6GF#=j6&000%tAL`1Z)`MRwmrSbzylos0&3U6`8qq=~PoO@`o#n`lH49Vu&T^Hf9 zK;#8te|lTMF&(A&CheM#il-edXB%62E6}6`D;#l7v>*mRuoJK zT#2mwP^;iD`ty~B8T#a>a@#hm`xugiV0V+2lnOl_|5%K05SifN2c0++M(f2Nx`(DZ z^c}oDorJ_Vg45q0e!rL@dFP~8eUGiNHTXb;qMFszf&b)o5#h6HOr_?O<(&1?;7^b# zzuTFa7w1pH*_Ceq$A5EekH~e&wCs6RLA>zN7E*6qw`j?^ciY3^Lep{yj$a6-l2Ou= z`equ@f1GY0&C2=vyqjEgb<29UT5IfB!_s_GM{n)UXQ2l*VPcEZ*}oS8o$>r0ex$Ez zdl7bv0ccS6-fN@#*18r`{{Vr;|6rh_2xD06(`LhDQsThfNa z(;-iYh)CF{zDMx2<*CW2NcVU!@S?os<*j9+eq+DD$iV{o>9RzIy+k@*hxJ*z(p@aU zHWehu09dN$jAIGC3)uTsbu$*bgtd9;i4ft8vkeW;K>^u;jqr+vWj8|2c?{ds}_gUn9c4wD-n^)4op{}`? zd#T_ae9)d|9R36j2m;+Ftsq)JFJ5OZD{6}JBi++mzmHDGaF`6JY2N_C3pxG-ak=7O z-PRr&Jg1MH_2%lQR}gJJh(BhmXsJz(%LRdE2})zeS%UDFUFgj4@AQS==aN$&|G1c= zba%=4=1=@AGcT-DO3Rw(Z4y5EY&Mv>SLIb!UVCS|dT6OXpsjC56d?Zkk3d835oauB zw_#R)AvH*@-6l__0~H~udf~pVh_fMUTco*DS0+PE+D`t9?T6ZdGX6TaG|K72)rcHn zg#V0TF}bqR%DH--tY9^WK0{B@>cS1dr$nGcO^n5$Lg2nC=yquG!!8XaRPaNL@DJzT z{r3XSE_pm#dj`@>M~NyC<*2Fn61X(t?gm+J8(a#4X`vu|Kzk!{-+RH9tMJRMfH6IT z*Ogv5^OxG2c80t)rl&p;n=*2JLw<}PuNctByjeoQt5X~Y14>l7oE`6fi{@8Xb;^)i z5UAM7gIl*L>mQfL;&WVKzuk<*>3J#v6p&kdL>_iz;fraK{DH`^xDd{#9na}%cEnH< z6+s2UJIAIWoT7-{iSZ+SI*zXEpFHu_l^+|lp!I@RUX~-DPu7dB^u23+-AuwqBX**=coZmqe{~1;;c=RpkdCjYX2#g#M)Jmm%rAEWUg^C6) z*fhpJ@kS?+6#4Z)ua%msR{WstFGxD$ZNX;jzvJ#BuNv@{wi^dLoh!}y9Xj%BZq7hV zv`vQKTyG@v=68i!CcJ71XtmI%VNd#L_qX`amqPfkZPovcZCK^Vmc@YOc%qFl3x`{&{0da3Q}cwcossP%a;HWikCcVLN;}_8C&~-O%Ph%r^-i3Dc&2vUr+Y>#%QAez=SsV--V zbS&jNb6V3=x$*I>aD)M_UjceN(IuHUs6SwMlELroF`Dbet@)PEHuse_HOnz}rG%{K zpSR8#?5iJs$FX64fxhq$;u#~-JaR_|5~14 zM>HtY-^qs!5>r6|A22bQ6Q`dv(&2`VAucQWudAt}aS$XE#)fD}WZ=g-yG%}=PKflh z*9OuWn~y&xp2#EzYpCNggvb#uarLGrt`Jq!&kp5?!(`z|EXR>DmDA7Z*wk%*Luf7x z+xR)uQeHJ9Y3@Yt*cTrxGG&gwKo!D^ycKR7R}Z0*;5MvP$daH~&EU7s67AQ*r%fxR z$d$pmffIe($iWQ$N1i51;U28^LD=@5_e!7!EyJml0FMPFISP%6rMG4(-7Gl-#*iUl zllUHc8Bp_WVuzLE>!}p-w7$PUNPf4x`$xkY)$=BYM>9qZ4R2QUDkRqLty0ve z-)=eMf~>+Y6h-{@DZF;AHm4h1mrc%{5Z8ze&@RphxkTQHXmsEnN&9oc2>09XO{jkI zfRr(Q2Fa@q9-@JW#q$I`#uO=dZ4&m2x-UZ{$vW(=7hjCreVK>cQj{Q2=4xypXz;GS zkf`#bpU@ZtW*_PW$}}Nife1{K^~)%aJ2aQ!tSxsRT6av|52G!XThz<*34!;O={7aav>*^ zVDJ43xFgPvqDl+yqd8%!dwmzPj#r?39Zhw*i|~#yCGE@gqlBpy2aRsILuvFW3VN!z z(=w=l4rlQ!Emw{uyDEfarucn3A9jv(Lx6fb4~G+u?2)s^Ow1U%qy~vECvI{j+3>*W z2s72tgVdWAi}g2i(Kkc4qs$ zY4d~#P5j`X@#ex%j%?!ZL}kON35IGeCPYR?>dOXoN0{Z%+an;9_Fl%0dh^-!Vc-kS zWbI9Nt}~_2YqswZFB_VU9l;k`@Sj!4WxrGHa zd0znVX2--0FDas3m($R8Da89)U3I-LZh3k^jz@)u5ysi<%gWTDg9tEC(M8dZjL5a? z6mS|5YNYREF-S~GQuo$Q4mQ;GgG^1&h#U?^iA=jVeWtt<&>D~PrVJufUBBsqGYoP! z?KVpD%B$tWngVfbqJ-@>X2OeWG!zY(^#NsX zneYGpW!3#N#*R)WBq$p3Cz>>^jU~#@1QoV$J-gGy7W?VAh~m#EQFMpUGK}%XxkZ2i zob)q>L!hBYi@t>y{5cYHgnqnB`seRQbq%ilG!_!jjl~aJe0$2t$(n;2{NV!xtB*_0 zSQ1iTwlQ(K`euC5-Rv{(1PMKuoyVX`TvtLir1I3o=>tY1IwVj7{;UzhFVK{ILF3#f zfcDUjCUi@Q6`5N_ND}y!?u7;B>uS1Z#KF|m4HU8$Qjsd$KRmbQcN);B$Iy-rDb0&| z& zp-Tob1wCsyqz{KpWbXYY{(y+=WrC`HdI$v zk?q+$`?sh#0=sXEJ+!?F9<%T&^(xCL$|@!g?b}}^ z1U(J(aPjtT4$c}xC9!{q4pnGcA|HQ@j=dTK);0n74cmX%-5c9BLQ1doCft8KwU~gqwTRHs*Df)s zia@lj=)9T_IHO3SXei3J@BYotSo9j6pDYl#IhNr82M^Vs98=z|nr54R9aLV4NtSN^sJ7rLF?M<*XLw5R?Tu%!|Vmp*!Y4cdWEZSE8Fe4 z0<5eA5~$*JgpeSV5~OeHire<4D4Kvp_^`7mJ2a>(#h6L@C@^DwW(!y3 z(Cp=irv;Q5UYnNf`*2-4Q0(8%c!iu%vN0e(%0+$NwL?MwD&b^2-ik@{z0aPhAdn33 z1zQ$!7DcY+M15pk6`V5L(MN8?grqnWI5`f{y`mVr>=qCfOpJ30tiRs%cHmoaXi!%% zpm$H#my;(>*Mqtr6wKWMXOQCqX9H)0oObtl8f{WxLts8Y4vJ_NNbD=dtHC}Y6DAJu z)(;lQh%;j+7F9X)W&c~FWEL2w<#>8=bZ?3!CfQ??VS!N!mP7aXfyO5)2M;(ZxCYYh zOlX6#eo<1;sX}W>z%?h%f77l$S2Z?)o+!2Dw;uX^VL^b!jy9~c>vUQG6$^bupC&WO zQ7de#0-=zZmE`ELCan#bIlfvEKm%YFcW-UT2(}^evE+`MSb4F8D^<8iy8Iw*jtl99 z%ZA1C<13$q^5jv!tt0%-K9ehr!wx|9AO*-CI)Et!#;5=90XJNFD|;@Sz8iA0e0{i! ze92SziUiOA{q{u3F74tgTR;7wGMb67rZM>Bi>LKzYDf1fYZ55rI_}E^E#Mc90&_gW zzO4$N_-NoQxlg_s-?AowD(NKPSSintd#s%x?Y^xDMCHgxY3%4HanfDn9~Hg+6%;qHDpv`knOXep;h6}$Vl2d*@RROYYS8~lpP_%4ri zeJ_`rvV9e^kWSt=;EAgr_FY`Km`@JWz6A+LZ0HG9fI+ z0X7VXS5qcM5Ry1;+UTSDa}x zJ!$>($FQxDzG_?Gc<=MxdPZI8MC}PsWF2+&)%uZ$62?= z)y3<4BGvFuQ*&1fmp?{ER$y}x_$&I4m;CX4ajwhc?ogwz@uJ7oe$aMqcu+LK$BNc- z-ONvmeSN&&o{2y=|EKgoVg;ke3=56_{|7@D(Cu43s#n24fCD1xf33aOzP$Dz4DsB% z>~8o7xuIF{L@}h>MV9~9xc;XZa(Eus0l)cixKQ0aD~DFD+os3)PN(i6om7y&fzls$ z6YdB@s28;#JN52S>v_fh`UhQ2&;0(iMOdUi8?wyNN%&5r3Ml3=+;k=V3aNq$ygIvl zt?mAK`C4#m!5x^dZmK?t%w_?R8S10R?0 z-TyP_+j_1=ld0PjTiR87GCb%kdJM!0V&Z5_(FyuXRYpGMWQzbn>mXiP1B>}-^D52$agHNH`LvZ+!e0cu5 zi`Q+UWzi~yf*t=2`Z~si+VF|d?#|UA(Ao2}k5=-UVs|$=49{V0X}nRd`ZtcwCS0A6 z>}NQDX@;R`v>)6szoQ>NG~^ty66yHs_F})w^kLdJfa_Iz#66@f(QkZ4rG4MBC_BRc z!S(~uu<(11BFUKSxm)ca%N18}U*rAV#aa-}GScSCyRt+r3m5rvGCea>IVL7X{szC zn+rCpzUTQ;NjBTD}u>09gU^+y>f9kwnKh-C+8WHFE=kJVD*G`lx?be$S zXFjl9|0s_Wf}G;U2q)Q)u49^sE{+WSd!gSC5ahNPrQ8_j^I*5s-<}k3(pL$hYG-O{ z19>+nwQmhwl;7W?ehlv?{67AdZ7~;F%83)ZTU$@0y4}0OQy%0Z{Y{BUF#2jjb&H*av14H-*Fj6?=i@3^K|&t z%#{!-)oXlrvof|#htNJJf}6v5pAZX zVvFhXvu`zDiq%lTsKYD$!Jh|&UJ?*ZGqHY;&y?2r%XgNY-E$IENor7lb{ylz{41iA zs&da>`|C3oILYvGjfJY?|&zm4G#WE{MBvDdfSqaCa2rvbv-(2QZ1Y-MGO^6 z6^BWXpu{cU4o^Qib4DpA9H)EDXK>nlh6XJNLf zR{VuW!KqU2h@fJT3jWR*nCfu;uXlxs(EcY{dd~_hgN@LD^rL`gC`pP4TGFIT`t9pm zz<3>4Xn3GO{|hc`s!sz1so18b zVUD3rfO+O~A#YCK^8?Q_`#1ys7trR`(}Rj9W8GS>bgEX3h3zLYyl}Tid=B%6Kh+P; zh(}Y%0)C=TGKG9p1PY`ZSJ9^ZVyw4FQIXD5}^{?rCCd;~M^wR~?{Bz*5emCugWRsZ@Dib*Xke@<_PT?;R{ zK1iRf4K1-k=q1IQNZscR`yc>Bf3-N?-NiDNGl8h+Ww>6OnxE!Q>}wo+vO@t zA?$AnpXGLhWmZwigvBig;kx+A61mwvKrj`|gKtexs##eQQMOWt3hrj7X6vj2275HqmE48OzP%IJM7xNuv zoO*hj-U-yPn$VbQZAK{l{B15|9}^Hz1=|tk646(^jN^*$koG9T%6jt^<^D&5GQ`Nc zf7*tf)pJFBQx!;K5OGL0JbxhKu~VUKV`cT@k>q8~1wUoE%U2au9`*DM;pOGE29#B2 z09rVXrcX=q@=A_hQlN2h(ab~ zp@TsloO_&YO{<@-mnP+RF4zba#_s&)*DKaykNUQCG-wMAqdxm>veexTqRdfW$e!xp z!d%98n{@lO(2l0~4T@;;=B^j(-g}8Si*kM>6!Hv=ewx$jqxp(USeHDJO7miOBKw}1 z4&|jp|Jm2TgT-Izk5V@uK*{hG`} zGk8!;t7JGsiUHwuud-4=YRdC%QP^KfRv!-`{&)O6|5(N2Z~QA{SZ^pyta!Za3ui zt1Dq)W;s;Qu)Nv({@zHLQ^(ES*KQ2~^I;5SRN~Gxe@J~U4tR`tH^PWq46wOg4Fgu! z{I*tH66Sp)`V6MjfIqdy^%5b3c+da)zX-2fK)Q;IP9=u=TpXrUp}Xm473nZhWmQNtB8{;`+Q;G6>$8dSt*e`obRmIj99f1MSu~TPJ zO+~#Pz1d$vDr2Omfp%(`6Q;N{qxm<{x8fc5 zEn5KYu#N;@=E%C?38|0~5K)#9{MHgOWf^24Bti~G;BFr}ZS1xqMg5vilJKIpA zy4u+VU?y)wpUvBioc|OqY()X*ctXS@?=7cYT{@%)1!~pCs63(ddU8`!&!2mkmHxTD zN7S)w8(k|136~pYv#gRI^=$A@9Qe}Qc^igXI7s>J#WrR9wLQE0$9{cxDSOg6UK&gI z68c$+&A#!2>hO$%g?rNZjJfk^w(Pe{e|MfAuSC$WAgn?!LLkpr1SJCYv4;h_qfk*P z*m~fYb%+j0p;E7zHo|^&4Kq(xT1-K@j`ATuas3hIDSrFmjs36R0f5r6Yg70s1&SYR zZ-|%`NVSkO@h3wp<^D9zKNp`$U|=RF6z^LQj?Iy5)i)N^cSgpU0h+qHP5V*OU~}ua z;h{wWYEh3Fc90FsPm)X)$R==0iwC_IWy;k_d_NIPspS z_kJTMmk{)f^?*EQf{b>()~Kn}4)eEr2avWT$9?Uzu&{7kDE{DZxmJ~5|DD+m{x!-t zsW!&q?)B1??aYwZWcbUn^|A^ejSDH<+tHOrh?gUm8TAg@{+bg0bu67d$5RTetu|`o zrgR_U)KmX1wH(9lomZ4$XHi;>&DS+pbx+-ilGtnhl|TQSLV971n31El_<^`P%4Zi} zT(Cv>eYG?MKLDG7!nN}_5FcMQbM1eF!lKcG>_72^`BT03DOK9j_5*Su6I8w?$t2zr zqg>V)V(4-qV1rxc7fFQ!LrZ6Szi@nMz~8rAk4idndhOc|CfzM4X;k=OW*lW3X`G+p zQ)-v2Fl|#hfv-X3aqLJ@Pn906M1J@}sJmjhfTiYT`eJitcDpQ7Iz!p01HWf5KXx>6 zjKi@s&)Sl_MS}47)A4g1ADQ{Jjk@dek0FLYu&<*CRe8ZM!oOWlR}y10wvbEF@E^G9 zTLFg!8W`Bi3?RMRua)KA&l;Neo z<;Yn+yyI%y3<&-*{qV;-`OS-c=Z-4&?8od$p)~m9!`fJ9g;}~*IAZ8EeJm+$Jn7>H z^>L$|4tu7+e!f8Gq-8)-LAqP%M!JSnKw6aU?gnWF z5Rh&uX^9~O1{h%Iv$+5FdCv3Vxz6?Q!fUvg+2h)?_MW}h@3X#Z8Q0}b4VNhwYgU+C zkI#j6X%wSFdx-cq~5^S5#Or&U5wiE5DbxLo)ttb!- zWcqo2H;XFG1@{sJmL-J-+mMwisDKl{6uSA0imEY38evkR=;Y@gryO`-LID+@j`bTJ zE8NOtgig5S+#W_X;`duPr_5+-5S!J><+{uD*VSNg^fweaGY=ZE(*u)BxPWXUVBKZ zLRCT)C#{w6mFJlNSCEahbv|;?wwQRQ!--2jEZcZ!LDA``>y`3a1NCKKo{ivIdW-JL?^*6|=HtouQB>681DbZ3j%wxuyYH%yhgc{=H1hNe?wubvO*Zyg z*fmSRV#bNP&pOrGFIG$8YrW59{N_~a3kd9}X)@v8gdwfAaU6%b2KqJO8x>6~MIwlm zrHs?>W8Sy}TTbSP^wuhmZm(?-%NO`GRGjblb)ba2t3q2pG_Wb?sODz7L?~U4=)xw3atXGi7udSwG zoyGleCNl#717+QN?Kfk&<3MBK*Svi%{CtF*n&aG>|a?_D_))d_BMC6}EJ`vvV0 z&Mt9bE-U8;wl6gpo|ZRqsho;-ZoM`9WXoEgXFkGwIo@ND-xXMb9Epo_@fPcuq>=%u zl6PThT>C;G_S&2Yj8&PkijA9~ZjfDQP7>GDe=5{Z7dyZBJK`+-4I&=E1Wf8^NbQ1H*1wO zCsFT_&-v$9KtxQdo*D%zVU>?E+p!m*{+J3ChwnIh?^3gvbX8H-LVPK@XGqo+-V&Xk zwTAi!Ds83XL8u_vw?5E=n^JGuG?~x+hscv_kX;~m8IXNY--f7poAS9o;y17i` zRLxVG;h2g?hW}$6;JGq3{Pym!DI9mB*%p(EPpmxMfG<)#(0wf883iMs-Rb|XC&lZY z^NEE=#r@;6(qT3~ej9`f|@Ra%$4KC zsxpTeDHe=oBn_?AGh9o^D2Lu`nP63#_qcix8626!dgr_4Y?G${3-k=P4<;Ao7{t&> zbahE-nMSL*Ck6MON$NK{hLAuV(QB2%wWl^idEa&vOomRt&Av4iqReFqF&nbQ9EY>x zY8`htbvdgi#z-vctXXTc<*qS}p3_<`?=YheZm{dGK+|I4=-z*gO=UE~c!v=j6{qt~ z>etZ4NY=iBnd4r2$OdmUZ)_|xnA@S^S2ehbuHWET$h3^ni}stu;_>P3N7V^lvm*@= zZDV@mohafVu+-?u76-X}SjTkVCm3Hu#1j!$#TRzGR^zpvv;!%$E0|_x@Hd1H9&uMq zwzo%k%9~BavxtX6`0%1EWmRGre%vF?F(%neS(G#bgK-&crg8d)I-Ws;Q)rVjD0(59 z_K^JYN3EhR$lSsCr~c$u5i#5c#G{4lotgKm`>m_&n4SCj*gn`xYPxamig>)0Ajlck ze>cRV#{6-_0+jjaf2~G7q)?nN=gRLumoZ@)iDzJwo9_0b!~7gyw&>x^n6=peMs9IX z@1C0sXOq=>-w+i@H?CA*K%U&-MWYq* z4pY0rrsVzx_1qcsQ0)=fmZ z>!*rO>keNAdwd0{Z>1(A0wr?-PDV82<@Lf&iw&6~}B4Aa&`M zwXri)ZB$AZ(wY^EV-v;#;fTv#C81wpWo0$(&F8$<($RUdZf0gG`%OSNn6sx#)!LkV zeeUn!qCKcjNIAaYzH;^?y2`C*L$2jWW1)OcThllhOmzRLQ3Qd|VY89_${52hlGs}; zq)CKIj&HQ+r@!pReKx^$g@R;f(%@QxAp`Xn3D_vJ+yb=-OM;$o%539RW?KGuT4Vd4 zL7!g>yKnIr#QVEM4)0~OJzhcJw+#QGtKIMJ?orN&qeQMQN^_1%E) zo50mPSb<;KTyr6K_jrHc^GRBTDEzzQwJN4T#AheZoA}GuOQ;xbN6u%bX{g7cTB;!K zMAqC3gnsc%e1npV4vdTxR5mpU&+f(PEa9_~gOKl-Ow;%8pMmJ>UhJd8!^bfx@{Qy% z-@dIL8d@AWOU{lQbU9q5mpx5*AidUCI@}OFqS&fMq5A$w{)l5z+wq|&;&?yki-YT5 zSH`8jT&g>Y&)PWVxI76#i2cCS;0~jk2`zDp^k>hO@=8X*b-SBM$B4E zHeo`iW_dA-Ze7TkSFL*4-lRCt;mCGJF+dB)qu3L-Jir0RYiMYQH28)ERWwLI*NWqh z6Nm$Z+9Ia4Ot^gi);dd4?Nv*fb;()79|YvKHKrcR6oUC9$aayJn{0RljhYC_ISNzt zqG#sngz5oUABz)zGOLSaUHgW^OJK#1eo=_Jj#}HD^nBH?ehL2`ujn zW5tAKd&xrVIc)peM4Q7p8>B<+<3rW~jA@XUt!oYiNR0-aI;g0q$jSVRDu+0hj7GVB zBdu0uat|xwQ9bnF=SY8l_S%{$wsvbYVrpb?Fc#vbsdp6{OR7fGx!ynRt!tFGeEtEx zD%;zyq@?8T>$e9zw(%%|*zc@HW~8UffF8EBUes92l%1MP<*+BoC-%eEmwm3k#{v-& z?GxtXu)ZoX?hQ(-Qy<=f3}DNR)fu~zXy?0-;b#s$SB+W69cXkkZ_%iJZ){h1R%RJIO1)6vqg)+3@0s2ythr?)>3d zKy+FrKwDLQgV4(lsDU1L<8>23BOgB*SaMq#sL^X|eM>S@{-Q*w(}+B*MgL8qA_em4mXG?#b%qG&Y7>Zzpd~Qo8-7QIq`j zqlYZyG|M6`-XzQ}%yqiLsNne}LDu3giP8JOcrA8)x5@4s8-2TAPCn7mpiBE_HN4Y3 z%OWA(Cd{?0-2zkG9e^=Q`Py3job#V0zw9)zsl$5gMo}|=5DJ?~W|%KAN%Qx5GX+Eg zo&nDI^LU5j*QiURVTy)a2=Sv2YmIL^flVB(ktcY`z=VoLl28Rrx==5`5k_b(kwYBF z->E#psWZJx(jvGe`#E&yGQd6*hSpVo^?ml5Wz4hC??w~KZgc~=2sqF%`h0yQepVbk zf(O++y7!|8^6Q@tTG^>bgU3rUMX-bgZ9rm3{&>j3O$^0jF~O4gp24_sq{a)Qp8D zpq3~9QDP$hDzP|%lVz0S9hxw-s@0+POrXH#`elh|-CEjwa&AGGicS>Qun~lHqzno@ zSUi_uV*DgL^bQ!YwLvP4QC`K&A>^0w2viFXiK4S(ff;fG(Cwi(3REs_J3(w zJ!Izn`bTaV^qNv=Da}$FF3{0_C0pCBW4|~6=n5xe1{UlN1+ZYY|I33$e*!uIbGjyF zi&O6EwaERx!y%c{EMF)r*L^X#hA2)y+vp3c#uvquj>w*6RzBEHZP7$+1(7(J&PzMl@ zw~%rsX#VnG=$4l_DlX4IF9A}lN2tP{$}Bf?`rv&}V!*32bEdkst0==BupM2nv46qJ zYf+V^y0@19zC1UyjQvGia z*3lsm^NEq05@`!Py{d&5=5!lhxGvvK9jw&2y8%~{BpePJ{x7rv+)2O3Vq-P&psR=5 zoF0%alD*r8+}fWf!V%OaZhv&ofCmE1(9s-X{psaFq$0GpzrMKUsD9t-7DKrhOYCMv zRg4-M<3RpFVdbkeCtzokf(Yz%l)0-LVqtQHg$9oY7(f^>fB^ZCpB_{NasW-V46e@p z*9R1c_iH~(e_>efLuBlMu#`>gqkuIFLu^wfFBeg_QpS6!^beXf0i+HbMgfJZHHQn6 zv+o7xGa$juD=9vWnztZt`T!8Jq-c|$Buk|a2p~}lb%3qR{|5qQ1VvL@Y+N))jr7an z$2JQKJNPK=08*7hLUPz8P}2Fa6gnUb8Z2tWfI7V%rtzPD3@d#~ghl)=c3VLYTq=&@oPk6cfK3e#>(RjJ z!Am_fVQo98N?$OC0ZUrHrw6AFC^rA_djkJ2oB5B{{D+I(QZV~LIfbVVJ`tsfabLy% zhQN%bM{RK+gW*<#UA~qVKsGyQj-h$}F*&u)6aZ3xF1r0O*PZs1G1V&(+nL?}xyG0S5tg{l9Hu0-Zz10kjF=#1i4=cjq1i zx83E}COESG*_?xhEXDn$(x~CvZUjUjJomppwJ&7bDmd-M<%7fj;`tno4)`%qp*zDgPCIqj zyEZP?#m~K>eNM_Qcc%3=x4)DB{!kUwGz?NRF6OpiRSYjYY{|HmisKdc&*pl-rAbh{!*;V{ppt_B z%JSgg+eWl;4;k3h(M1djl!~!u-56Mv31u;H7FW4pj;mcSx_+;Tuh;-i_e`zR_C&L~-?(*y}gi<#~BU z>`{&){($m5r;_!hLXsL}H-BBXmaFAVi1|O?N0M!`Di(K*_N{y#V|Wo{ZU2gZKDO!q z&xOx-WH-0{+G?tQv~w@p&Ce^GW5}>4>YaZQ0l)TnAa}5@Q)g4Y~1l1oW zlN%T$u5i|q*0fK@ySe}?W~^C+_L5WnhRfqflT4F&eKx{DThfL$XLy~vhU<4HT}eHdv5A+78FG4&Y*L1ro*1GSA?YNz11@6|>jB#$CBSc!fEKi(TLewHT<&FXUhmbLl^)F-W)1wel zw6`x(1(V8FhS@#BQvPok=5F7yTM9ylnWgEy@L*4|>wkiq88R#Pw997R!o2n57S5LP zazh8o{;Fu}HHa~mBABU0`g|4Q1Ma^M&Fk?X7+C~QX=%UK*)P3ncH7SqJdj_p5sdpB zdy!$}UI3U=|FP@cmAwWU-$TmCN4$!Qk9y@ASoSQ8d=)@CvEw{ZDj8uAJxaO#?|2Wz zQ$(7aRkk~M{6ufsq`In|&+bX9(!%#8 zZj|N-B)N?(8SODvRG4DP1fXXeEqb~#aHexf;zAG);Ev+$319r;U^)gY8I6clLw-tv zD7LIg_;yj&Y<*I!z>f!KU;|rw7c|)&k|7fy3V9O|lXKWAc5`f)EaV;KORfeI{QR#y zrbTOgdN^Ad%t)5Kvv$~!C*Cx@a!!6lKwh{+qh4}XIeM`z%gM_Z$fkQF!}b%|x_L+fQ9YQHXfZ;^Q&rJXY9Gbtp= zs1*!lh9_{=CqFoPWAh6i*+o2(WgrGU=)z$+NrU_uMEK@AgOMI(tqsaC4 zVM+vIC-k?m!kn^P`t=PSTRe>st4vB`xqT}yd^Xs_BBK+A27t~9UymvP$)+k^UNR+~ zN4Pj^!olIT9*coNq*T+7$}NYV5bbo;4W)}1fy8nzWS@m@0Kv@@F-KKJUKJ?laRkB+ z!2`>BD9JVFSc}1T z^RmV4TBJ)8?a4P`7Oe^cSx{e3ZF&hp)Gs8oum>Ta`%_;`)-R2bpvNK*4s8J+88Sgo zG~kKJsrjJd-)FwR^yDvXJL1Kz$OmOb3}&=a03$1H4@J-Z;vVjYek%~ppm}cpgI;LK znG43gT45~Wb-X3JMM+8gD_ZopRR7*a8hd=6mb%LeQRF6xlW^&(;{g|BAiTmQ$Io}d zz$1D>&3LixdiRyrZkc;gZm!gDf(t{^ZJ2QqwMnbx4+vvsu&+v9cXrWDeaI7{i!E-i z?Z&o&<9AICu$`?IEEEXz?XuYZa|s2Dc0Z3K%fL3Y(xLpwb^nk7^`(O>bjEQVt@Zo0 z-ubCj^u(^(lWZde&qQC2obbzwlRknmY7iR**YcGZ$l+IeRc2rwbiV$b0qFyU+%p1m zdJZ|8mELm3jVN_Hs~ut*%fSw&X7!^9!RiGT^+hs3hXPOEjU0{Z8SVviQxTjR}FxJ1`0( z&&Vf4%DT<=qU&oLVSWOyU$+1@=d@sHje&E6`jZhGlO)~vN96pn#DV_hA1RsOo`>iU zxm>*sh8?(AzgD74jTkk&Hd|l+r6x`|vC2z%tRmCXHJYO&N_R1)m?6ZJlGA#Q|HXfo zwQm1Mw|s0|8pk`ZGGdBkNK|h?`jgg+enw=f=aJb)i0xsYMMlDqJCf;7>#b7yfk$bl z=@!+iUC?ia1|8KoINS3>shop2@HI=SCl{yd(+dpjAXL7~x5ONQ^$G@3V}tO@Phn%o z3q%k7vai4{_j#*-h7Q`NHll;RDc@M-ue2#Y&({~N1v{7o;YY%-&!y(u!Z{K4JG+)Y zw^ud*-rz-hmw?Ssd!UM8(3;;9N!6Sj%XG97!Jgxp+%pk1v9JL6Ai^AXd z9B=b^!9PZuO}r3m^UohjWQafTxAYp$k0mW-iu+M861$^Q|Na~1b~w7WC8S)xRmMCI z3oFRUaPsdHL61UABXySuw`0XS(B7^uWvq5fjq*@fTj&Oy6z+{n!384s8>9K{f3y4H zq1D=h=yJYEf%jpIV%6P}Abj>Vz%d0!TlcU%aC!>#ET4q{VVB~ zdYQ}nzosU+QGrj5{3{BxVtFu;kooZtU!ko#^2=MR6yIoKhn7L@t%6sqZR#Lzrd;)u zFAg~ufNq$5b}o-msS||0w2%{Sb(Z91{opz3%l8#iX(Q5DHXg#%&pAC&2~5I@x~3Yq z)GCgQ!_Vpb*H3klwg8@xM|hQ@aJA;xVJ6~;+R|~oppA$ z*`e){-I{tdn?ax>1kPzq# zcke4ghr6OkSLk8D9~gSZ_!-jN%l*C3mAzXTt< zLeWp8s8SvX2kLn*Mz2CL@m{K8fJ5g`1s)Knv3`;LW--^3mF8)P6~Ds*<308-Nl_Ut zyf*4NO+2e4I}$ha=jt)Db=Od$L)n6e3sfL2kbzU$Z`g9`U;8gu=bx5<)7Zp-ST5|M zNAMg&dV|8(tx7-NS2R5s7Cpk`3f1YJwse1yoF}lebxdT78 z5z1EGZK}5uGm$29{6bN`yq>J zAmz2S5%CrJ4DH(NMs%x^8B1Ty*lJ5Y$ekR7nzNz>UJHufV`|y1=`3@8t{P#6BvO{B zA?P0*jK2zg=m^Yy#NUvVXUSo5oiY($pU+miHbW!xIjf6vdK3i<#`>vIjF0>s()=NJ zpFqx$yh%kM6!DJw=7V-|oV$?mao$apDw^ijGk$>gdaoKBkH{N2ilZIEdxNXWHbc5J z*a`Cu8okOesn3}yp#itL#5~xy)j|s@)s}uNR z6z?ecQ(Z-Qy;rBeU|u@jUd5y-*F4)7TasR7RCH4m@LQohTP&W8BHAZmQtESkuZLEx zW$33S@#qR_-R0S}$z6^1_+kSr;_!8%*y$Mx^}fUFju@%a>QRA5X_25OFCI9J-?{jIJ)@5* z)v&wJ8MBp$CLrPKLt>~_9~3Afk@9b{?!^Zl(mQ2(>jk0E{LqHNC(??MYZb{gGt*_Bk3 zFYHh`E+O|t|LRKaQ!^p6v5?AB+3^9_u;}`eysyD^k%)7AXd4M=W(Zp4-eVmJaG9uT z=Cjig+WXi1+*SumVO|T=a?36bq8Rzc25S2CB5`skALAidvQCD-vEd<=CvX&0mCBFo z8mIoa8E8vM=A3!y>XtYE!m(Z-u`x)&jGZ^|F5z0_|JFRxgOdtWAR1%aPZ_R?E7c!3 zqvyBQy8f(U)ClgDxAf37z<-gBXCnNA#D0)QMjblbp^GP7`vhhyu}ax^i}k?i3BIg0 zq>bEqR^n}K&9eL1I@4wYj(Lm295gk?EA0Ih=k7hU1%s?@HDIAQPM zWuaY*V#P3Lj^FNkpZ~l?>FwUi2Xi@^$}YCe)wC`zw(5M*KcA^HEZ6fnid{`d6l|?X zJB%mI4-k8)@H=k`B{xNnN`^0Hc76TDKY4VP7d3(dWmTT4W||rsQCD#{c<@f!VC5e; zSy2~A5YQ31*BnwG_|W>`x~i=0lM!HoyKf;4ENWzrwD=h!S z4&gG1rJh@prPSdL<`wRSJNpv|aC>;5gB@ZW4LTmPkPcC+^ba{)uJ-*JI}B5`hL$2Z zbi$-`G9+Oe=}{B}QZghx=u0y>`4P+rHg2iNzwi)L%Zq7)BMT8dg7q=VYcW!GRl0zT*x2Neem0!;j`aay9wD>3!u)%YePk9bLM6G*! z&3}icUv<$)TT!gp5j**bq1~Mf)=x0^x7^fWr$#Gf6}LH0O^+$rZtkLjBh%$drX3~} z*7epm$VNess?}y;-wU^K_HDio^l5@VhGI#inW)Z07USmp(aaT+wIxripz}U$b{2<@ zkai!WaxC75h4P*KBYI!HJlq8#_kG->Lr0YP)}Tkkip zQNIjyKa`xSWQW?6xTu}|y|>bj?HVWTs!(=hjj2zT)w&?^Y3oCcau>iwmScymPF^iy zPM0pfE4v-F;;m!iC=Dh@fBj?CNYM7> zP%_$`_AO$nH_YD;~>zDg^o@3ht2k2kjvhmAsf^Bzw@$3;`Vc&1th|e}<`*q+G+BASq{cWoxYRpUmOO1m^#&cdsnmu$i{h$1ZkOLQF z*HHfHff~H3To;6EoCM2i70cltA1Za{|f`;O_gu)(c z^ad@D6cnhqdgIq!LxeRI)9ov*TLMRTDR2))?;Rlz3goihy|wjyiaDQn{b>r zO zXw|vE*=h)Ra>;`&fTwQRaoUa|J{Mb>-Gp&7)FRc>T9WBzlG{wb501x<^Hgun!8++j zhmFEW+aD>$#Z9_MqwO@q!~|;^LMJxH`d#(bmWqp-jd#uAPE)osG9*9X;~ICQP5pS` zLiQq9SweFTy$^+5kfd>OTFOhWa<)`It?seT(D^Sn4MmAa>|miO;bkzr`t96)zkjfQ zQi=Av?J;ALXMV1k@_3wjmCuA!a2OoZJ?dUkjlBZwZbsOjU?*7)L%XWMO~noUFOH4R z;}30i$8Bu-!kQ5Sh2?STH#FtHW~f&;{a7nn6-N}=^4`ytU@S(%715S1yGv3l*jP(U zc@HkT?XOl#H+VR}-f0T#aqhu-&B665#HwfB%h@}@^_T@6-Fdco27 zr#sYo+SFRt=ovWk$xn2i-W4b8-Mc(fKFIV}!miI^Ep`)sKtMFY$=!yfR*8w=&~EyD z)>POab+RBWC6Z3j{BG+}GJSH3<#qOK5hkJtPbt7*AwOm0_Y7>u{}cXQiRb9e)(e^i zAps;2b$j#fiT%<)?;*NFH!6{D+e2KXy1VMP1ps+t;Sqb`Go-Gr%G%KY@NROmB?Ezd zCEmIN0!ug$2Nv}ddGL&cm6IdKW?I)0z9{+2puNj*%_E-OdSetGY@Rn{xRy$wu;3}L zDI^;d*&fn?nW(yA>%_K}RRJ;bc6Y0OX1{E%I@vMVMgyUl*|bLb1(LDI&@zYg9% z$Li;*ebZnio29Zh4)kzRSusS4eBkvnT!_*1K~UJ}FRYg8YUlMy!rmXXk+0exWC}T5 z)@~Tg3pK>hN;?ZOa;ltMn1!9$Nk@e|c-5U98hT19&OM&&!4Nc@+RQ$+!fd9pUyyGm zC&>D~9bNoWrNM8u7tbFO6Opmi@r@_FlIYz7BgzBtkIWq-y@N4yzu4nNfDpXY?U^3K z+8okpP`#-my-Nr_TBaICw2*Eb4e(b}YNK(oo2dg=^(-Sru`ih~yjO5$Uy3?kS1%e@ zNyt6JhKYkezIgw<{cp!Bv-P;@mY{ANSFo2o)DyamDZ{{JK$URXKhwP@SYkVRTyLk2 zD}yHglNx7!8%lSZ8s}7^jG19YZDd}4z!9miizHF9+6&G&TL73GCek zdCFi?yM~RdtMkzzi(IXrBatYekn=b~h{-r*Q;8Ok2nQcTJs&3`qLMx1KaFNK1zxg3 z89f7-zj2%z79LJRrw1+HmL*dtI-7+&JzcnklUa*V)nH0GZ%I+!EY?$M&TZ<^0T#Ag zdG4yO4$A;1YH7KXK-bsTW8;(FBy>x82Jw3u3C%P1Jx|78G+E-Md2&&{U*k71-A3`Q zt*$EFmXD!ar3AZmdib&hhbnKwVj^!wTMpo$CEyfOb&MVgpCj%?R9CBpDZS1F5!P~O zw4zcZ+EfYICZ_Gv-t~WWmMk)SPUb?mMy^mEoeht71S^jB?{!7jARGp1C@Gle_@Bi% zN$202(NI@ce)9&Kfss*FSNBm+R%>`E!QJJAxHyKP)YQlQq<$ac!PskTa=N0`79}ZD z`$I}H>As?(SXS~;)~*+hGiY3wGlCDr0vL_%EeiTqZuO z@VUBpa`D3%-r326-1q)tlG1w84XZhiKeKCyi~7; z4A0(owAVd*C!TUinif6sfo|L$`sA@u=o>9&F|iD=MkM`TP>fB5Vocle1AG+DSjNAD zLCwu=ULB5YTPMN#pQO0NOD#-tTT-V-A5Uj*_lM6`Nt@YyMly18xx(39x4sInuxKAZ kfD5T;^=V)hL-q{;$=5?TN0Sp18u)o5tteG4@h\n", + " \"image\"\n", + "

\n", + "\n", + "Now add the following components to your training pipeline code:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "@dsl.component(base_image=\"python:3.9\")\n", + "def get_worker_pool_specs(\n", + " input_data: Input[Dataset],\n", + " arguments: dict,\n", + ") -> list:\n", + " \"\"\"\n", + " Generate a specification for worker pools to perform hyperparameter tuning.\n", + "\n", + " Args:\n", + " input_data (Input[Dataset]): Input data for training\n", + " hparams (dict): Hyperparameters\n", + "\n", + " Returns:\n", + " list: A list of worker pool specifications, each specifying the machine type,\n", + " Docker image, and command arguments for a worker pool.\n", + "\n", + " \"\"\"\n", + "\n", + " COMMAND = [\"python\"]\n", + " ARGS = [\n", + " \"-m\", \"training\",\n", + " \"--input_path\", input_data.path,\n", + " ]\n", + " for key, value in arguments.items():\n", + " ARGS.extend([\"--\" + str(key), str(value)])\n", + "\n", + " return [{\n", + " \"machine_spec\": {\n", + " \"machine_type\": \"n1-standard-4\",\n", + " },\n", + " \"replica_count\": 1,\n", + " \"container_spec\": {\n", + " \"image_uri\": TRAINING_IMAGE,\n", + " \"command\": COMMAND,\n", + " \"args\": ARGS,\n", + " },\n", + " }]" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "@dsl.component(\n", + " packages_to_install=[\n", + " \"google-cloud-aiplatform\",\n", + " \"google-cloud-pipeline-components\",\n", + " \"protobuf\",\n", + " ],\n", + " base_image=\"python:3.9\",\n", + ")\n", + "def get_trials(gcp_resources: str) -> list:\n", + " \"\"\"Retrieve all trials after a successful hyperparameters tuning job.\n", + "\n", + " Args:\n", + " gcp_resources (str): Proto tracking the hyperparameter tuning job.\n", + "\n", + " Returns:\n", + " List of strings representing the intermediate JSON representation of the\n", + " trials from the hyperparameter tuning job.\n", + " \"\"\"\n", + " from google.cloud import aiplatform\n", + " from google_cloud_pipeline_components.proto.gcp_resources_pb2 import GcpResources\n", + " from google.protobuf.json_format import Parse\n", + " from google.cloud.aiplatform_v1.types import study\n", + "\n", + " api_endpoint_suffix = \"-aiplatform.googleapis.com\"\n", + " gcp_resources_proto = Parse(gcp_resources, GcpResources())\n", + " gcp_resources_split = gcp_resources_proto.resources[0].resource_uri.partition(\n", + " \"projects\"\n", + " )\n", + " resource_name = gcp_resources_split[1] + gcp_resources_split[2]\n", + " prefix_str = gcp_resources_split[0]\n", + " prefix_str = prefix_str[: prefix_str.find(api_endpoint_suffix)]\n", + " api_endpoint = prefix_str[(prefix_str.rfind(\"//\") + 2) :] + api_endpoint_suffix\n", + "\n", + " client_options = {\"api_endpoint\": api_endpoint}\n", + " job_client = aiplatform.gapic.JobServiceClient(client_options=client_options)\n", + " response = job_client.get_hyperparameter_tuning_job(name=resource_name)\n", + "\n", + " return [study.Trial.to_json(trial) for trial in response.trials]" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "@dsl.component(packages_to_install=[\"google-cloud-aiplatform\"], base_image=\"python:3.9\")\n", + "def get_best_trial(trials: list, study_spec_metrics: list) -> str:\n", + " \"\"\"Retrieves the best trial from the trials.\n", + "\n", + " Args:\n", + " trials (list): Required. List representing the intermediate\n", + " JSON representation of the trials from the hyperparameter tuning job.\n", + " study_spec_metrics (list): Required. List serialized from dictionary\n", + " representing the metrics to optimize.\n", + " The dictionary key is the metric_id, which is reported by your training\n", + " job, and the dictionary value is the optimization goal of the metric\n", + " ('minimize' or 'maximize'). example:\n", + " metrics = hyperparameter_tuning_job.serialize_metrics(\n", + " {'loss': 'minimize', 'accuracy': 'maximize'})\n", + "\n", + " Returns:\n", + " String representing the intermediate JSON representation of the best\n", + " trial from the list of trials.\n", + "\n", + " Raises:\n", + " RuntimeError: If there are multiple metrics.\n", + " \"\"\"\n", + " from google.cloud.aiplatform_v1.types import study\n", + "\n", + " if len(study_spec_metrics) > 1:\n", + " raise RuntimeError(\n", + " \"Unable to determine best parameters for multi-objective\"\n", + " \" hyperparameter tuning.\"\n", + " )\n", + " trials_list = [study.Trial.from_json(trial) for trial in trials]\n", + " best_trial = None\n", + " goal = study_spec_metrics[0][\"goal\"]\n", + " best_fn = None\n", + " if goal == study.StudySpec.MetricSpec.GoalType.MAXIMIZE:\n", + " best_fn = max\n", + " elif goal == study.StudySpec.MetricSpec.GoalType.MINIMIZE:\n", + " best_fn = min\n", + " best_trial = best_fn(\n", + " trials_list, key=lambda trial: trial.final_measurement.metrics[0].value\n", + " )\n", + "\n", + " return study.Trial.to_json(best_trial)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "**Training pipeline:** In the define new variables and use the just implemented components by replacing the initial `train` component:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "@dsl.pipeline(name=\"...\")\n", + "def pipeline():\n", + "\n", + " # ...\n", + "\n", + " # define static arguments which will be used for all tuning jobs\n", + " # `learning_rate` is excluded as we'll optimise this hyperparameter\n", + " static_arguments = dict(\n", + " n_estimators=200,\n", + " label=label,\n", + " # ...\n", + " )\n", + " # the metric(s) we want to optimise for\n", + " metrics = hpt_job.serialize_metrics({\"rootMeanSquaredError\": \"minimize\"})\n", + " \n", + " # prepare your data ...\n", + " \n", + " data_op = # ...\n", + " \n", + " # now use the new components ...\n", + " \n", + " specs_op = get_worker_pool_specs(\n", + " input_data=data_op.outputs[\"data\"],\n", + " arguments=training_arguments,\n", + " )\n", + " \n", + " tuning_op = hpt_job_op(\n", + " display_name=\"tuning-job\",\n", + " project=project,\n", + " location=location,\n", + " worker_pool_specs=specs_op.output,\n", + " \n", + " # TODO: adjust to your use case\n", + " base_output_directory=\"...\", # e.g. a folder in your pipeline bucket root\n", + " max_trial_count=3, # e.g. run up to 3x trials\n", + " parallel_trial_count=3, # e.g. run all of them in parallel\n", + " study_spec_algorithm=\"ALGORITHM_UNSPECIFIED\",\n", + " study_spec_measurement_selection_type=\"BEST_MEASUREMENT\",\n", + " study_spec_metrics=metrics,\n", + " \n", + " # TODO: specify the hyperparameters to be tuned\n", + " study_spec_parameters=hpt_job.serialize_parameters({\n", + " \"learning_rate\": hpt.DoubleParameterSpec(min=0.001, max=1, scale=\"log\"),\n", + " }),\n", + " )\n", + " \n", + " trials_op = get_trials(gcp_resources=tuning_op.outputs[\"gcp_resources\"])\n", + " \n", + " best_trial_op = get_best_trial(\n", + " trials=trials_op.output, \n", + " study_spec_metrics=metrics,\n", + " )\n", + " \n", + " # TODO: upload the model with the best trial ..." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Run pipeline\n", + "\n", + "Vertex AI Pipelines uses KubeFlow to orchestrate your training steps, as such you'll need to:\n", + "\n", + "1. Compile the pipeline\n", + "2. Re-build the training container (as we've update the dependencies and code)\n", + "3. Run the pipeline in Vertex AI\n", + "\n", + "Don't worry about executing steps 1-3 manually (and each time you run your pipeline!), simply run the following command:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "! make run pipeline=training build=true targets=training" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "You've successfully updated your training pipeline to support hyperparameter tuning! 🎉 \n", + "\n", + "Continue by reading more about [hyperparameter tuning in Vertex AI](https://cloud.google.com/vertex-ai/docs/training/hyperparameter-tuning-overview) or by improving your new training pipeline by uploading the model with the best evaluation result." + ] + } + ], + "metadata": { + "colab": { + "collapsed_sections": [], + "name": "notebook_template.ipynb", + "toc_visible": true + }, + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.9.16" + }, + "vscode": { + "interpreter": { + "hash": "bb5c7b0035bb37e2e2e56e6840dfdd8f7fa070884ae8e041fbcae450545b1006" + } + } + }, + "nbformat": 4, + "nbformat_minor": 4 +} From 5c7b75afacd9443a97ce9ad684b4af4e0e3dccce Mon Sep 17 00:00:00 2001 From: Felix Schaumann Date: Mon, 13 Nov 2023 18:21:39 +0100 Subject: [PATCH 2/3] Minor improvements to notebook --- .../extras/01_hyperparameter_tuning.ipynb | 17 +++++++++++++++-- 1 file changed, 15 insertions(+), 2 deletions(-) diff --git a/docs/notebooks/extras/01_hyperparameter_tuning.ipynb b/docs/notebooks/extras/01_hyperparameter_tuning.ipynb index 7b3789f8..c04db9da 100644 --- a/docs/notebooks/extras/01_hyperparameter_tuning.ipynb +++ b/docs/notebooks/extras/01_hyperparameter_tuning.ipynb @@ -29,7 +29,7 @@ "source": [ "# Hyperparameter Tuning with Vertex AI Turbo Templates\n", "\n", - "This notebook guides you through production-ready pipelines on Google Cloud with hyperparameter tuning enabled. \n", + "This notebook guides you through production-ready pipelines on Google Cloud and how to enable hyperparameter tuning in your training pipeline. \n", "If you're new to the Vertex AI Turbo Template, start with the [three-part notebook series](../01_infrastructure_setup.ipynb) first. \n", "\n", "**Prerequisites:**\n", @@ -57,6 +57,12 @@ "source": [ "## Model training\n", "\n", + "Start by updating the model training code and it's dependencies by:\n", + "\n", + "- updating your Python dependencies\n", + "- extending your training code\n", + "- reporting evaluation metrics for each training trial\n", + "\n", "**Update Python dependencies:** Add the following line to `model/pyproject.toml` to install (cloudml-hypertune)[https://github.com/GoogleCloudPlatform/cloudml-hypertune] as part of your model training code:" ] }, @@ -114,7 +120,14 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "## Training pipeline" + "## Training pipeline\n", + "\n", + "Since your training code is updated now, let's navigate to the training pipeline and start adapting to train multiple models concurrently with different hyperparameters while optimising your primary evaluation metric. \n", + "This is done by:\n", + "\n", + "- adding new Python imports\n", + "- implementing new pipeline components\n", + "- replacing the training operation with a new set of component operations" ] }, { From 7a28ae2e87a226a038445cac5dd91bc59ef752e7 Mon Sep 17 00:00:00 2001 From: Felix Schaumann Date: Mon, 13 Nov 2023 18:25:43 +0100 Subject: [PATCH 3/3] Minor improvements to notebook --- .../notebooks/extras/01_hyperparameter_tuning.ipynb | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/docs/notebooks/extras/01_hyperparameter_tuning.ipynb b/docs/notebooks/extras/01_hyperparameter_tuning.ipynb index c04db9da..a19170f0 100644 --- a/docs/notebooks/extras/01_hyperparameter_tuning.ipynb +++ b/docs/notebooks/extras/01_hyperparameter_tuning.ipynb @@ -63,7 +63,7 @@ "- extending your training code\n", "- reporting evaluation metrics for each training trial\n", "\n", - "**Update Python dependencies:** Add the following line to `model/pyproject.toml` to install (cloudml-hypertune)[https://github.com/GoogleCloudPlatform/cloudml-hypertune] as part of your model training code:" + "**Update Python dependencies:** Add the following line to `model/pyproject.toml` to install [cloudml-hypertune](https://github.com/GoogleCloudPlatform/cloudml-hypertune) as part of your model training code:" ] }, { @@ -79,8 +79,8 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "**Extend training code:** Add hyperparameters which require tuning to the expected command-line arguments `model/training/train.py`.\n", - "In this example we're adding the `learning_rate` hyperparamter:" + "**Extend training code:** Add hyperparameters which require tuning to the expected command-line arguments in `model/training/train.py`.\n", + "In this example we're adding the `learning_rate` hyperparameter:" ] }, { @@ -112,7 +112,7 @@ "hpt = hypertune.HyperTune()\n", "hpt.report_hyperparameter_tuning_metric(\n", " hyperparameter_metric_tag=\"rootMeanSquaredError\",\n", - " metric_value=hp_metric,\n", + " metric_value=calculated_metric,\n", ")" ] }, @@ -122,7 +122,8 @@ "source": [ "## Training pipeline\n", "\n", - "Since your training code is updated now, let's navigate to the training pipeline and start adapting to train multiple models concurrently with different hyperparameters while optimising your primary evaluation metric. \n", + "Since your training code is updated now, let's navigate to the training pipeline and adapt it. \n", + "You'll train multiple models concurrently with different hyperparameters while optimising for your primary evaluation metric. \n", "This is done by:\n", "\n", "- adding new Python imports\n", @@ -313,7 +314,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "**Training pipeline:** In the define new variables and use the just implemented components by replacing the initial `train` component:" + "**Training pipeline:** In the training pipeline define new variables and replace the original training component with the newly implemented components." ] }, {