Skip to content

DMWM Packaging with PyPi

Jump to bottom
Erik Gough edited this page Apr 26, 2019 · 19 revisions

This document outlines the transition of DMWM software to the Python Package Index, https://pypi.org/

General Information

DMWM PyPi account: cms-oc-dmwm

Erik Gough and Andrew Melo have the password if needed.

General docs for packaging projects: https://packaging.python.org/tutorials/packaging-projects/

Test packages location: https://test.pypi.org/user/cms-oc-dmwm/

Production packages location: https://www.pypi.org/user/cms-oc-dmwm/

Create a virtualenv for building packages. Need setuptools and twine.

$ virtualenv pypibuild
$ source pypibuild/bin/activate
(pypibuild) $ pip install twine

Packaging and uploading WMAgent to pypi

Clone the WMCore repo and fetch the set of tags

$ git clone https://github.com/dmwm/WMCore.git
$ cd WMCore
$ git fetch && git fetch --tags

Checkout what tag you want

$ git checkout <tag>

Create a source tarball for distribution

(pypibuild) $ python setup.py sdist

Upload to test pypi using twine (username and password required)

(pypibuild) $ twine upload --repository-url https://test.pypi.org/legacy/ dist/wmagent-<tag>.tar.gz

Upload to production pypi using twine (username and password required)

(pypibuild) $ twine upload dist/wmagent-<tag>.tar.gz

WMAgent Dependencies

This is the list of dependencies that are required before one can install wmagent via pip. Python dependencies are automatically pulled when the wmagent package is installed.

  • MariaDB
  • CouchDB
  • Oracle Client
  • Condor
  • libcurl-devel
  • jemalloc
  • libuuid
  • yui (might not be necessary)
  • pystack (might not be necessary)
  • gcc

If any of these packages are installed in a non-standard location, you will need to set some environment variables as well.

  • Add mariadb bin and gcc bin directory to PATH
  • Add gcc lib directory to LD_LIBRARY_PATH
  • Set ORACLE_HOME to Oracle client location

Download and install WMAgent

Create a wmagent virtualenv

$ virtualenv wmagent
$ source wmagent/bin/activate
(wmagent) $ pip install wmagent

Packaging DBS Applications

Instructions on how to build and upload DBS Client and Pycurl Client to PyPi.

One time setup for users wanting to upload new releases to PyPi

Create a .pypirc file in your home directory with the following contents:

[distutils]
index-servers =
  pypi

[pypi]
repository: https://upload.pypi.org/legacy/
username: cms-oc-dmwm

Changes to be made at release time

The following changes should be made when a new version of DBS is released. Some version numbers will have to be updated manually for now. This will (hopefully) be automated in the future.

  1. dbs-client version in setup_dbs_client.py, L10
  2. pycurl-client version in setup_pycurl_client.py, L10
  3. pycurl-client version in requirements.dbs_client.py, L1

Build and upload to PyPi

Clone the DBS repo and checkout the latest tag:

$ git clone https://github.com/dmwm/DBS.git
$ git fetch && git fetch --tags
$ git checkout <tag>

Run the build_pypi_packages.sh script build and upload the packages. The script takes the following options, but you will usually want to build "all".

  all              Build all DBS packages
  dbs-client       Build dbs-client package
  pycurl-client    Build pycurl-client package

$ sh Tools/build_pypi_packages.sh all

Enter the cms-oc-dmwm account password when prompted. This can also be set in the .pypirc file if you don't mind storing passwords in clear text.

Check the details of the new build here: https://pypi.org/user/cms-oc-dmwm/

Installing DBS client and Pycurl client via PyPI on lxplus7

Source a CMSSW environment.

[user@lxplusXXX ~]$ export SCRAM_ARCH=<arch>
[user@lxplusXXX ~]$ cmsrel CMSSW_X_Y_Z
[user@lxplusXXX ~]$ cd CMSSW_X_Y_Z/src
[user@lxplusXXX src]$ cmsenv

To use DBS client, one needs to install it via pip and then activate a CMSSW envrionment. Using DBS client with user certificates and proxies requires a curl shared library built with special configuration options. This curl library is pulled in when setting the CMSSW environment. Using virtual python environments (virtualenv) in conjunction with the CMSSW environment gets messy real quick, so it is best to just install DBS client into your home directory with the --user option.

[user@lxplusXXX ~]$ pip install --user dbs-client
Collecting dbs-client
<snip>
Successfully installed dbs-client-3.7.8 pycurl-7.19.0.3 pycurl-client-3.7.8 python-cjson-1.2.1

Create a proxy

[user@lxplusXXX ~]$ voms-proxy-init -rfc -voms cms

Test access with a simple python script

[user@lxplusXXX ~]$ cat dbs-test.py
from dbs.apis.dbsClient import DbsApi
dbs = DbsApi('https://cmsweb.cern.ch/dbs/prod/global/DBSReader')
ds = dbs.listDatasets(dataset = '/MinBias/Summer*/*')
print(ds)

[user@lxplusXXX ~]$ python dbs-test.py 
[{'dataset': '/MinBias/Summer09-MC_31X_V3-v1_HLT_AlCa1E31-v1/RAW'}, {'dataset': '/MinBias/Summer09-MC_31X_V3-v1_HLT_AlCa8E29-v1/RAW'}, {'dataset': '/MinBias/Summer09-MC_31X_V3_CS_DiJetAve15U-v1/USER'}, {'dataset': '/MinBias/Summer09-MC_31X_V3_7TeV_CS_DiJetAve15U-v1/USER'}, {'dataset': '/MinBias/Summer11-DESIGN42_V11_428_SLHC1-v1/GEN-SIM'}, {'dataset': '/MinBias/Summer11-DESIGN42_V11_428_SLHChcal-v1/GEN-SIM'}]
Clone this wiki locally