This python-based app (pygrams.py) is designed to extract popular or emergent n-grams/terms (words or short phrases) from free text within a large (>1,000) corpus of documents. Example corpora of granted patent document abstracts are included for testing purposes.
The app pipeline (more details in the user option section):
- Input Text Data Text data can be input by several text document types (ie. csv, xls, pickled python dataframes, etc)
- TFIDF Dictionary This is the processed list of terms (ngrams) out of the whole corpus. These terms are the columns of the TFIDF sparse matrix. The user can control the following parameters: minimum document frequency, stopwords, ngram range.
- TFIDF Computation Grab a coffee if your text corpus is long (>1 million docs) :)
- Filters These are filters to use on the computed TFIDF matrix. They consist of document filters and term filters
- Document Filters These filters work on document level. Examples are: date range, column features (eg. cpc classification).
- Term Filters These filters work on term level. Examples are: search terms list (eg. pharmacy, medicine, chemist)
- Mask the TFIDF Matrix Apply the filters to the TFIDF matrix
- Emergence
- Emergence Calculations Options include Porter 2018 emergence calculations, curve fitting, or calculations designed to favour exponential like emergence.
- Emergence Forecasts Options include ARIMA, linear and quadratic regression, Holt-Winters, state-space models.
- Outputs The default 'report' output is a ranked and scored list of 'popular' ngrams or emergent ones if selected. Other outputs include a word cloud and an html document as emergence report.
Ensure that docker is installed on your machine.
Navigate to root directory of the project and build the docker image
docker build -t pygrams
-t - tags the image
pygrams - image tag
The latest version of pyGrams has been added to docker.io at https://hub.docker.com/r/datasciencecampus/pygrams. To use this:
docker pull datasciencecampus/pygrams
Run the built or pulled docker image using
docker run pygrams
If you would like to pass parameters when running the program as described in User guide, append the parameters at the end of docker run:
docker run pygrams -mn=1 -mx=3
Have a look at User guide for further runtime parameters.
pyGrams.py has been developed to work on both Windows and MacOS. To install:
-
Please make sure Python 3.6 is installed and set in your path.
To check the Python version default for your system, run the following in command line/terminal:
python --versionNote: If Python 2.x is the default Python version, but you have installed Python 3.x, your path may be setup to use
python3instead ofpython. -
To install pyGrams packages and dependencies, from the root directory (./pyGrams) run:
pip install -e .This will install all the libraries and then download their required datasets (namely NLTK's data). Once installed, setup will run some tests. If the tests pass, the app is ready to run. If any of the tests fail, open a GitHub issue here.
The system performance was tested using a 2.7GHz Intel Core i7 16GB MacBook Pro using 3.2M US patent abstracts from approximately 2005 to 2018. Indicatively, it initially takes about 6 hours to produce a specially optimised 100,000 term TFIDF Dictionary with a file size under 100MB. Once this is created however, it takes approximately 1 minute to run a pyGrams popular terminology query, or approximately 7 minutes for an emerging terminology query.
pyGrams is command line driven, and called in the following manner:
python pygrams.py
This argument is used to select the corpus of documents to analyse. The default source is a pre-created random 1,000 patent dataset from the USPTO, USPTO-random-1000.pkl.bz2.
Pre-created datasets of 100, 1,000, 10,000, 100,000, and 500,000 patents are available in the ./data folder:
USPTO-random-100.pkl.bz2USPTO-random-1000.pkl.bz2USPTO-random-10000.pkl.bz2USPTO-random-100000.pkl.bz2USPTO-random-500000.pkl.bz2
For example, to load the 10,000 pickled dataset for patents, use:
python pygrams.py -ds=USPTO-random-10000.pkl.bz2
To use your own document dataset, please place in the ./data folder of pyGrams. File types currently supported are:
- pkl.bz2: compressed pickle file containing a dataset
- xlsx: new Microsoft excel format
- xls: old Microsoft excel format
- csv: comma separated value file (with headers)
Datasets should contain the following columns:
| Column | Required? | Comments |
|---|---|---|
| Free text field | Yes | Terms extracted from here |
| Date | Optional | Compulsory for emergence |
| Other headers | Optional | Can filter by content |
When loading a document dataset, it is mandatory to provide the date column header name, with the text heading having an assumed default:
-th: free text field column ('text heading'; default is 'abstract')-dh: date column ('date heading'; default is None as pyGrams can also work as a keywords extraction only without timeseries analysis), format of dates is 'YYYY/MM/DD'
For example, for a corpus of book blurbs you could use:
python pygrams.py -th='blurb' -dh='published_date'
In order save processing time, at various stages of the pipeline, we cache data structures that are costly and slow to compute, like the compressed tf-idf matrix, the timeseries matrix, the smooth series and its derivatives from kalman filter and others:
python pygrams.py -uc all-mdf-0.05-200501-201841
An n-gram is a contiguous sequence of n items (source). N-grams can be unigrams (single words, e.g., vehicle), bigrams (sequences of two words, e.g., aerial vehicle), trigrams (sequences of three words, e.g., unmanned aerial vehicle) or any n number of continuous terms.
The following arguments will set the n-gram limit to be, e.g. unigrams, bigrams, and trigrams (the default):
python pygrams.py -mn=1 -mx=3
To analyse only unigrams:
python pygrams.py -mn=1 -mx=1
Terms identified are filtered by the maximum number of documents that use this term; the default is 0.05, representing an upper limit of 5% of documents containing this term. If a term occurs in more that 5% of documents it is rejected.
For example, to set the maximum document frequency to 5% (the default), use:
python pygrams.py -mdf 0.05
Using a small (5% or less) maximum document frequency may help remove generic words, or stop words.
There are three configuration files available inside the config directory:
- stopwords_glob.txt
- stopwords_n.txt
- stopwords_uni.txt
The first file (stopwords_glob.txt) contains stopwords that are applied to all n-grams. The second file contains stopwords that are applied to all n-grams for n > 1 (bigrams and trigrams). The last file (stopwords_uni.txt) contains stopwords that apply only to unigrams. The users can append stopwords into this files, to stop undesirable output terms.
Given that many of the terms will actually be very rare, they will not be of use when looking for popular terms. The total number of terms can easily exceed 1,000,000 and slow down pygrams with irrelevant terms. To circumvent this, a prefilter is applied as soon as the TFIDF matrix is created which will retain the highest scoring terms by TFIDF (as is calculated and reported at the end of the main pipeline). The default is to retain the top 100,000 terms; setting it to 0 will disable it, viz:
python pygrams.py -pt 0
Or changed to a different threshold such as 10,000 terms (using the longer argument name for comparison):
python pygrams.py -prefilter_terms 10000
Note that the prefilter will change TFIDF results as it will remove rare n-grams - which will result in bi-grams & tri-grams having increased scores when rare uni-grams and bi-grams are removed, as we unbias results to avoid double or triple counting contained n-grams.
This argument can be used to filter documents to a certain timeframe. For example, the below will restrict the document cohort to only those from 20 Feb 2000 up to now (the default start date being 1 Jan 1900).
python pygrams.py -dh publication_date -df=2000/02/20
The following will restrict the document cohort to only those between 1 March 2000 and 31 July 2016.
python pygrams.py -dh publication_date -df=2000/03/01 -dt=2016/07/31
If you want to filter results, such as for female, British in the example below, you can specify the column names you wish to filter by, and the type of filter you want to apply, using:
-fh: the list of column names (default is None)-fb: the type of filter (choices are'union'(default), where all fields need to be 'yes', or'intersection', where any field can be 'yes')
python pygrams.py -fh=['female','british'] -fb='union'
This filter assumes that values are '0'/'1', or 'Yes'/'No'.
This subsets the chosen patents dataset to a particular Cooperative Patent Classification (CPC) class, for example Y02. The Y02 classification is for "technologies or applications for mitigation or adaptation against climate change". An example script is:
python pygrams.py -cpc=Y02 -ds=USPTO-random-10000.pkl.bz2
In the console the number of subset patents will be stated. For example, for python pygrams.py -cpc=Y02 -ps=USPTO-random-10000.pkl.bz2 the number of Y02 patents is 197. Thus, the TFIDF will be run for 197 patents.
This subsets the TFIDF term dictionary by only keeping terms related to the given search terms.
python pygrams.py -st pharmacy medicine chemist
An option to choose between popular or emergent terminology outputs. Popular terminology is the default option; emergent terminology can be used by typing:
python pygrams.py -ts
An option to choose between quadratic fitting, Porter 2018 or gradients from state-space model using kalman filter smoothing emergence indexes. Porter is used by default; quadratic fitting can be used instead, for example:
python pygrams.py -ts -ei quadratic
An option designed to favour exponential like emergence, based on a yearly weighting function that linearly increases from zero, for example:
python pygrams.py -ts -exp
Various options are available to control how emergence is forecasted.
The forecast method is selected using argument pns, in this case corresponding to Linear (2=default) and Holt-Winters (6).
Python pygrams.py -pns=2
Python pygrams.py -pns=6
The full list of options is included below, with multiple inputs are allowed.
- All options
- Naive
- Linear
- Quadratic
- Cubic
- ARIMA
- Holt-Winters
- SSM
number of terms to analyse (default: 25)
Python pygrams.py -nts=25
minimum number of patents per quarter referencing a term (default: 15)
Python pygrams.py -mpq=15
number of steps ahead to analyse for (default: 5)
Python pygrams.py -stp=5
analyse using test or not (default: False)
Python pygrams.py -tst=False
analyse using normalised patents counts or not (default: False)
Python pygrams.py -nrm=False
Pygrams outputs a report of top ranked terms (popular or emergent). Additional command line arguments provide alternative options, for example a word cloud.
python pygrams.py -o wordcloud
Time series analysis also supports a multiplot to present up to 30 terms time series (emergent and declining), output in the outputs/emergence folder:
python pygrams.py -ts -dh 'publication_date' -o multiplot
The output options generate:
- Report is a text file containing top n terms (default is 250 terms, see
-npfor more details) wordcloud: a word cloud containing top n terms (default is 250 terms, see-ndfor more details)
Note that all outputs are generated in the outputs subfolder. Below are some example outputs:
The report will output the top n number of terms (default is 250) and their associated TFIDF score. Below is an example for patent data, where only bigrams have been analysed.
| Term | TFIDF Score |
|---|---|
| 1. fuel cell | 2.143778 |
| 2. heat exchanger | 1.697166 |
| 3. exhaust gas | 1.496812 |
| 4. combustion engine | 1.480615 |
| 5. combustion chamber | 1.390726 |
| 6. energy storage | 1.302651 |
| 7. internal combustion | 1.108040 |
| 8. positive electrode | 1.100686 |
| 9. carbon dioxide | 1.092638 |
| 10. control unit | 1.069478 |
A wordcloud, or tag cloud, is a novel visual representation of text data, where words (tags) importance is shown with font size and colour. Here is a wordcloud using patent data. The greater the TFIDF score, the larger the font size of the term.
- pygrams.py is the main python program file in the root folder (Pygrams).
- README.md is this markdown readme file in the root folder
- pipeline.py in the scripts folder provides the main program sequence along with pygrams.py.
- The 'data' folder is where to place the source text data files.
- The 'outputs' folder contains all the program outputs.
- The 'config' folder contains the stop word configuration files.
- The setup file in the root folder, along with the meta folder, contain installation related files.
- The test folder contains unit tests.
A help function details the range and usage of these command line arguments:
python pygrams.py -h
The help output is included below. This starts with a summary of arguments:
usage: pygrams.py [-h] [-ds DOC_SOURCE] [-uc USE_CACHE] [-th TEXT_HEADER]
[-dh DATE_HEADER] [-fc FILTER_COLUMNS]
[-st SEARCH_TERMS [SEARCH_TERMS ...]]
[-stthresh SEARCH_TERMS_THRESHOLD] [-df DATE_FROM]
[-dt DATE_TO] [-tsdf TIMESERIES_DATE_FROM]
[-tsdt TIMESERIES_DATE_TO] [-mn {1,2,3}] [-mx {1,2,3}]
[-mdf MAX_DOCUMENT_FREQUENCY] [-ndl] [-pt PREFILTER_TERMS]
[-o [{wordcloud,multiplot} [{wordcloud,multiplot} ...]]]
[-on OUTPUTS_NAME] [-wt WORDCLOUD_TITLE] [-nltk NLTK_PATH]
[-np NUM_NGRAMS_REPORT] [-nd NUM_NGRAMS_WORDCLOUD]
[-nf NUM_NGRAMS_FDG] [-cpc CPC_CLASSIFICATION] [-ts]
[-pns PREDICTOR_NAMES [PREDICTOR_NAMES ...]] [-nts NTERMS]
[-mpq MINIMUM_PER_QUARTER] [-stp STEPS_AHEAD]
[-ei {porter,net-growth}] [-sma {kalman,savgol}] [-exp]
[-nrm]
extract popular n-grams (words or short phrases) from a corpus of documents
It continues with a detailed description of the arguments:
-h, --help show this help message and exit
-ds DOC_SOURCE, --doc_source DOC_SOURCE
the document source to process (default: USPTO-
random-1000.pkl.bz2)
-uc USE_CACHE, --use_cache USE_CACHE
Cache file to use, to speed up queries (default: None)
-th TEXT_HEADER, --text_header TEXT_HEADER
the column name for the free text (default: abstract)
-dh DATE_HEADER, --date_header DATE_HEADER
the column name for the date (default: None)
-fc FILTER_COLUMNS, --filter_columns FILTER_COLUMNS
list of columns with binary entries by which to filter
the rows (default: None)
-st SEARCH_TERMS [SEARCH_TERMS ...], --search_terms SEARCH_TERMS [SEARCH_TERMS ...]
Search terms filter: search terms to restrict the
tfidf dictionary. Outputs will be related to search
terms (default: [])
-stthresh SEARCH_TERMS_THRESHOLD, --search_terms_threshold SEARCH_TERMS_THRESHOLD
Provides the threshold of how related you want search
terms to be Values between 0 and 1: 0.8 is considered
high (default: 0.75)
-df DATE_FROM, --date_from DATE_FROM
The first date for the document cohort in YYYY/MM/DD
format (default: None)
-dt DATE_TO, --date_to DATE_TO
The last date for the document cohort in YYYY/MM/DD
format (default: None)
-tsdf TIMESERIES_DATE_FROM, --timeseries-date-from TIMESERIES_DATE_FROM
The first date for the document cohort in YYYY/MM/DD
format (default: None)
-tsdt TIMESERIES_DATE_TO, --timeseries-date-to TIMESERIES_DATE_TO
The last date for the document cohort in YYYY/MM/DD
format (default: None)
-mn {1,2,3}, --min_ngrams {1,2,3}
the minimum ngram value (default: 1)
-mx {1,2,3}, --max_ngrams {1,2,3}
the maximum ngram value (default: 3)
-mdf MAX_DOCUMENT_FREQUENCY, --max_document_frequency MAX_DOCUMENT_FREQUENCY
the maximum document frequency to contribute to TF/IDF
(default: 0.05)
-ndl, --normalize_doc_length
normalize tf-idf scores by document length (default:
False)
-pt PREFILTER_TERMS, --prefilter_terms PREFILTER_TERMS
Initially remove all but the top N terms by TFIDF
score before pickling initial TFIDF (removes 'noise'
terms before main processing pipeline starts)
(default: 100000)
-o [{wordcloud,multiplot} [{wordcloud,multiplot} ...]], --output [{wordcloud,multiplot} [{wordcloud,multiplot} ...]]
Note that this can be defined multiple times to get
more than one output. (default: [])
-on OUTPUTS_NAME, --outputs_name OUTPUTS_NAME
outputs filename (default: out)
-wt WORDCLOUD_TITLE, --wordcloud_title WORDCLOUD_TITLE
wordcloud title (default: Popular Terms)
-nltk NLTK_PATH, --nltk_path NLTK_PATH
custom path for NLTK data (default: None)
-np NUM_NGRAMS_REPORT, --num_ngrams_report NUM_NGRAMS_REPORT
number of ngrams to return for report (default: 250)
-nd NUM_NGRAMS_WORDCLOUD, --num_ngrams_wordcloud NUM_NGRAMS_WORDCLOUD
number of ngrams to return for wordcloud (default:
250)
-cpc CPC_CLASSIFICATION, --cpc_classification CPC_CLASSIFICATION
the desired cpc classification (for patents only)
(default: None)
-ts, --timeseries denote whether timeseries analysis should take place
(default: False)
-pns PREDICTOR_NAMES [PREDICTOR_NAMES ...], --predictor_names PREDICTOR_NAMES [PREDICTOR_NAMES ...]
0. All standard predictors, 1. Naive, 2. Linear, 3.
Quadratic, 4. Cubic, 5. ARIMA, 6. Holt-Winters, 7.
SSM; multiple inputs are allowed. (default: [2])
-nts NTERMS, --nterms NTERMS
number of terms to analyse (default: 25)
-mpq MINIMUM_PER_QUARTER, --minimum-per-quarter MINIMUM_PER_QUARTER
minimum number of patents per quarter referencing a
term (default: 15)
-stp STEPS_AHEAD, --steps_ahead STEPS_AHEAD
number of steps ahead to analyse for (default: 5)
-ei {porter,net-growth}, --emergence-index {porter,net-growth}
Emergence calculation to use (default: porter)
-sma {kalman,savgol}, --smoothing-alg {kalman,savgol}
Time series smoothing to use (default: savgol)
-exp, --exponential_fitting
analyse using exponential type fit or not (default:
False)
-nrm, --normalised analyse using normalised patents counts or not
(default: False)
Patent data was obtained from the United States Patent and Trademark Office (USPTO) through the Bulk Data Storage System (BDSS). In particular we used the Patent Grant Full Text Data/APS (JAN 1976 - PRESENT) dataset, using the data from 2004 onwards in XML 4.* format.
Sections of this code are based on scikit-learn sources.
Various 3rd party libraries are used in this project; these are listed on the dependencies page, whose contributions we gratefully acknowledge.