APIP

Latent Question Interpretation using a module that Adapts Parameters through Interpretation Policy

In this work we propose a training framework and a neural network architecture for learning multiple question interpretations during the question answering procedure.

_{Structure overview of integrated APSN module with DrQA. In this illustration number of interpretations is 2 and a sampled interpretation is 1.}

	Answers from the APIP model produced by inducing the value of a latent interpretation and other competitive methods, given the manually constructed document
D	Maria had a big lunch today. She ate a sandwich. Maria ate a salad with coffee. Finally, she wandered into a store and ate an ice cream.
Q	What did Maria eat for lunch ?
A	['sandwich', 'salad with coffee', 'ice cream']
APIP	['ice cream', 'sandwich', 'salad']
Other	Mnemon: ['a sandwich'], QANet: ['coffee'], R-Net, DrQA: ['a sandwich']
D	Parrot have learned how to reproduce human language. The Bird speaks Japanese now. In fact, the parrot speaks Russian too. And of course, british owner taught this bird how to speak English.
Q	What languages does parrot speak ?
A	['Japanese', 'Russian', 'English']
APIP	['Russian too', 'English', 'Japanese']
Other	Mnemon, R-Net, DrQA: ['Russian'], QANet: ['English']
D	German language is wide spread in Europe. Obviously, it is mainly spoken in Germany. Moreover, it is one of the used languges in Switzerland and Austria as well.
Q	In which countries German language is spoken ?
A	['Germany', 'Switzerland', 'Austria']
APIP	['Switzerland and Austria', 'Germany']
Other	Mnemon, R-Net, DrQA: ['Germany'], QANet: ['Switzerland and Austria']
D	Alice was listening to Beatles yesterday. It was a sunny day, and the song "Come Together" fitted perfectly. Indeed, that song was very cheerful and bright.
Q	What kind of song was Alisce listening ?
A	['Come Together', 'cheerful', 'bright']
APIP	['Come Together', 'very cheerful and bright']
Other	Mnemon, QANet, R-Net, DrQA: ['Come Together']
D	Bob keeps his postage marks in a case that is green colored. He have been collecting this marks since his childhood. The case is made of wood. The notable thing about it is that it is carved with waves.
Q	How does the Bob 's case look like ?
A	['wood', 'green', 'carved with waves']
APIP	['carved with waves', 'green colored', 'The case is made of wood']
Other	Mnemon, QANet, R-Net: ['wood'], DrQA: ['The case is made of wood']
D	Africa has varied array of wild animals. The giraffes, the world's tallest animal, inhabit Africa. Also African elephants live here. The world's fastest land mammal, the cheetah, lives in Africa too.
Q	What animals live in Africa ?
A	['giraffes', 'elephants', 'cheetah']
APIP	['African elephants', 'giraffes', 'cheetah']
Other	Mnemon: ['the cheetah'], QANet: ['African elephants'], R-Net: ['wild animals'], DrQA: ['giraffes']
D	It is well known that dry air is mainly made up of nitrogen (78.09%) and oxygen (20.95%). However, many of us could not imagine that the rest of dry air is made of argon, carbon dioxide and other trace gases (0.94%).
Q	What does dry air comprise ?
A	['nitrogen', 'oxygen', 'argon', 'carbon dioxide', 'trace gases']
APIP	['argon, carbon dioxide and other trace gases', 'nitrogen']
Other	Mnemon, QANet, R-Net,DrQA: ['nitrogen']
D	Manager was late for work and his boss was angry about it. It is because at first manager went to a bank. Then manager went to a friends house. Eventually, a manager went to the cafe.
Q	Where did manager go before the work ?
A	['bank', 'friend house', 'cafe']
APIP	['a friends house', 'the cafe']
Other	Mnemon, R-Net, DrQA: ['the cafe'], QANet: ['cafe']
D	Pulp Fiction is a an American crime film by Quentin Tarantino. In the movie Uma Thurman played Mia. Another main role was given to John Travolta. And lastly, Samuel Jackson also played in the movie and it elevated his career.
Q	Who took a part in the movie ?
A	['Thurman', 'Jackson', 'Travolta']
APIP	['Samuel Jackson', 'Uma Thurman', 'John Travolta']
Other	Mnemon: ['Uma Thurman played Mia. Another main role was given to John Travolta'], QANet, R-Net, DrQA: ['Samuel Jackson']
D	Town A is located 150 km away from town B. The towns are connected via rail system. A journey between these towns takes around 1 hour by train.
Q	How far are the two towns from each other ?
A	['150 km', '1 hour by train']
APIP	['150 km', 'around 1 hour by train', 'around 1 hour']
Other	Mnemon: ['150 km'], QANet, DrQA: ['1 hour'], R-Net: ['150 km away']

Tested on GeForce GTX Titan X.

Setup

GPU and CUDA 8 are required

Install

• Python >= 3.5
• PyTorch
• CuPy, pynvrtc
• Spacy 1.10.1
• Cython, Pandas, NumPy, Scikit-learn
• msgpack, tensorboardX, Matplotlib

Download

• the SQuAD
• GloVe word vectors
• Sent2vec with wiki_bigram.bin
• Use checkpoints init1, init2 to initialize models for training; m1, m2 for testing
• Comparison of APIP flavors: details and checkpoints

using bash download.sh

Preprocess Dataset

# prepare the data
python prepro.py

# prepare semi-supervised labels for question-answer pairs
python semisup_labels.py

# make sure CUDA lib path can be found, e.g.:
export LD_LIBRARY_PATH=/usr/local/cuda/lib64

Running Code

# training in semi-supervised VAE framework
python train.py -e 240 -bs 50 -rd init2 -rs best_model.pt -lr  0.0005 --pi_q_rnn pi_d_nqd --control_d sru_00_kconv5_gram_s_cos --critic_loss --n_actions 5 --vae --semisup --rl_start 80 --rl_tuning pg

# sample answers with induced values for interpretations
python interpret.py -bs 32 -rd m1 -rs best_model.pt --pi_q_rnn pi_d_nqd --control_d sru_00_kconv5_gram_s_cos --n_actions 5 --vae

# visualize document encodings with interpretation-adapted parameters
python tsne.py -bs 1 -rd m1 -rs best_model.pt --pi_q_rnn pi_d_nqd --control_d sru_00_kconv5_gram_s_cos --n_actions 5 --vae

# get scores for F1 Threshold(rho) experiments for APIP 
python f1_thresh_apip.py -bs 100 -rd m1 -rs best_model.pt -lr  0.00015 --pi_q_rnn pi_d_nqd --control_d sru_00_kconv5_gram_s_cos --critic_loss --n_actions 5 --vae

# produce answers for different values of interpretation on SQuAD dev set
python text_interpret.py -bs 100 -rd m1 -rs best_model.pt -lr  0.00015 --pi_q_rnn pi_d_nqd --control_d sru_00_kconv5_gram_s_cos --critic_loss --n_actions 5 --vae

# get answers from APIP on manual samples
python text_test.py -bs 100 -rd m1 -rs best_model.pt -lr  0.00015 --pi_q_rnn pi_d_nqd --control_d sru_00_kconv5_gram_s_cos --critic_loss --n_actions 5 --vae

_{Indices represent different documents, colors correspond to induced interpretations (interpretation marked with cross was chosen by the policy during testing)}

	Sample answers produced by inducing the value of a latent interpretation given D, Q from SQuAD
D	ITV Tyne Tees was based at City Road for over 40 years after its launch in January 1959. In 2005 it moved to a new facility on The Watermark business park next to the MetroCentre in Gateshead.
Q	Where did ITV Tyne Tees move in 2005?
A	[’a new facility’]
APIP	['The Watermark business park', 'Gateshead']
D	Research shows that student motivation and attitudes towards school are closely linked to student-teacher relationships. Enthusiastic teachers are particularly good at creating beneficial relations with their students.
Q	What type of relationships do enthusiastic teachers cause?
A	[’beneficial’]
APIP	['student-teacher', 'beneficial relations']
D	For Luther, also Christ’s life, when understood as an example, is nothing more than an illustration of the Ten Commandments, which a Christian should follow in his or her vocations on a daily basis.
Q	What should a Christian follow in his life?
A	[’Ten Commandments’]
APIP	['Ten Commandment', 'vocations on a daily basis']
D	It is believed that the civilization was later devastated by the spread of diseases from Europe, such as smallpox.
Q	What was believed to be the cause of devastation to the civilization?
A	[’spread of diseases from Europe’]
APIP	['smallpox', 'spread of diseases from Europe']

Credits

Author of the Document Reader model: @danqi

Pytorch implementation of DrQA with SRU cells @taolei87