Merge pull request #84 from hddm-devs/rlssm_dev

Rlssm dev

CONTRIBUTORS.rst

@@ -2,8 +2,13 @@ The following people have contributed to HDDM:
 
 * Thomas Wiecki
 * Imri Sofer
+* Alexander Fengler
+* Lakshmi Govindarajan
+* Krishn Bera
 * Michael J. Frank
 * Guido Biele
 * Øystein Sandvik
 * Mads Pedersen
-* Alex Fengler
+
+
+

LICENSE

@@ -1,4 +1,4 @@
-Copyright (c) 2021, Thomas V. Wiecki, Imri Sofer, Mads Lund Pederson, Alexander Fengler, Michael J. Frank, Brown University 
+Copyright (c) 2021, Thomas V. Wiecki, Imri Sofer, Mads Lund Pederson, Alexander Fengler, Lakshmi Govindarajan, Krishn Bera, Michael J. Frank, Brown University 
 All rights reserved.
 
 Redistribution and use in source and binary forms, with or without

README.rst

@@ -2,8 +2,8 @@
 Introduction
 ************
 
-:Author: Thomas V. Wiecki, Imri Sofer, Mads L. Pedersen, Alexander Fengler, Michael J. Frank
-:Contact: [email protected], [email protected], [email protected], [email protected], [email protected]
+:Author: Thomas V. Wiecki, Imri Sofer, Mads L. Pedersen, Alexander Fengler, Lakshmi Govindarajan, Krishn Bera, Michael J. Frank
+:Contact: [email protected], [email protected], [email protected], [email protected], [email protected], [email protected]
 :Web site: https://hddm.readthedocs.io
 :Github: http://github.com/hddm-devs/hddm
 :Mailing list: https://groups.google.com/group/hddm-users/
@@ -105,6 +105,17 @@ Features
              Note also that the usage of **indirect betas** as well as **indirect regressors** may affect the speed of sampling in general.
              Both translate into more computational work at the stage of regression likelihood evaluation.
 
+* HDDM 0.9.6 brings a host of new features. 
+             HDDM now includes use of `likelihood approximation networks`_ in conjunction with reinforcement learning models via the **HDDMnnRL** class. 
+             This allows researchers to study not only the across-trial dynamics of learning but the within-trial dynamics of choice processes, using a single model. 
+             This module greatly extends the previous functionality for fitting RL+DDM models (via HDDMrl class) by allowing fitting of a number of variants of sequential sampling models in conjuction with a learning process (RL+SSM models).
+
+             We have included a new **simulator**, which allows data generation for a host of variants of sequential sampling models 
+              in conjunction with the Rescorla-Wagner update rule on a 2-armed bandit task environment.
+             There are some new, out-of-the-box **plots** and **utility function** in the **hddm.plotting** and **hddm.utils** modules, respectively, to facilitate posterior visualization and posterior predictive checks.
+             Lastly you can also save and load **HDDMnnRL** models. 
+             Please see the **documentation** (under **HDDMnnRL Extension**) for illustrations on how to use the new features.
+
 
 Comparison to other packages
 ============================
@@ -211,6 +222,11 @@ If HDDM was used in your research, please cite the publication_:
 Wiecki TV, Sofer I and Frank MJ (2013). HDDM: Hierarchical Bayesian estimation of the Drift-Diffusion Model in Python.
 Front. Neuroinform. 7:14. doi: 10.3389/fninf.2013.00014
 
+If you use the HDDMnn, HDDMnnRegressor, HDDMnnStimCoding or HDDMnnRL class, please cite the publication2_:
+
+Alexander Fengler, Lakshmi N Govindarajan, Tony Chen, Michael J Frank (2021). Likelihood approximation networks (LANs) for fast inference of simulation models in cognitive neuroscience.
+eLife 10:e65074. doi: 10.7554/eLife.65074
+
 Published papers using HDDM
 ===========================
 
@@ -245,5 +261,6 @@ Join our low-traffic `mailing list`_.
 .. _SciPy Superpack: http://fonnesbeck.github.com/ScipySuperpack/
 .. _Anaconda: http://docs.continuum.io/anaconda/install.html
 .. _publication: http://www.frontiersin.org/Journal/10.3389/fninf.2013.00014/abstract
+.. _publication2: https://elifesciences.org/articles/65074
 .. _published papers: https://scholar.google.com/scholar?oi=bibs&hl=en&cites=17737314623978403194
 .. _thread: https://groups.google.com/forum/#!topic/hddm-users/bdQXewfUzLs

docs/source/conf.py

@@ -70,16 +70,16 @@
 
 # General information about the project.
 project = u'HDDM'
-copyright = u'2021, Thomas V. Wiecki, Mads Lund Pedersen, Alexander Fengler, Michael J. Frank, Brown University'
+copyright = u'2022, Thomas V. Wiecki, Mads Lund Pedersen, Alexander Fengler, Krishn Bera, Michael J. Frank, Brown University'
 
 # The version info for the project you're documenting, acts as replacement for
 # |version| and |release|, also used in various other places throughout the
 # built documents.
 #
 # The short X.Y version.
-version = '0.9.5'
+version = '0.9.6'
 # The full version, including alpha/beta/rc tags.
-release = '0.9.5'
+release = '0.9.6'
 
 # The language for content autogenerated by Sphinx. Refer to documentation
 # for a list of supported languages.

docs/source/demo_HDDMnnRL.rst

@@ -0,0 +1,223 @@
+.. index:: Demos
+.. _chap_rl:
+
+Tutorial for analyzing instrumental learning data with the HDDMnnRL module
+==========================================================================
+
+.. code:: ipython3
+
+    import hddm
+    import pickle
+    import pandas as pd
+
+Load the data
+^^^^^^^^^^^^^
+
+.. code:: ipython3
+
+    with open('./angle_d1_c3_s20_t500.pickle', 'rb') as handle:
+        datafile = pickle.load(handle)
+    
+    # Here, datafile is saved as a list of datasets. We pick the first dataset.
+    dataset = datafile[0]
+
+
+.. parsed-literal::
+
+    1
+
+
+.. code:: ipython3
+
+    # Reformat the dataset as a dataframe
+    data = hddm.utils.get_dataset_as_dataframe_rlssm(dataset)
+
+Initialize the HDDMnnRL model and sample
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+.. code:: ipython3
+
+    # Specify number of samples and burnins
+    nsamples = 100
+    nburn = 50
+
+.. code:: ipython3
+
+    m = hddm.HDDMnnRL(data, model='angle', rl_rule='RWupdate', non_centered=True, include=['z', 'theta', 'rl_alpha'], p_outlier = 0.0)
+    m.sample(nsamples, burn=nburn, dbname='traces.db', db='pickle')
+
+
+.. parsed-literal::
+
+    
+    Printing model specifications -- 
+    ssm:  angle
+    rl rule:  RWupdate
+    using non-centered dist.:  False
+    Using default priors: Uninformative
+    Supplied model_config specifies params_std_upper for  z as  None.
+    Changed to 10
+    Supplied model_config specifies params_std_upper for  rl_alpha as  None.
+    Changed to 10
+     [-----------------101%-----------------] 101 of 100 complete in 256.1 sec
+
+
+
+.. parsed-literal::
+
+    <pymc.MCMC.MCMC at 0x7f5b5c224f10>
+
+
+
+Save the model
+^^^^^^^^^^^^^^
+
+.. code:: ipython3
+
+    # Save the model
+    m.save('rlssm_model')
+
+.. code:: ipython3
+
+    # Load the model
+    # model = hddm.load('rlssm_model')
+
+Check the posterior results
+^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+.. code:: ipython3
+
+    m.plot_posteriors()
+
+
+.. parsed-literal::
+
+    Plotting v
+    Plotting v_std
+    Plotting a
+    Plotting a_std
+    Plotting z
+    Plotting z_std
+    Plotting t
+    Plotting t_std
+    Plotting theta
+    Plotting theta_std
+    Plotting rl_alpha
+    Plotting rl_alpha_std
+
+
+
+.. image:: demo_HDDMnnRL_files/demo_HDDMnnRL_12_1.png
+
+
+
+.. image:: demo_HDDMnnRL_files/demo_HDDMnnRL_12_2.png
+
+
+
+.. image:: demo_HDDMnnRL_files/demo_HDDMnnRL_12_3.png
+
+
+
+.. image:: demo_HDDMnnRL_files/demo_HDDMnnRL_12_4.png
+
+
+
+.. image:: demo_HDDMnnRL_files/demo_HDDMnnRL_12_5.png
+
+
+
+.. image:: demo_HDDMnnRL_files/demo_HDDMnnRL_12_6.png
+
+
+
+.. image:: demo_HDDMnnRL_files/demo_HDDMnnRL_12_7.png
+
+
+
+.. image:: demo_HDDMnnRL_files/demo_HDDMnnRL_12_8.png
+
+
+
+.. image:: demo_HDDMnnRL_files/demo_HDDMnnRL_12_9.png
+
+
+
+.. image:: demo_HDDMnnRL_files/demo_HDDMnnRL_12_10.png
+
+
+
+.. image:: demo_HDDMnnRL_files/demo_HDDMnnRL_12_11.png
+
+
+
+.. image:: demo_HDDMnnRL_files/demo_HDDMnnRL_12_12.png
+
+
+.. code:: ipython3
+
+    # Load the trace
+    with open('./traces.db', 'rb') as handle:
+        tracefile = pickle.load(handle)
+
+.. code:: ipython3
+
+    # Re-format traces as a dataframe
+    traces = hddm.utils.get_traces_rlssm(tracefile)
+
+.. code:: ipython3
+
+    model_ssm = 'angle'
+    model_rl = 'RWupdate'
+    
+    config_ssm = hddm.model_config.model_config[model_ssm]
+    config_rl = hddm.model_config_rl.model_config_rl[model_rl]
+
+.. code:: ipython3
+
+    hddm.plotting.plot_posterior_pairs_rlssm(tracefile, config_ssm['params'] + config_rl['params'])
+
+Posterior Predictive Checks
+^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+.. code:: ipython3
+
+    num_posterior_samples = 3
+    p_lower = {0: 0.15, 1:0.30, 2:0.45}
+    p_upper = {0: 0.85, 1:0.70, 2:0.55}
+    ppc_sdata = hddm.plotting.gen_ppc_rlssm(model_ssm, config_ssm, model_rl, config_rl, data, traces, num_posterior_samples, p_lower, p_upper, save_data=True, save_name='ppc_data')
+
+
+.. parsed-literal::
+
+    100%|██████████| 3/3 [05:49<00:00, 116.55s/it]
+
+
+.. parsed-literal::
+
+    ppc data saved at ./ppc_data.csv
+
+
+.. code:: ipython3
+
+    # Load the saved ppc data
+    # ppc_sdata = pd.read_csv('./ppc_data.csv')
+
+.. code:: ipython3
+
+    _ = hddm.plotting.plot_ppc_choice_rlssm(data, ppc_sdata, 40, 10)
+
+
+
+.. image:: demo_HDDMnnRL_files/demo_HDDMnnRL_20_0.png
+
+
+.. code:: ipython3
+
+    _ = hddm.plotting.plot_ppc_rt_rlssm(data, ppc_sdata, 40, 0.06)
+
+
+
+.. image:: demo_HDDMnnRL_files/demo_HDDMnnRL_21_0.png
+
+

docs/source/demos.rst

@@ -9,4 +9,5 @@ Demos
     :maxdepth: 2
 
     demo_gonogo
-    demo_RLHDDMtutorial
+    demo_RLHDDMtutorial
+    demo_HDDMnnRL

hddm/__init__.py

@@ -2,7 +2,7 @@
 
 __docformat__ = "restructuredtext"
 
-__version__ = "0.9.5"
+__version__ = "0.9.6"
 
 from . import simulators
 from . import likelihoods
@@ -20,6 +20,7 @@
 from . import network_inspectors
 from . import models
 from . import model_config
+from . import model_config_rl
 import cdfdif_wrapper
 
 from .models import *