searchlight_NaN

[max-kat]

Hi there!

I have some questions regarding the RSA_searchlight. Since I just started reading into RSA (+ fairly new to python) they might be very basic but please bear with me...

Current state:
Our experiment was divided into 4 runs. In each run participants saw up to 10 pictures of aesthetic stimuli, which they had to rate (how aes moving is it 1-7). Then they imagined the same stimuli and rated them again. I am interested if being aesthetically moved is the same between the two modalities (perception vs imagery).

I created a LSA GLM with all runs in it and got the t-maps for each subj. These maps are individually split between high_aes_moving, low_aes_moving and missing_data. Hence, we have an unbalanced design, which varies between and within participants. (some have e.g. 30 high and 27 low, others have 40 high and 20 low)

I followed your demo_searchlight and tried to do the analysis for one participants on 61 tmaps. The SL_RDM was easy to create with your tutorial; yet, I had some troubles with the categorial one

I created a condition_array, which had 1&0, 1 = high aes, 2=low aes, corresponding to the tmap index. shape (61,)
I had troubles creating the cat_rdm with your demo material, so I first tried it in a different way:

cat_rdm = pairwise_distances(condition_array[:, np.newaxis], metric='manhattan')
AES_MODEL = ModelFixed('categorial model', upper_tri(cat_rdm))
eval_results = evaluate_models_searchlight(SL_RDM, AES_MODEL, eval_fixed, method='spearman', n_jobs=15)

this resulted in the message: rdm1 and rdm2 have different nan positions

then I tried to do it in a different way with

category_rdm = rsatoolbox.rdm.rdms.get_categorical_rdm(condition_array, category_name='category')
eval_results = evaluate_models_searchlight(SL_RDM, AES_MODEL, eval_fixed, method='spearman', n_jobs=15)

this results in model should be an rsatoolbox.model.Model or a list of such objects

if I try:

AES_MODEL = ModelFixed('AES_MODEL', upper_tri(category_rdm.dissimilarities)) #had to use dissim, otherwise I get the error: shape
eval_results = evaluate_models_searchlight(SL_RDM, AES_MODEL, eval_fixed, method='spearman', n_jobs=15)

I get: rdm1 and rdm2 must be RDMs of equal shape

So I either have to deal with the NANs or covert the category_rdm to a model_object. I guess there is a simple solution for these problems but I am stuck for a few days now and could really need some advice... I would really appreciate if you could help me or point me into the right direction....

[HeikoSchuett]

Hi @max-kat,

I should note that the searchlight code might change in the near future as we have current pull request #253 & #292 that change them substantially.

For your current problem the solution should be simple though: ModelFixed can be based on an RDMs object, so AES_MODEL = ModelFixed('AES_MODEL', category_rdm) should work and this should result in a valid input for the searchlight. In fact, this is intended to be the usual input to the model constructors.

Please tell us if this works for you!

[max-kat]

Hi @HeikoSchuett,

thank you for the fast respondse! Yes, I managed to create the categorial RDM as well as got rid of the NaNs. But I have some difficulties understanding some of the steps involved in the RSA workflow.

let's assume I have a loop:

1. I create a dataset.object per person N (trials) x N (voxels) (each person has a different amount of trials per condition)
2. Since I am only interested in two conditions ( imagery high vs perception high) I subset the dataset.
3. Then I need to create the categorial RDM. Since N trials are imbalanced between conditions, I use:
   unbalanced_rdm = rsatoolbox.rdm.calc_rdm_unbalanced(subsetted_data, descriptor='conds')
4. Then I make the model for this person:
   model = rsatoolbox.model.ModelFixed('Aesthetic Model', unbalanced_rdm)
5. Then I need to create the searchlight:
   centers, neighbors = get_volume_searchlight(mask, radius=2, threshold=0.5) --> this should actually be the same for each participant since their tmaps have the same dimensions
6. Now I am stuck.... According to your demo, I need to run:
   eval_results = evaluate_models_searchlight(SL_RDM, my_categorial_model, eval_fixed, method='spearman', n_jobs=20)

But if I run this for every participant individually, how do I combine their results and calculate the inference? Or do I have to append each model for each participant (?) and then provide this list of models (model_list) as input. Also the models differ between each participant since they have different N of trials... --> So do I have to create the categorial_rdm for each participant, append them and then use ? from_partials(categorial_rdm_list)`

eval_results = evaluate_models_searchlight(SL_RDM, model_list, eval_fixed, method='spearman', n_jobs=20)

But if I want to generalize over more conditions, shouldn't I also change the eval_fixed? but the command does not take bootstrap as input...

Lastly, how do I get the voxels that are still significant after permutation testing as well as the test statistics?
Is it just with: eval_score = [np.float(e.evaluations) for e in eval_results] ??

Sorry, for bothering you with so many (probably basic) questions but I really want to use your toolbox and get a basic understanding of RSA and unfortunately there is no one in our institute, whom I could ask....

Thanks in advance!

All the best,
Max

[HeikoSchuett]

Ok, so I am somewhat confused here. Why do you have different models for the different subjects in your case? Usually, my expectation would be that you create one RDM for the expectation according some theory which is the basis of your model. The searchlight is then supposed to go through the brain with this RDM and find locations that have similar RDMs. Those RDMs in the brain will of course be different for each subject, but the model would be the same right?

[JasperVanDenBosch]

In order to use the model-based inference in the toolbox, the categories would have to be reflected in the RDM. So 4*conds by 4*conds, as the models are specified as RDMS. You could deal with the variance in number of samples by doing some aggregation step, i.e. if the least sampled category for a subject is (A) 3 samples, then for BCD take three random samples for instance, and create three full 4*conds RDMs. Your statistical power here is limited by the number of observations for condition A. That is unless you were to consider it a fixed-effects design.

[max-kat]

Thank you for the suggestion!
Thinking about that wouldn't it make more sense then to just use a contrast beta map per category instead of single trial beta maps?

Another idea:
Is it possible to run a searchlight individually for each person and then somehow average the results across participants? Does that make sense conceptually?

[JasperVanDenBosch]

Thank you for the suggestion! Thinking about that wouldn't it make more sense then to just use a contrast beta map per category instead of single trial beta maps?

That's also valid; (and probably easier) it just depends on where you want to use your variance. In the scenario I suggested you could potentially use the variance of the three repetitions to get the noise.

Another idea: Is it possible to run a searchlight individually for each person and then somehow average the results across participants? Does that make sense conceptually?

Absolutely and I'd recommend that in many cases. One the second level you'd just do your normal FDR or clustering threshold.

PS Let me add that in this specific case, you can't do both of the approaches together; if you have only one repetition per RDM then you can't do inference on this within the subject I think.

[jonathanwehnert]

This particular situation is not covered by the searchlight code we have right now. I will put in on our wishlist for the revision we are currently working on. Allowing different predictions for different subjects is probably a sensible request.

@HeikoSchuett have participant-specific models been targeted in any way (whether searchlight-compatible or not)?

In my design all participants learn the same 30 stimuli in 2 different learning conditions. Assignment of stimuli to learning conditions is counterbalanced, so one half of participants will learn stimuli 1-15 in condition A and 16-30 in condition B, and vice versa for the other half of participants. My model is based on individual stimuli, so I have an overall 30x30 model RDM, but only 15x15 neural RDMs for each condition within-participant. I inserted NaNs at the appropriate locations (via from_partials()), combining my 15x15 neural RDM with a fake 30x30 RDM full of NaNs) to turn the neural RDMs into 30x30 RDMs with columns matching the model RDM, but when using compare() (on its own or within eval_bootstrap_pattern()), this results in "rdm1 and rdm2 have different nan positions". --> how am I supposed to make use of from_partials() to combine the RDMs with differing conditions?

Ideally I would want to run eval_dual_bootstrap separately for each condition, but across all participants – meaning that for participants from group A it would have to compare to the first half of the model RDM, and for group B to the second half of the model RDM. Is this at all possible? For example, would it be possible to pass an rdm_descriptor not just for the neural RDMs but also for a FixedModel object that contains both A & B model RDMs, and for the eval functions to select the appropriate model matching the neural RDM's rdm_descriptor?
If not, can you think of any sensible way to get inference testing in this scenario?
I guess a last resort would be to run separate streams of analysis for groups A & B – but in that case I'm not sure if it would be valid to then somehow merge the results at a later stage, or if they would have to be treated as completely separate experiments?

@max-kat Another idea: Is it possible to run a searchlight individually for each person and then somehow average the results across participants? Does that make sense conceptually?

@JasperVanDenBosch Absolutely and I'd recommend that in many cases. One the second level you'd just do your normal FDR or clustering threshold.

Perhaps results could be merged this way? Running analysis within-subject and using only eval_bootstrap_pattern, and then averaging the results on the second level? This would lose the benefits of bootstrapping over participants though, right?

Many thanks for your work on this toolbox!

[JasperVanDenBosch]

Perhaps results could be merged this way? Running analysis within-subject and using only eval_bootstrap_pattern, and then averaging the results on the second level? This would lose the benefits of bootstrapping over participants though, right?

Correct

The NaN-based approach I think we don't have plans to support.

The other option would be the two separate groups throughout approach. You could just report them as separate results (potentially with correction for multiple comparisons if you think of them both as tests of the same general hypothesis).

Hope this helps