Error: Mat::init(): requested size is too large; suggest to enable ARMA_64BIT_WORD

[GregorDall]

Dear Eduardo,

We are trying to use A.mat with 75000k lines and 10k markers, getting the following error:
Error: Mat::init(): requested size is too large; suggest to enable ARMA_64BIT_WORD
Still seems to work with 50k lines, but 75k doesn't work any more. Must be related to the number of individuals, because I can replicate it with only 5 markers too. rrBLUP::A.mat does not seem to have this problem, but I thought sommer::A.mat might be more efficient?

I am not sure if that is something related to sommer or Rcpp...

Best Gregor

[yunmika]

Dear Eduardo,

We are trying to use A.mat with 75000k lines and 10k markers, getting the following error: Error: Mat::init(): requested size is too large; suggest to enable ARMA_64BIT_WORD Still seems to work with 50k lines, but 75k doesn't work any more. Must be related to the number of individuals, because I can replicate it with only 5 markers too. rrBLUP::A.mat does not seem to have this problem, but I thought sommer::A.mat might be more efficient?

I am not sure if that is something related to sommer or Rcpp...

Best Gregor

Hello, I also encountered the same problem as you. Is there any solution?

[GregorDall]

Dear Eduardo,
We are trying to use A.mat with 75000k lines and 10k markers, getting the following error: Error: Mat::init(): requested size is too large; suggest to enable ARMA_64BIT_WORD Still seems to work with 50k lines, but 75k doesn't work any more. Must be related to the number of individuals, because I can replicate it with only 5 markers too. rrBLUP::A.mat does not seem to have this problem, but I thought sommer::A.mat might be more efficient?
I am not sure if that is something related to sommer or Rcpp...
Best Gregor

Hello, I also encountered the same problem as you. Is there any solution?

Hi, my solution was to use A.mat from the rrBLUP package.

Best Gregor

[yunmika]

Dear Eduardo,
We are trying to use A.mat with 75000k lines and 10k markers, getting the following error: Error: Mat::init(): requested size is too large; suggest to enable ARMA_64BIT_WORD Still seems to work with 50k lines, but 75k doesn't work any more. Must be related to the number of individuals, because I can replicate it with only 5 markers too. rrBLUP::A.mat does not seem to have this problem, but I thought sommer::A.mat might be more efficient?
I am not sure if that is something related to sommer or Rcpp...
Best Gregor

Hello, I also encountered the same problem as you. Is there any solution?

Hi, my solution was to use A.mat from the rrBLUP package.

Best Gregor

Okay thanks for your reply.

[covaruber]

Sorry that I never followed up with this issue. I tried some solutions but I haven't succeeded. I am not sure why this happens. It seems to be more related to RcppArmadillo than sommer itself but I will keep trying things and see if I can make it work.

Cheers,

[yunmika]

Sorry that I never followed up with this issue. I tried some solutions but I haven't succeeded. I am not sure why this happens. It seems to be more related to RcppArmadillo than sommer itself but I will keep trying things and see if I can make it work.

Cheers,

Hi, the problem seems to be in MNR.cpp, I tried the following changes to avoid such error messages.

// -*- mode: C++; c-indent-level: 4; c-basic-offset: 4; indent-tabs-mode: nil; -*-

// we only include RcppArmadillo.h which pulls Rcpp.h in for us
#define ARMA_64BIT_WORD
#define ARMA_DONT_PRINT_ERRORS
#include "RcppArmadillo.h"
#include "stdlib.h"
#include <progress.hpp>
// [[Rcpp::export]]
arma::mat amat(const arma::mat & Xo, const bool & vanraden, double minMAF) {
  // remove min.MAF
  arma::rowvec pfreq = mean(Xo+1,0)/2; // frequency of p
  arma::mat pqfreq = arma::join_cols(pfreq,1-pfreq); // frequencies of p and q
  arma::rowvec MAF = min(pqfreq,0); // minor allele freqs
  arma::uvec indexMAF = find(MAF > minMAF); // index for good markers > minMAF
  arma::mat Xo2 = Xo.cols(indexMAF); // new X only with polymorphic markers
  // remove monomorphic markers
  arma::rowvec xVar = var(Xo2,0); // column variance
  arma::uvec index = find(xVar > 0); // index for good markers
  arma::mat X = Xo2.cols(index); // new X only with polymorphic markers
  // initialize A
  int p = X.n_cols;// number of markers
  int n = X.n_rows;
  arma::mat A(n,n);
  if(vanraden == true){ //  regular vanRaden 
    arma::rowvec ms012 = mean( X+1, 0 ); // means of columns
    arma::rowvec freq = ms012/2;
    double v = 2 * mean(freq % (1 - freq));
    arma::mat one(n, 1, arma::fill::ones);
    arma::mat freqmat = one * freq;
    arma::mat W = (X + 1) - (2 * freqmat);
    //
    arma::mat K = W * W.t();
    A = K/v/p;
  }else{ // Endelman (currently we have a bug here)
    // IN R: M <- scale(X, center = TRUE, scale = FALSE)
    arma::rowvec ms = mean( X, 0 ); // means of columns
    arma::mat M = X.each_row() - ms;
    // IN R: tcrossprod(M)
    arma::mat K = M * M.t();
    // IN R: K/mean(diag(K))   mean(K.diag())
    double v = mean(diagvec(K));
    A = K/v;
  }
  return A;
}

[covaruber]

I already tried that and it solves it for isolated functions but it does not work for compiling the entire package. But thanks for trying on your side. This issue seems to arise from functions that use the sparse matrix functions from the Armadillo library. I will keep trying other things.