CHOLMOD out of memory exception in cosSparse for very large matrix

[katherine-benjamin]

I'm attempting to run cosSparse on a sparse matrix of ~30k rows by ~100k columns, with around 90% sparsity, on a machine with plenty (~3TB) of memory. However I'm getting the following exception which seems to suggest we're running out of memory:

Error in .local(x, y = y, ...) :
  CHOLMOD error 'problem too large' at file '../Core/cholmod_aat.c', line 170

Unless I'm making an order of magnitude error (very possible), I'd expect the output dense matrix to take about 80GB of memory. Is this outside the limits of the cosSparse implementation?

[cysouw]

Do you really have 3 TB of memory? I'd like that machine :-)! The bottleneck is the RAM-memory, and on my current machine I have 24 GB...

With random data for a matrix of your size with 10% filled cells, my machine becomes very slow, and does not seem to finish the calculations. The bottleneck is the number of filled cells, in the output matrix, which might be almost full with 90% sparsity.

It is also important which sides you want to correlate: if you want to end up with a 100k by 100k matrix which is almost full, then you easily get to the limits of your machine. if the result is 30k by 30k matrix it is much easier

For example: this works:

system.time(X <- rSparseMatrix(1e7, 1e4, 1e7))
system.time(M <- cosSparse(X))

but this does not on my machine:

system.time(X <- rSparseMatrix(1e4, 1e7, 1e7))
system.time(M <- cosSparse(X))