OutOfMemoryException when declaring large managed arrays

[johnearnshaw]

I'm eager to see this project working :)
So... I was playing with the dev branch and I'm unable to load the pre-trained GoogleNews-vectors-negative300.bin (300-dimensional vectors for 3 million words and phrases) due to the memory allocation restriction in managed arrays.

Have you considered using System.IO.MemoryMappedFiles.MemoryMappedFile to overcome this rather than declaring large arrays?

[eabdullin]

@johnearnshaw, hi. I tried to load my trained 200-dimensional vectors for 1 million words - was ok. So, i think your computer doesn't have enough memory, because algorithm uses manage arrays and allocate memory for needed size

Array.Resize(ref _vocab, _vocabMaxSize);

[tomachristian]

Well, still doesn't work for GoogleNews-vectors-negative300.bin

[GuntaButya]

2GB is the max Memory Size for an Object, you will need to split the Dataset, or use a smaller one.
See: http://stackoverflow.com/questions/982051/net-max-memory-use-2gb-even-for-x64-assemblies

long memory = GC.GetTotalMemory(false);
// Load your DataSet...
Console.WriteLine(String.Format(CultureInfo.InvariantCulture, "Corpus Memory Use: {0:0.0} G-bytes", ((Convert.ToDouble(totalMem) / 1024.0) / 1024.0) / 1024.0));