stream bz xml file

[leojoubert]

Hi,

I need to parse wikipedia dumps for research purpose. The dumps are very very big (much more than 1TB). These are the file "page-meta-history.xml" -> http://dumps.wikimedia.org
So I can't unzip the file. Can I word with bz file into the xml-steam API ?

Thank you

[jbielick]

You can pipe the read stream into an unzip transformer and then pass the stream to XmlStream.

try this:

const request = require('request');
const zlib = require('zlib');

let readStream = request('http://dumps.wikimedia.org/some-dump.xml').pipe(zlib.createGunzip());
let parser  = new XmlStream(readStream);

[cigolpl]

@cafeine05 you can try this https://github.com/spencermountain/wikipedia-to-mongodb for doing whole job

In very short decompressing bz in stream has such a logic:

  var bz2 = require('unbzip2-stream');
  var stream = fs.createReadStream(file).pipe(bz2());
  var stream = fs.createReadStream(file);
  var xml = new XmlStream(stream);