Can webpack runs in multi-thread mode?

[littlee]

It takes a while when run webpack -p,
but it only use single thread to run.

[bebraw]

Your configuration file can return an array of configurations. Each of those will be run by a separate instance.

[littlee]

How can I speed up webpack -p?

[thabti]

@littlee show us your config file.

[littlee]

var path = require('path');
var webpack = require('webpack');
var I18nPlugin = require('i18n-webpack-plugin');
var languages = {
    'en_US': null,
    'zh_CN': require('./js/i18n/zh_CN.json'),
    'vi_VN': require('./js/i18n/vi_VN.json')
};

var ExtractTextPlugin = require('extract-text-webpack-plugin');

module.exports = Object.keys(languages).map(function(language) {
    return {
        name: language,
        entry: {
            error: './js/error.js',
            businessCollege: './js/businessCollege.js',
            basConfirm: './js/basConfirm.js',
            basDeliver: './js/basDeliver.js',
            basStock: './js/basStock.js',
            basSettle: './js/basSettle.js',
            basHistory: './js/basHistory.js',
            basBillAccount: './js/basBillAccount.js',
            basWithdraw: './js/basWithdraw.js',
            basWithdrawHistory: './js/basWithdrawHistory.js',
            basWithdrawDetail: './js/basWithdrawDetail.js',
            basPrintTable: './js/basPrintTable.js',
            cart: './js/cart.js',
            entry: './js/entry.js',
            index: './js/index.js',
            favouriteProduct: './js/favouriteProduct.js',
            pay: './js/pay.js',
            payFail: './js/payFail.js',
            paySuccess: './js/paySuccess.js',
            productDetail: './js/productDetail.js',
            payOrder: './js/payOrder.js',
            refundApply: './js/refundApply.js',
            refundAwb: './js/refundAwb.js',
            refundDetail: './js/refundDetail.js',
            refundWaiting: './js/refundWaiting.js',
            rank: './js/rank.js',
            search: './js/search.js',
            userAddress: './js/userAddress.js',
            userCenter: './js/userCenter.js',
            userComment: './js/userComment.js',
            userOrder: './js/userOrder.js',
            userOrderDetail: './js/userOrderDetail.js',
            userProfile: './js/userProfile.js',
            internationalFashion: './js/internationalFashion.js',
            securityCenter: './js/securityCenter.js',
            refundApplication: './js/refundApplication.js',
            refundFinish: './js/refundFinish.js',
            refundProcessing: './js/refundProcessing.js',
            bulletinsAll: './js/bulletinsAll.js',
            bulletinsDetail: './js/bulletinsDetail.js',
            feedbackCenter: './js/feedbackCenter.js',
            billHistory: './js/billHistory.js',
            billDetail: './js/billDetail.js',
            recharge: './js/recharge.js',
            withdraw: './js/withdraw.js',
            transactionHistory: './js/transactionHistory.js',
            withdrawHistory: './js/withdrawHistory.js',
            rechargeHistory: './js/rechargeHistory.js',
            paySuccessLine: './js/paySuccessLine.js',
            payLine: './js/payLine.js',
            sign: './js/sign.js',
            resetPw: './js/resetPw.js',
            userProtocol: './js/userProtocol.js',
            helpCenter: './js/helpCenter.js',
            mBasSign: './js/mBasSign.js',
            mBasConfirm: './js/mBasConfirm.js',
            mBasConfirmIn: './js/mBasConfirmIn.js',
            mBasSearch: './js/mBasSearch.js',
            mBasDelivery: './js/mBasDelivery.js',
            mBasStock: './js/mBasStock.js',
            mBasStockIn: './js/mBasStockIn.js',
            mBasSettle: './js/mBasSettle.js',
            channel:'./js/channel.js',
            channel2:'./js/channel2.js',
            withdrawDetail: './js/withdrawDetail.js',
            channel3: './js/channel3.js',
            channel4:'./js/channel4.js'
        },
        output: {
            path: path.join(__dirname, 'build'),
            filename: language + '.[name].js',
            publicPath: './build/'
        },
        module: {
            loaders: [{
                test: /\.less$/,
                loader: ExtractTextPlugin.extract('style-loader', 'css-loader!less-loader', {
                        publicPath: './'
                    })
            }, {
                test: /\.css$/,
                loader: ExtractTextPlugin.extract('style-loader', 'css-loader', {
                        publicPath: './'
                    })
            }, {
                test: /\.(ttf|eot|svg|woff(2)?)(\?v=[\d.]+)?(\?[a-z0-9#-]+)?$/,
                loader: 'file-loader'
            }, {
                test: /\.(png|jpg|gif)$/,
                loader: 'url-loader?limit=8192'
            }, {
                test: /\.handlebars$/,
                loader: 'handlebars-loader',
                query: {
                    helperDirs: [__dirname + '/js/hbs/helpers']
                }
            }]
        },
        plugins: [
            new I18nPlugin(
                languages[language]
            ),
            new webpack.ProvidePlugin({
                $: 'jquery',
                jQuery: 'jquery',
                'window.jQuery': 'jquery',
                Handlebars: 'handlebars/runtime'
            }),
            new webpack.DefinePlugin({
                'require.specified': 'require.resolve',
            }),
            new ExtractTextPlugin('[name].css')
        ]
    };
});

this is the config file I used in my project www.cookabuy.com
it tasks at least 20 mins to run webpack -p, and our server has 16 core CPU, 8G RAM,
but wepack only run in single thread, I thought it was a waste of system resource.
so how can I make webpack take resource as more as possile to speed up?
Thanks

[bebraw]

Here's a rough idea to expand on what I answered before. Feel free to adapt:

var _ = require('lodash');
var os = require('os');

var entries = [...]; // your entries go here

module.exports = _.chunk(entries, os.cpus().length).map(function(entry) {
  return {
    entry: entry,
    ... rest of your config ...
  };
});

It's possibly you might be able to skip that os bit even. I just added it there so the tasks get split up evenly.

[littlee]

Notice that I've already use i18n-webpack-plugin and I have a map call in module.exports

I think this has already return a array.

[bebraw]

@littlee Correct. You get just three instances at best, though. It would be better to split per language and per entry. You will need to expand my example to achieve that.

If it looks like webpack isn't creating the instances as you expect (I'm not sure what it does internally), you can try something like worker-farm to control the process (no shared memory but you get simple instancing at least). Anyway, see what top or a similar tool says while you are running webpack. Ideally each of your cores would get something to do.

[littlee]

@bebraw I followed you suggesstion to change my config file, but it can not run

var _ = require('lodash');
var os = require('os');
var path = require('path');
var webpack = require('webpack');
var I18nPlugin = require('i18n-webpack-plugin');
var languages = {
    'en_US': null,
    'zh_CN': require('./js/i18n/zh_CN.json'),
    'vi_VN': require('./js/i18n/vi_VN.json')
};
var ExtractTextPlugin = require('extract-text-webpack-plugin');

var entries = [
    {
        basBillAccount: './js/basBillAccount.js'
    },
    {
        basConfirm: './js/basConfirm.js'
    },
    {
        basDeliver: './js/basDeliver.js'
    },
    {
        basHistory: './js/basHistory.js'
    },
    {
        basPrintTable: './js/basPrintTable.js'
    },
    {
        basSettle: './js/basSettle.js'
    },
    {
        basStock: './js/basStock.js'
    },
    {
        basWithdraw: './js/basWithdraw.js'
    },
    {
        basWithdrawDetail: './js/basWithdrawDetail.js'
    },
    {
        basWithdrawHistory: './js/basWithdrawHistory.js'
    },
    {
        billDetail: './js/billDetail.js'
    },
    {
        billHistory: './js/billHistory.js'
    },
    {
        bulletinsAll: './js/bulletinsAll.js'
    },
    {
        bulletinsDetail: './js/bulletinsDetail.js'
    },
    {
        businessCollege: './js/businessCollege.js'
    },
    {
        cart: './js/cart.js'
    },
    {
        channel: './js/channel.js'
    },
    {
        channel2: './js/channel2.js'
    },
    {
        channel3: './js/channel3.js'
    },
    {
        channel4: './js/channel4.js'
    },
    {
        entry: './js/entry.js'
    },
    {
        error: './js/error.js'
    },
    {
        favouriteProduct: './js/favouriteProduct.js'
    },
    {
        feedbackCenter: './js/feedbackCenter.js'
    },
    {
        helpCenter: './js/helpCenter.js'
    },
    {
        index: './js/index.js'
    },
    {
        internationalFashion: './js/internationalFashion.js'
    },
    {
        mBasConfirm: './js/mBasConfirm.js'
    },
    {
        mBasConfirmIn: './js/mBasConfirmIn.js'
    },
    {
        mBasDelivery: './js/mBasDelivery.js'
    },
    {
        mBasSearch: './js/mBasSearch.js'
    },
    {
        mBasSettle: './js/mBasSettle.js'
    },
    {
        mBasSign: './js/mBasSign.js'
    },
    {
        mBasStock: './js/mBasStock.js'
    },
    {
        mBasStockIn: './js/mBasStockIn.js'
    },
    {
        pay: './js/pay.js'
    },
    {
        payFail: './js/payFail.js'
    },
    {
        payLine: './js/payLine.js'
    },
    {
        payOrder: './js/payOrder.js'
    },
    {
        paySuccess: './js/paySuccess.js'
    },
    {
        paySuccessLine: './js/paySuccessLine.js'
    },
    {
        productDetail: './js/productDetail.js'
    },
    {
        rank: './js/rank.js'
    },
    {
        recharge: './js/recharge.js'
    },
    {
        rechargeHistory: './js/rechargeHistory.js'
    },
    {
        refundApplication: './js/refundApplication.js'
    },
    {
        refundApply: './js/refundApply.js'
    },
    {
        refundAwb: './js/refundAwb.js'
    },
    {
        refundDetail: './js/refundDetail.js'
    },
    {
        refundFinish: './js/refundFinish.js'
    },
    {
        refundProcessing: './js/refundProcessing.js'
    },
    {
        refundWaiting: './js/refundWaiting.js'
    },
    {
        resetPw: './js/resetPw.js'
    },
    {
        search: './js/search.js'
    },
    {
        securityCenter: './js/securityCenter.js'
    },
    {
        sign: './js/sign.js'
    },
    {
        transactionHistory: './js/transactionHistory.js'
    },
    {
        userAddress: './js/userAddress.js'
    },
    {
        userCenter: './js/userCenter.js'
    },
    {
        userComment: './js/userComment.js'
    },
    {
        userOrder: './js/userOrder.js'
    },
    {
        userOrderDetail: './js/userOrderDetail.js'
    },
    {
        userProfile: './js/userProfile.js'
    },
    {
        userProtocol: './js/userProtocol.js'
    },
    {
        withdraw: './js/withdraw.js'
    },
    {
        withdrawDetail: './js/withdrawDetail.js'
    },
    {
        withdrawHistory: './js/withdrawHistory.js'
    }
];

module.exports = _.chunk(entries, os.cpus().length).map(function(entry) {
    return Object.keys(languages).map(function(language) {
        return {
            name: language,
            entry: entry,
            output: {
                path: path.join(__dirname, 'build'),
                filename: language + '.[name].js',
                publicPath: './build/'
            },
            module: {
                loaders: [{
                    test: /\.less$/,
                    loader: ExtractTextPlugin.extract('style-loader', 'css-loader!less-loader', {
                        publicPath: './'
                    })
                }, {
                    test: /\.css$/,
                    loader: ExtractTextPlugin.extract('style-loader', 'css-loader', {
                        publicPath: './'
                    })
                }, {
                    test: /\.(ttf|eot|svg|woff(2)?)(\?v=[\d.]+)?(\?[a-z0-9#-]+)?$/,
                    loader: 'file-loader'
                }, {
                    test: /\.(png|jpg|gif)$/,
                    loader: 'url-loader?limit=8192'
                }, {
                    test: /\.handlebars$/,
                    loader: 'handlebars-loader',
                    query: {
                        helperDirs: [__dirname + '/js/hbs/helpers']
                    }
                }]
            },
            plugins: [
                new I18nPlugin(
                    languages[language]
                ),
                new webpack.ProvidePlugin({
                    $: 'jquery',
                    jQuery: 'jquery',
                    'window.jQuery': 'jquery',
                    // Handlebars: 'handlebars/runtime'
                    Handlebars: __dirname + '/js/hbs/handlebars.runtime-v4.0.4.js'
                }),
                new webpack.DefinePlugin({
                    'require.specified': 'require.resolve',
                    'ENV': {
                        // 'debug': true
                        'debug': false
                    }
                }),
                new ExtractTextPlugin('[name].css')
            ]
        };
    });
});

[bebraw]

This is just a hunch but you could try altering filename: language + '.[name].js',. Instead of relying on [name], pass it directly there (you know the entry name already). Earlier it got the name from the entry array configuration. Perhaps it fails in this particular case.

[littlee]

I have change my config file to below:

module.exports = _.chunk(entries, os.cpus().length).map(function(entry) {

    var mEntry = {};

    entry.forEach(function(e) {
        _.assign(mEntry, e);
    });

    return Object.keys(languages).map(function(language) {
        return {
            name: language,
            entry: mEntry,
            output: {
                path: path.join(__dirname, 'build'),
                filename: language + '.[name].js',
                publicPath: './build/'
            },
...

but [name] still can not be resolved, how do I fix it?

mEntry has many keys (same as cpu length), so I should dynamicly get [name]

[sokra]

Your configuration file can return an array of configurations. Each of those will be run by a separate instance.

Nope. webpack doesn't do any threading or "process"ing. That's up to you.

[bebraw]

@sokra Ah, great to know! I think hooking up worker-farm or similar is the ticket then.

[littlee]

can you guys provides some config code examples about hook up with worker-farm?

[sokra]

@pago

[bebraw]

@littlee Quick idea:

build.js

var workerFarm = require('worker-farm');
var workers = workerFarm(require.resolve('./build_worker'));
var async = require('async');

...

// tasks will contain your worker specific bits (i.e., entry) you want each
// worker to receive. you cannot pass functions here given they are
// instances of their own (no memory sharing)
async.each(tasks, function(task, cb) {
  console.log('Starting task', task);

  workers(task, function(err) {
    log('Finished task', task);

    cb(err);
  });
}, function(err) {
  console.log('Tasks finished');

  workerFarm.end(workers);

  if(err) {
    return reject(err);
  }

  resolve();
});

build_worker.js

var webpack = require('webpack');

module.exports = function(task, cb) {
  // process through webpack now
  webpack(Object.assign({}, common, task), cb);
};

You'll need to adapt a little to fit your case but that gives you the basic idea. It's pretty much the same code than at worker_farm readme.

[pago]

If you can wait for a week I'll be able to open source the solution we're using at Trivago. It also uses worker-farm and has been working very well for us for the past few months, allowing us to build 32 different variants in about 2 minutes.

[littlee]

@pago looking forward to your surprise 😍

[littlee]

@pago It's been a week.

[pago]

@littlee First of all: The day isn't over yet. ;)
More serious though: We're working with high speed on publishing our tool but I can't guarantee a specific date. We're trying to release it today but it could happen that it'll take a few more days.

[littlee]

😂 keep going

[sokra]

https://github.com/trivago/parallel-webpack

[littlee]

let me try try

[pago]

@littlee Please let me know if you have any problems with the tool or would like to see some feature added to it.

[littlee]

we do meet some problem, when the entry points is too much(201 in my case), parallel-webpack will keep retrying, and it stuck.
Though it built rapidly in my test case(I gave it 9 entry points)

I can show you my config file, see whether you can adjust it to help me not get stuck while building

webpack.config.js

var path = require('path');
var webpack = require('webpack');
var I18nPlugin = require('i18n-webpack-plugin');
var languages = {
    'en_US': null,
    'zh_CN': require('./js/i18n/zh_CN.json'),
    'vi_VN': require('./js/i18n/vi_VN.json')
};

var ExtractTextPlugin = require('extract-text-webpack-plugin');

var entries = {
    error: './js/error.js',
    businessCollege: './js/businessCollege.js',
    basConfirm: './js/basConfirm.js',
    basDeliver: './js/basDeliver.js',
    basStock: './js/basStock.js',
    basSettle: './js/basSettle.js',
    basHistory: './js/basHistory.js',
    basBillAccount: './js/basBillAccount.js',
    basWithdraw: './js/basWithdraw.js',
    basWithdrawHistory: './js/basWithdrawHistory.js',
    basWithdrawDetail: './js/basWithdrawDetail.js',
    basPrintTable: './js/basPrintTable.js',
    cart: './js/cart.js',
    entry: './js/entry.js',
    index: './js/index.js',
    favouriteProduct: './js/favouriteProduct.js',
    pay: './js/pay.js',
    payFail: './js/payFail.js',
    paySuccess: './js/paySuccess.js',
    productDetail: './js/productDetail.js',
    payOrder: './js/payOrder.js',
    refundApply: './js/refundApply.js',
    refundAwb: './js/refundAwb.js',
    refundDetail: './js/refundDetail.js',
    refundWaiting: './js/refundWaiting.js',
    rank: './js/rank.js',
    search: './js/search.js',
    userAddress: './js/userAddress.js',
    userCenter: './js/userCenter.js',
    userComment: './js/userComment.js',
    userOrder: './js/userOrder.js',
    userOrderDetail: './js/userOrderDetail.js',
    userProfile: './js/userProfile.js',
    internationalFashion: './js/internationalFashion.js',
    securityCenter: './js/securityCenter.js',
    refundApplication: './js/refundApplication.js',
    refundFinish: './js/refundFinish.js',
    refundProcessing: './js/refundProcessing.js',
    bulletinsAll: './js/bulletinsAll.js',
    bulletinsDetail: './js/bulletinsDetail.js',
    feedbackCenter: './js/feedbackCenter.js',
    billHistory: './js/billHistory.js',
    billDetail: './js/billDetail.js',
    recharge: './js/recharge.js',
    withdraw: './js/withdraw.js',
    transactionHistory: './js/transactionHistory.js',
    withdrawHistory: './js/withdrawHistory.js',
    rechargeHistory: './js/rechargeHistory.js',
    paySuccessLine: './js/paySuccessLine.js',
    payLine: './js/payLine.js',
    sign: './js/sign.js',
    resetPw: './js/resetPw.js',
    userProtocol: './js/userProtocol.js',
    helpCenter: './js/helpCenter.js',
    mBasSign: './js/mBasSign.js',
    mBasConfirm: './js/mBasConfirm.js',
    mBasConfirmIn: './js/mBasConfirmIn.js',
    mBasSearch: './js/mBasSearch.js',
    mBasDelivery: './js/mBasDelivery.js',
    mBasStock: './js/mBasStock.js',
    mBasStockIn: './js/mBasStockIn.js',
    mBasSettle: './js/mBasSettle.js',
    channel: './js/channel.js',
    channel2: './js/channel2.js',
    withdrawDetail: './js/withdrawDetail.js',
    channel3: './js/channel3.js',
    channel4: './js/channel4.js'
};

var config = Object.keys(entries).map(function(entry) {
    return Object.keys(languages).map(function(language) {
        return {
            name: language,
            entry: entries[entry],
            output: {
                path: path.join(__dirname, 'build'),
                filename: language + '.' + entry + '.js',
                publicPath: './build/'
            },
            module: {
                loaders: [{
                    test: /\.less$/,
                    loader: ExtractTextPlugin.extract('style-loader', 'css-loader!less-loader', {
                        publicPath: './'
                    })
                }, {
                    test: /\.css$/,
                    loader: ExtractTextPlugin.extract('style-loader', 'css-loader', {
                        publicPath: './'
                    })
                }, {
                    test: /\.(ttf|eot|svg|woff(2)?)(\?v=[\d.]+)?(\?[a-z0-9#-]+)?$/,
                    loader: 'file-loader'
                }, {
                    test: /\.(png|jpg|gif)$/,
                    loader: 'url-loader?limit=8192'
                }, {
                    test: /\.handlebars$/,
                    loader: 'handlebars-loader',
                    query: {
                        helperDirs: [__dirname + '/js/hbs/helpers']
                    }
                }]
            },
            plugins: [
                new I18nPlugin(
                    languages[language]
                ),
                new webpack.ProvidePlugin({
                    $: 'jquery',
                    jQuery: 'jquery',
                    'window.jQuery': 'jquery',
                    // Handlebars: 'handlebars/runtime'
                    Handlebars: __dirname + '/js/hbs/handlebars.runtime-v4.0.4.js'
                }),
                new webpack.DefinePlugin({
                    'require.specified': 'require.resolve',
                    'ENV': {
                        // 'debug': true
                        'debug': false
                    }
                }),
                new ExtractTextPlugin('[name].css')
            ]
        };
    });
});

var f = [];
config.forEach(function(c) {
    f = f.concat(c);
});

console.log(f.length);
module.exports = f;

[damianobarbati]

@bebraw

Your configuration file can return an array of configurations. Each of those will be run by a separate instance.

So I don't need external libs/tools like https://github.com/trivago/parallel-webpack if I'm exporting an array of configurations?
Are they going to spread across different cores automatically?

Using webpack@4 here

[bebraw]

@damianobarbati My understanding is that webpack doesn't run parallel like that just yet so it's going to be just a single instance for now.

[damianobarbati]

@sokra would you consider this as a possible feature? Having each entry in a config array spread on available cores?

[heisian]

@sokra Is it possible to fork a process for each chunk's processing? One of node's strengths is easy inter-process communication and running even a single build in a single process is pretty inefficient IMO. of course I could be trivializing what it takes to separate/fork out the build process.

[moonjoungyoung]

Any updates?

[chakaponi]

@moonjoungyoung parallel-webpack
This version present only on npm (not github). Older version here

[Patrokl42]

Hi guys.
It seems parallel-webpack is not working with webpack 5 and is not maintained anymore also webpack still does not support multi-thread from the box.
That's why I created the npm package webpack-multithread it works like parallel-webpack but uses worker_threads.
That package is compatible with wepback v4 and v5.
I hope that package will be helpful for someone.