Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Leverage micromatch to speed up glob matching #911

Open
wants to merge 5 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
51 changes: 51 additions & 0 deletions benchmark.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
// Description: Benchmarking the performance of micromatch vs minimatch
import minimatch from 'minimatch';
import micromatch from 'micromatch';

const Benchmark = require('benchmark');

(() => {
var suite = new Benchmark.Suite();

const ignorePatterns = ['*.test.js', 'tests/**', 'yarn.lock', 'change/*.json', 'CHANGELOG.{md,json}'];

const filesToCheck = [
'src/foo.test.js',
'tests/stuff.js',
'yarn.lock',
'playwright',
'browser',
'edge',
'foo',
'bar',
'baz',
'qux',
'src/f',
];

suite
.add('micromatch', function () {
for (const file of filesToCheck) {
ignorePatterns.find(pattern => micromatch.isMatch(file, pattern, { matchBase: true }));
}
})
.add('minimatch', function () {
for (const file of filesToCheck) {
ignorePatterns.find(pattern => minimatch(file, pattern, { matchBase: true }));
}
})
.on('complete', function (this: any) {
console.log('Fastest is ' + this.filter('fastest').map('name'));
const tableData = this.map(benchmark => ({
Function: benchmark.name,
'Mean Time (ms)': (benchmark.stats.mean * 1000).toFixed(2), // Convert to milliseconds
'Operations per Second': benchmark.hz.toFixed(2),
'Standard Error of the Mean': benchmark.stats.sem.toFixed(2),
'Relative Margin of Error': benchmark.stats.rme.toFixed(2),
}));

// Create a console table
console.table(tableData);
})
.run();
})();
62 changes: 62 additions & 0 deletions benchmarkLarge.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
// Description: Benchmarking the performance of micromatch vs minimatch
import minimatch from 'minimatch';
import micromatch from 'micromatch';

const Benchmark = require('benchmark');

(() => {
var suite = new Benchmark.Suite();

const ignorePatterns = ['*.test.js', 'tests/**', 'yarn.lock', 'change/*.json', 'CHANGELOG.{md,json}'];

const filesToCheck = [
'src/foo.test.js',
'tests/stuff.js',
'yarn.lock',
'playwright',
'browser',
'edge',
'foo',
'bar',
'baz',
'qux',
'src/f',
];

const numberOfRandomFiles = 10000;

// Generate additional random files
for (let i = 0; i < numberOfRandomFiles; i++) {
const randomIndex = Math.floor(Math.random() * filesToCheck.length);
const randomFileName = filesToCheck[randomIndex];
const randomVariant = Math.floor(Math.random() * 1000); // Generate a random number for variation

filesToCheck.push(`${randomFileName}_${randomVariant}`);
}

suite
.add('micromatch', function () {
for (const file of filesToCheck) {
ignorePatterns.find(pattern => micromatch.isMatch(file, pattern, { matchBase: true }));
}
})
.add('minimatch', function () {
for (const file of filesToCheck) {
ignorePatterns.find(pattern => minimatch(file, pattern, { matchBase: true }));
}
})
.on('complete', function (this: any) {
console.log('Fastest is ' + this.filter('fastest').map('name'));
const tableData = this.map(benchmark => ({
Function: benchmark.name,
'Mean Time (ms)': (benchmark.stats.mean * 1000).toFixed(2), // Convert to milliseconds
'Operations per Second': benchmark.hz.toFixed(2),
'Standard Error of the Mean': benchmark.stats.sem.toFixed(2),
'Relative Margin of Error': benchmark.stats.rme.toFixed(2),
}));

// Create a console table
console.table(tableData);
})
.run();
})();
2 changes: 1 addition & 1 deletion docs/concepts/groups.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,6 @@ module.exports = {
}
```

`beachball` uses `minimatch` to match which packages belong to which group via this configuration. In the above configuration, packages located inside `packages/groupfoo` would be bumped together.
`beachball` uses `micromatch` to match which packages belong to which group via this configuration. In the above configuration, packages located inside `packages/groupfoo` would be bumped together.

> NOTE: Beachball does not guarantee currently that these packages have the same version number, but that it will be bumped at the same rate. This is an area of active development, so please consider submitting feature request issues to change its behavior with justifications
2 changes: 1 addition & 1 deletion docs/overview/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -71,7 +71,7 @@ For the latest full list of supported options, see `RepoOptions` [in this file](
| `groups` | `VersionGroupOptions[]` ([details][3]) | | repo | specifies groups of packages that need to be version bumped at the same time |
| `groupChanges` | bool | `false` | repo | will write multiple changes to a single changefile |
| `hooks` | `HooksOptions` ([details][4]) | | repo | hooks for custom pre/post publish actions |
| `ignorePatterns` | string[] | | repo | ignore changes in these files (minimatch patterns; negations not supported) |
| `ignorePatterns` | string[] | | repo | ignore changes in these files (micromatch patterns; negations not supported) |
| `package` | string | | repo | specifies which package the command relates to (overrides change detection based on `git diff`) |
| `prereleasePrefix` | string | | repo | prerelease prefix for packages that are specified to receive a prerelease bump |
| `publish` | bool | `true` | repo | whether to publish to npm registry |
Expand Down
3 changes: 3 additions & 0 deletions package.json
Original file line number Diff line number Diff line change
Expand Up @@ -41,10 +41,12 @@
"update-snapshots": "yarn test:unit -u && yarn test:func -u && yarn test:e2e -u"
},
"dependencies": {
"benchmark": "^2.1.4",
"cosmiconfig": "^7.0.0",
"execa": "^5.0.0",
"fs-extra": "^10.0.0",
"lodash": "^4.17.15",
"micromatch": "^3.1.10",
"minimatch": "^3.0.4",
"p-limit": "^3.0.2",
"prompts": "^2.4.2",
Expand All @@ -58,6 +60,7 @@
"@jest/globals": "^29.0.0",
"@types/fs-extra": "^9.0.13",
"@types/lodash": "^4.14.191",
"@types/micromatch": "^3.1.0",
"@types/minimatch": "^5.0.0",
"@types/node": "^14.0.0",
"@types/prompts": "^2.4.2",
Expand Down
4 changes: 2 additions & 2 deletions src/changefile/getChangedPackages.ts
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
import fs from 'fs-extra';
import path from 'path';
import minimatch from 'minimatch';
import micromatch from 'micromatch';
import { ChangeFileInfo, ChangeInfoMultiple } from '../types/ChangeInfo';
import { changeFolder, getChangePath } from '../paths';
import { getChanges, getStagedChanges, git } from 'workspace-tools';
Expand Down Expand Up @@ -89,7 +89,7 @@ function getAllChangedPackages(options: BeachballOptions, packageInfos: PackageI
// Also ignore the CHANGELOG files and change files because they're generated by beachball.
const ignorePatterns = [...(options.ignorePatterns || []), `${changeFolder}/*.json`, 'CHANGELOG.{md,json}'];
const nonIgnoredChanges = changes.filter(moddedFile => {
const ignorePattern = ignorePatterns.find(pattern => minimatch(moddedFile, pattern, { matchBase: true }));
const ignorePattern = ignorePatterns.find(pattern => micromatch.isMatch(moddedFile, pattern, { matchBase: true }));
ignorePattern && logIgnored(moddedFile, `ignored by pattern "${ignorePattern}"`);
return !ignorePattern;
});
Expand Down
8 changes: 4 additions & 4 deletions src/monorepo/isPathIncluded.ts
Original file line number Diff line number Diff line change
@@ -1,19 +1,19 @@
import minimatch from 'minimatch';
import micromatch from 'micromatch';

/**
* Check if a relative path should be included given include and exclude patterns using minimatch.
* Check if a relative path should be included given include and exclude patterns using micromatch.
*/
export function isPathIncluded(relativePath: string, include: string | string[], exclude?: string | string[]): boolean {
const includePatterns = typeof include === 'string' ? [include] : include;
let shouldInclude = includePatterns.reduce(
(included, pattern) => included || minimatch(relativePath, pattern),
(included, pattern) => included || micromatch.isMatch(relativePath, pattern),
false
);

if (exclude) {
const excludePatterns = typeof exclude === 'string' ? [exclude] : exclude;
shouldInclude = excludePatterns.reduce(
(excluded: boolean, pattern: string) => excluded && minimatch(relativePath, pattern),
(excluded: boolean, pattern: string) => excluded && micromatch.isMatch(relativePath, pattern),
shouldInclude
);
}
Expand Down
27 changes: 21 additions & 6 deletions src/types/BeachballOptions.ts
Original file line number Diff line number Diff line change
Expand Up @@ -116,7 +116,7 @@ export interface RepoOptions {
gitTags: boolean;
/** Custom pre/post publish actions */
hooks?: HooksOptions;
/** Ignore changes in these files (minimatch patterns; negations not supported) */
/** Ignore changes in these files (micromatch patterns; negations not supported) */
ignorePatterns?: string[];
/** For the `change` command, change message. For the `publish` command, commit message. */
message: string;
Expand Down Expand Up @@ -179,10 +179,10 @@ export interface PackageOptions {
* Options for bumping package versions together.
*/
export interface VersionGroupOptions {
/** minimatch pattern (or array of minimatch) to detect which packages should be included in this group */
/** micromatch pattern (or array of micromatch) to detect which packages should be included in this group */
include: string | string[];

/** minimatch pattern (or array of minimatch) to detect which packages should be excluded in this group */
/** micromatch pattern (or array of micromatch) to detect which packages should be excluded in this group */
exclude?: string | string[];

disallowedChangeTypes: ChangeType[] | null;
Expand All @@ -204,7 +204,12 @@ export interface HooksOptions {
* @param version The post-bump version of the package to be published
* @param packageInfos Metadata about other packages processed by Beachball. Computed post-bump. Readonly.
*/
prepublish?: (packagePath: string, name: string, version: string, packageInfos: Readonly<PackageInfos>) => void | Promise<void>;
prepublish?: (
packagePath: string,
name: string,
version: string,
packageInfos: Readonly<PackageInfos>
) => void | Promise<void>;

/**
* Runs for each package after the publish command.
Expand All @@ -215,7 +220,12 @@ export interface HooksOptions {
* @param version The post-bump version of the package to be published
* @param packageInfos Metadata about other packages processed by Beachball. Computed post-bump. Readonly.
*/
postpublish?: (packagePath: string, name: string, version: string, packageInfos: Readonly<PackageInfos>) => void | Promise<void>;
postpublish?: (
packagePath: string,
name: string,
version: string,
packageInfos: Readonly<PackageInfos>
) => void | Promise<void>;

/**
* Runs for each package, before writing changelog and package.json updates
Expand All @@ -236,7 +246,12 @@ export interface HooksOptions {
* @param version The post-bump version of the package to be published
* @param packageInfos Metadata about other packages processed by Beachball. Computed post-bump. Readonly.
*/
postbump?: (packagePath: string, name: string, version: string, packageInfos: Readonly<PackageInfos>) => void | Promise<void>;
postbump?: (
packagePath: string,
name: string,
version: string,
packageInfos: Readonly<PackageInfos>
) => void | Promise<void>;

/**
* Runs once after all bumps to all packages before committing changes
Expand Down
4 changes: 2 additions & 2 deletions src/types/ChangelogOptions.ts
Original file line number Diff line number Diff line change
Expand Up @@ -36,10 +36,10 @@ export interface ChangelogGroupOptions {
*/
masterPackageName: string;

/** minimatch pattern (or array of minimatch) to detect which packages should be included in this group */
/** micromatch pattern (or array of micromatch) to detect which packages should be included in this group */
include: string | string[];

/** minimatch pattern (or array of minimatch) to detect which packages should be excluded in this group */
/** micromatch pattern (or array of micromatch) to detect which packages should be excluded in this group */
exclude?: string | string[];

changelogPath: string;
Expand Down
2 changes: 1 addition & 1 deletion tsconfig.json
Original file line number Diff line number Diff line change
Expand Up @@ -18,5 +18,5 @@
"lib": ["ES2020"],
"types": ["node"]
},
"include": ["src/**/*"]
"include": ["src/**/*", "benchmark.ts", "benchmarkLarge.ts"]
}
41 changes: 40 additions & 1 deletion yarn.lock
Original file line number Diff line number Diff line change
Expand Up @@ -1349,6 +1349,11 @@
"@types/connect" "*"
"@types/node" "*"

"@types/braces@*":
version "3.0.2"
resolved "https://registry.yarnpkg.com/@types/braces/-/braces-3.0.2.tgz#70009e5d385bc0d804f40f0a3f92285022536730"
integrity sha512-U5tlMYa0U/2eFTmJgKcPWQOEICP173sJDa6OjHbj5Tv+NVaYcrq2xmdWpNXOwWYGwJu+jER/pfTLdoQ31q8PzA==

"@types/connect-history-api-fallback@*":
version "1.5.0"
resolved "https://registry.yarnpkg.com/@types/connect-history-api-fallback/-/connect-history-api-fallback-1.5.0.tgz#9fd20b3974bdc2bcd4ac6567e2e0f6885cb2cf41"
Expand Down Expand Up @@ -1472,6 +1477,13 @@
resolved "https://registry.yarnpkg.com/@types/mdurl/-/mdurl-1.0.2.tgz#e2ce9d83a613bacf284c7be7d491945e39e1f8e9"
integrity sha512-eC4U9MlIcu2q0KQmXszyn5Akca/0jrQmwDRgpAMJai7qBWq4amIQhZyNau4VYGtCeALvW1/NtjzJJ567aZxfKA==

"@types/micromatch@^3.1.0":
version "3.1.1"
resolved "https://registry.yarnpkg.com/@types/micromatch/-/micromatch-3.1.1.tgz#98eac88894da6606c2690624a9893a59c812d9fa"
integrity sha512-Wr5y4uv3r7JP4jEUqv7rZeYiMBGRHcbojDVsl11wq6gw1v/ZZQvJexd9rtvVx3EIVqw8dwtcRjSs8m2DV9qHjQ==
dependencies:
"@types/braces" "*"

"@types/mime@*", "@types/mime@^1":
version "1.3.2"
resolved "https://registry.yarnpkg.com/@types/mime/-/mime-1.3.2.tgz#93e25bf9ee75fe0fd80b594bc4feb0e862111b5a"
Expand Down Expand Up @@ -2684,6 +2696,14 @@ [email protected]:
resolved "https://registry.yarnpkg.com/bcryptjs/-/bcryptjs-2.4.3.tgz#9ab5627b93e60621ff7cdac5da9733027df1d0cb"
integrity sha512-V/Hy/X9Vt7f3BbPJEi8BdVFMByHi+jNXrYkW3huaybV/kQ0KJg0Y6PkEMbn+zeT+i+SiKZ/HMqJGIIt4LZDqNQ==

benchmark@^2.1.4:
version "2.1.4"
resolved "https://registry.yarnpkg.com/benchmark/-/benchmark-2.1.4.tgz#09f3de31c916425d498cc2ee565a0ebf3c2a5629"
integrity sha512-l9MlfN4M1K/H2fbhfMy3B7vJd6AGKJVQn2h6Sg/Yx+KckoUA7ewS5Vv6TjSq18ooE1kS9hhAlQRH3AkXIh/aOQ==
dependencies:
lodash "^4.17.4"
platform "^1.3.3"

big.js@^3.1.3:
version "3.2.0"
resolved "https://registry.yarnpkg.com/big.js/-/big.js-3.2.0.tgz#a5fc298b81b9e0dca2e458824784b65c52ba588e"
Expand Down Expand Up @@ -3142,6 +3162,13 @@ char-regex@^1.0.2:
resolved "https://registry.yarnpkg.com/char-regex/-/char-regex-1.0.2.tgz#d744358226217f981ed58f479b1d6bcc29545dcf"
integrity sha512-kWWXztvZ5SBQV+eRgKFeh8q5sLuZY2+8WUIzlxWVTg+oGwY14qylx1KbKzHd8P6ZYkAg0xyIDU9JMHhyJMZ1jw==

charm@*:
version "1.0.2"
resolved "https://registry.yarnpkg.com/charm/-/charm-1.0.2.tgz#8add367153a6d9a581331052c4090991da995e35"
integrity sha512-wqW3VdPnlSWT4eRiYX+hcs+C6ViBPUWk1qTCd+37qw9kEm/a5n2qcyQDMBWvSYKN/ctqZzeXNQaeBjOetJJUkw==
dependencies:
inherits "^2.0.1"

chokidar@^2.0.3, chokidar@^2.1.8:
version "2.1.8"
resolved "https://registry.yarnpkg.com/chokidar/-/chokidar-2.1.8.tgz#804b3a7b6a99358c3c5c61e71d8728f041cff917"
Expand Down Expand Up @@ -3231,6 +3258,13 @@ cli-boxes@^2.2.0:
resolved "https://registry.yarnpkg.com/cli-boxes/-/cli-boxes-2.2.1.tgz#ddd5035d25094fce220e9cab40a45840a440318f"
integrity sha512-y4coMcylgSCdVinjiDBuR8PCC2bLjyGTwEmPb9NHR/QaNU6EUOXcTY/s6VjGMD6ENSEaeQYHCY0GNGS5jfMwPw==

cli-chart@^0.1.0:
version "0.1.0"
resolved "https://registry.yarnpkg.com/cli-chart/-/cli-chart-0.1.0.tgz#eddaeb6558e8bb2020bc93efebbe245048abf5e1"
integrity sha512-8SxT0CbRZaPOPCLz0poBHYDKdVMx++EWTqU8FUvBYhTI/GrSQ6PX9ViGF25CK3VYPUACiFVuNeqc13E4XgXoHw==
dependencies:
charm "*"

cliui@^5.0.0:
version "5.0.0"
resolved "https://registry.yarnpkg.com/cliui/-/cliui-5.0.0.tgz#deefcfdb2e800784aa34f46fa08e06851c7bbbc5"
Expand Down Expand Up @@ -7740,7 +7774,7 @@ lodash.uniq@^4.5.0:
resolved "https://registry.yarnpkg.com/lodash.uniq/-/lodash.uniq-4.5.0.tgz#d0225373aeb652adc1bc82e4945339a842754773"
integrity sha512-xfBaXQd9ryd9dlSDvnvI0lvxfLJlYAZzXomUYzLKtUeOQvOP5piqAWuGtrhWeqaXK9hhoM/iyJc5AV+XfsX3HQ==

[email protected], lodash@^4.17.11, lodash@^4.17.14, lodash@^4.17.15, lodash@^4.17.19, lodash@^4.17.20, lodash@^4.17.21, lodash@^4.17.3, lodash@^4.17.5:
[email protected], lodash@^4.17.11, lodash@^4.17.14, lodash@^4.17.15, lodash@^4.17.19, lodash@^4.17.20, lodash@^4.17.21, lodash@^4.17.3, lodash@^4.17.4, lodash@^4.17.5:
version "4.17.21"
resolved "https://registry.yarnpkg.com/lodash/-/lodash-4.17.21.tgz#679591c564c3bffaae8454cf0b3df370c3d6911c"
integrity sha512-v2kDEe57lecTulaDIuNTPy3Ry4gLGJ6Z1O3vE1krgXZNrsQ+LFTGHVxVjcXPs17LhbZVGedAJv8XZ1tvj5FvSg==
Expand Down Expand Up @@ -8853,6 +8887,11 @@ [email protected]:
resolved "https://registry.yarnpkg.com/pkginfo/-/pkginfo-0.4.1.tgz#b5418ef0439de5425fc4995042dced14fb2a84ff"
integrity sha512-8xCNE/aT/EXKenuMDZ+xTVwkT8gsoHN2z/Q29l80u0ppGEXVvsKRzNMbtKhg8LS8k1tJLAHHylf6p4VFmP6XUQ==

platform@^1.3.3:
version "1.3.6"
resolved "https://registry.yarnpkg.com/platform/-/platform-1.3.6.tgz#48b4ce983164b209c2d45a107adb31f473a6e7a7"
integrity sha512-fnWVljUchTro6RiCFvCXBbNhJc2NijN7oIQxbwsyL0buWJPG85v81ehlHI9fXrJsMNgTofEoWIQeClKpgxFLrg==

pn@^1.1.0:
version "1.1.0"
resolved "https://registry.yarnpkg.com/pn/-/pn-1.1.0.tgz#e2f4cef0e219f463c179ab37463e4e1ecdccbafb"
Expand Down