Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: too many open files while creating a workspace #804

Open
wants to merge 4 commits into
base: main
Choose a base branch
from
Open

fix: too many open files while creating a workspace #804

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
1045ef5
fix: too many open files while creating a workspace
mahmuttaskiran Dec 14, 2024
ff85492
Triggering workflow
mahmuttaskiran Dec 17, 2024
c80c8e8
added 'no dependencies would change in' case to the matcher
mahmuttaskiran Dec 21, 2024
762fdb8
'Would change' case is added
mahmuttaskiran Dec 21, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
76 changes: 37 additions & 39 deletions packages/melos/lib/src/package.dart
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -531,54 +531,52 @@ class PackageMap {

final packageMap = <String, Package>{};

await Future.wait<void>(
pubspecFiles.map((pubspecFile) async {
final pubspecDirPath = pubspecFile.parent.path;
final pubspec = Pubspec.parse(pubspecFile.readAsStringSync());
final name = pubspec.name;

if (packageMap.containsKey(name)) {
throw MelosConfigException(
'''
await Stream.fromIterable(pubspecFiles).parallel((pubspecFile) async {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just realized that we should have a higher number here than the number of cores, since this isn't running on isolates. I don't know what a fitting number would be though, maybe try a few different and see how it affects performance?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For context, the project has 294 packages. I've run time melos list three times for each case below (default, 25, 50, and 100) and didn't see much improvement. The higher numbers will increase the risk of "too many open files" issue

Here are some benchmarks when parallelism:

  • was null (number of cores by default which is 10 on my device):
melos list  1.27s user 0.68s system 89% cpu 2.179 total
melos list  1.31s user 0.71s system 92% cpu 2.164 total
melos list  1.32s user 0.77s system 90% cpu 2.311 total
  • was 25:
melos list  1.32s user 0.72s system 68% cpu 2.968 total
melos list  1.31s user 0.74s system 89% cpu 2.285 total
melos list  1.28s user 0.72s system 91% cpu 2.204 total
  • was 50:
melos list  1.32s user 0.72s system 88% cpu 2.312 total
melos list  1.30s user 0.74s system 86% cpu 2.351 total
melos list  1.31s user 0.75s system 88% cpu 2.341 total
  • was 100:
melos list  1.32s user 0.74s system 86% cpu 2.375 total
melos list  1.30s user 0.73s system 91% cpu 2.220 total
melos list  1.31s user 0.73s system 89% cpu 2.290 total

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just out of curiosity, what does it say with 1?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The first three run:

melos list  1.27s user 0.75s system 54% cpu 3.752 total
melos list  1.26s user 0.70s system 90% cpu 2.161 total
melos list  1.29s user 0.72s system 90% cpu 2.212 total

Interesting :)

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems like we don't need parallelism then 😄

final pubspecDirPath = pubspecFile.parent.path;
final pubspec = Pubspec.parse(pubspecFile.readAsStringSync());
final name = pubspec.name;

if (packageMap.containsKey(name)) {
throw MelosConfigException(
'''
Multiple packages with the name `$name` found in the workspace, which is unsupported.
To fix this problem, consider renaming your packages to have a unique name.

The packages that caused the problem are:
- $name at ${printablePath(relativePath(pubspecDirPath, workspacePath))}
- $name at ${printablePath(relativePath(packageMap[name]!.path, workspacePath))}
''',
);
}
);
}

final filteredCategories = <String>[];

categories.forEach((key, value) {
final isCategoryMatching = value.any(
(category) => category.matches(
relativePath(pubspecDirPath, workspacePath),
),
);

if (isCategoryMatching) {
filteredCategories.add(key);
}
});

packageMap[name] = Package(
name: name,
path: pubspecDirPath,
pathRelativeToWorkspace: relativePath(pubspecDirPath, workspacePath),
version: pubspec.version ?? Version.none,
publishTo: pubspec.publishTo.let(Uri.parse),
packageMap: packageMap,
dependencies: pubspec.dependencies.keys.toList(),
devDependencies: pubspec.devDependencies.keys.toList(),
dependencyOverrides: pubspec.dependencyOverrides.keys.toList(),
pubspec: pubspec,
categories: filteredCategories,
final filteredCategories = <String>[];

categories.forEach((key, value) {
final isCategoryMatching = value.any(
(category) => category.matches(
relativePath(pubspecDirPath, workspacePath),
),
);
}),
);

if (isCategoryMatching) {
filteredCategories.add(key);
}
});

packageMap[name] = Package(
name: name,
path: pubspecDirPath,
pathRelativeToWorkspace: relativePath(pubspecDirPath, workspacePath),
version: pubspec.version ?? Version.none,
publishTo: pubspec.publishTo.let(Uri.parse),
packageMap: packageMap,
dependencies: pubspec.dependencies.keys.toList(),
devDependencies: pubspec.devDependencies.keys.toList(),
dependencyOverrides: pubspec.dependencyOverrides.keys.toList(),
pubspec: pubspec,
categories: filteredCategories,
);
}).drain<void>();

return PackageMap(packageMap, logger);
}
Expand Down
2 changes: 2 additions & 0 deletions packages/melos/test/matchers.dart
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,8 @@ Matcher ignoringDependencyMessages(String expected) {
!line.startsWith('Got dependencies!') &&
// Removes lines like " pub_updater 0.4.0 (0.5.0 available)"
!(line.startsWith(' ') && line.contains(' available)')) &&
!line.startsWith('No dependencies would change in') &&
!line.startsWith('Would change') &&
!line.contains(
'newer versions incompatible with dependency constraints',
) &&
Expand Down
Loading