Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Test balena push with docker-compose using symlinks #2408

Closed
wants to merge 1 commit into from
Closed

Test balena push with docker-compose using symlinks #2408

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
53 changes: 53 additions & 0 deletions tests/commands/push.spec.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -409,6 +409,59 @@ describe('balena push', function () {
});
});

it('should create the expected tar stream (docker-compose, symbolic links in docker-compose, --gitignore)', async () => {
const projectPath = path.join(
projectsPath,
'docker-compose',
'symbolic-links',
);
const expectedFiles: ExpectedTarStreamFiles = {
'.balena/balena.yml': { fileSize: 197, type: 'file' },
'docker-compose.yml': { fileSize: 340, type: 'file' },
'service1/Dockerfile.template': { fileSize: 144, type: 'file' },
'service1/file1.sh': { fileSize: 12, type: 'file' },
'service1/test-ignore-from-symlink-path.txt': {
fileSize: 12,
type: 'file',
},
'service2/Dockerfile-alt': { fileSize: 13, type: 'file' },
'service2/src/file1.sh': { fileSize: 12, type: 'file' },
};
const regSecretsPath = await addRegSecretsEntries(expectedFiles);
const responseFilename = 'build-POST-v3.json';
const responseBody = await fs.readFile(
path.join(builderResponsePath, responseFilename),
'utf8',
);
const expectedResponseLines: string[] = [
...commonResponseLines[responseFilename],
...getDockerignoreWarn1(
[path.join(projectPath, 'service2', '.dockerignore')],
'push',
),
];
if (isWindows) {
expectedResponseLines.push(
`[Info] Converting line endings CRLF -> LF for file: ${path.join(
projectPath,
'service2',
'file2-crlf.sh',
)}`,
);
}

await testPushBuildStream({
builderMock: builder,
commandLine: `push testApp -s ${projectPath} -R ${regSecretsPath}`,
expectedFiles,
expectedQueryParams: commonQueryParams,
expectedResponseLines,
projectPath,
responseBody,
responseCode: 200,
});
});

it('should create the expected tar stream (single container, --multi-dockerignore)', async () => {
const projectPath = path.join(
projectsPath,
Expand Down
7 changes: 7 additions & 0 deletions tests/test-data/projects/docker-compose/symbolic-links/.balena/balena.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
build-variables:
global:
- MY_VAR_1=This is a variable
- MY_VAR_2=Also a variable
services:
service1:
- SERVICE1_VAR=This is a service specific variable
3 changes: 3 additions & 0 deletions tests/test-data/projects/docker-compose/symbolic-links/.dockerignore
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
service1/test-ignore-from-actual-path.*
service1-symlink/test-ignore-from-symlink-path.*
**/.dockerignore
16 changes: 16 additions & 0 deletions tests/test-data/projects/docker-compose/symbolic-links/docker-compose.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
version: '2'
volumes:
resin-data:
services:
service1:
volumes:
- 'resin-data:/data'
build: ./service1-symlink
service2:
volumes:
- 'resin-data:/data'
build:
context: ./service2
dockerfile: Dockerfile-alt
args:
- 'COMPOSE_ARG=an argument defined in the docker-compose.yml file'
1 change: 1 addition & 0 deletions tests/test-data/projects/docker-compose/symbolic-links/service1
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
../symlink-fixtures/service1/
Copy link
Contributor

@pdcastro pdcastro Dec 13, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Build is failing on Windows with:

glob error [Error: EPERM: operation not permitted, scandir 'C:\var\lib\concourse\worker\volumes\live\a9a3a529-20a6-4d2c-748a-bf5059dd242b\volume\tests\test-data\projects\docker-compose\symbolic-links\service1'] {
  errno: -4048,
  code: 'EPERM',
  syscall: 'scandir',
  path: 'C:\\var\\lib\\concourse\\worker\\volumes\\live\\a9a3a529-20a6-4d2c-748a-bf5059dd242b\\volume\\tests\\test-data\\projects\\docker-compose\\symbolic-links\\service1'
}

It rings a bell that Windows doesn't play nicely with soft links, however out of the 5 existing tests that touch the dockerignore2 test project that uses soft links, I think only one is currently being skipped on Windows:

So maybe some investigation on how come that 4 other tests don't need to be skipped on Windows (maybe they don't really exercise the soft link aspect?) before skipping this test on Windows as well.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

how come that 4 other tests don't need to be skipped on Windows

This is the one that is completely skipped on Windows:

const itSkipWindows = process.platform === 'win32' ? it.skip : it;
itSkipWindows('should produce a compatible tar stream (symbolic links)'

This test is not skipped but it cheats :-) with:

			...(isWindows
				? {
						'lib/src-a.txt': { fileSize: 5, type: 'file' },
						'src/src-a.txt': { fileSize: 5, type: 'file' },
				  }
				: {}),

I think these 3 tests pass on Windows because git clone on Windows makes a real copy of the file that is pointed to by the symlink.

So, I think it would be fair to skip this test-data/projects/docker-compose/symbolic-links/service1 test case on Windows, for example with the itSkipWindows trick.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like it's related to symlinks to folders on windows 🤔
I will skip it for now since I don't have the bandwith to further investigate this, and it already adds value as is.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh wait, this might actually be a balena-lint bug :)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think so, the balena-lint issue was caused by the commit that added the itNoWin call.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This might work, based on the comments in that issue of glob:
See: balena-io-modules/node-balena-lint#82

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agrrrrr. I now fails is the test:source step.
Should I just drop the tests to get this fixed sooner than later?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See: #2410

And I will leave this PR having only the test case, just in case someone has the patience to fix them for Windows...

Copy link
Contributor

@pdcastro pdcastro Dec 14, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Before dropping the tests, how about:

  • Do not commit the service1 soft link to the git repo.
  • Have the new test case (added by this PR) dynamically create (and clean up) the soft link with fs.symlink that you had suggested in Flowdock.
  • Re-add itNoWin to the test case so that the soft link is never created on Windows.

Copy link
Member Author

@thgreasi thgreasi Dec 14, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

During my research I saw ppl facing such issues when creating symlinks as well, but at this point it sounds more promising than the committed one. I will add a reminder for later this week.

Note for self:
See: balanced-mt/serverless-plugin-typescript@0f57f0c

1 change: 1 addition & 0 deletions tests/test-data/projects/docker-compose/symbolic-links/service2/.dockerignore
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
**/file1.sh
1 change: 1 addition & 0 deletions tests/test-data/projects/docker-compose/symbolic-links/service2/Dockerfile-alt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
FROM busybox
2 changes: 2 additions & 0 deletions tests/test-data/projects/docker-compose/symbolic-links/service2/src/file1.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
line1
line2
3 changes: 3 additions & 0 deletions tests/test-data/projects/docker-compose/symlink-fixtures/service1/Dockerfile.template
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
FROM balenalib/%%BALENA_MACHINE_NAME%%-alpine
COPY ./file1.sh /
CMD i=1; while :; do echo "service1 $i $(uname -a)"; sleep 10; i=$((i+1)); done
2 changes: 2 additions & 0 deletions tests/test-data/projects/docker-compose/symlink-fixtures/service1/file1.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
line1
line2
1 change: 1 addition & 0 deletions ...t-data/projects/docker-compose/symlink-fixtures/service1/test-ignore-from-actual-path.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
test-ignore
1 change: 1 addition & 0 deletions ...-data/projects/docker-compose/symlink-fixtures/service1/test-ignore-from-symlink-path.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
test-ignore