Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

problem of resource contention #16

Open
SchoIsles opened this issue Sep 20, 2017 · 4 comments
Open

problem of resource contention #16

SchoIsles opened this issue Sep 20, 2017 · 4 comments

Comments

@SchoIsles
Copy link

Hi,
while assign more than three pods to one kubelet node at one moment,caused vf resource contention,report error like “failed to move vf 16 to netns: no such device”
It looks like two pods allocFreeVF got same vf,the second pod try to use device of the vf but already linked into the first pod's ns.
does your production envirment scheduler doesn't consider this question?
and if I need to perfect the plugin by add some lock, or my use method is wrong?

@SchoIsles
Copy link
Author

add ex-lock in getVFDeviceName and unlock in setupVF resolved

@rkamudhan
Copy link
Collaborator

We didn't facing such issue. Can you explain your deployment model ?

@rkamudhan rkamudhan reopened this Sep 21, 2017
@SchoIsles
Copy link
Author

SchoIsles commented Sep 22, 2017

@rkamudhan
kubernetes 1.7.3, create an deployment with nginx image,2 kubelet node(256G memory, 40c)

scala replicas from 2 to 8, every node assign 3 pods, successful created 2, but the 3rd pod's network container failed because of "failed to move vf 16 to netns: no such device", change into the directory named virtfn16/net, there is blank. After I remove the failed container, kubelet recreate it successful

after add the lock, test replicas 0 to 60, all succeed

@hustcat
Copy link
Owner

hustcat commented Sep 25, 2017

I haven't test this, but I think this maybe happen when creating multiple pods at the same time. Thanks @okletswin

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants