In pyvenv tracking mode how can I declare for a file that it uses NO virtual environment?

[halloleo]

I have setup

(pyvenv-activate "/my/venv/for/elpy")
(pyvenv-tracking-mode 1)

and this works great: Normally the elpy venv is active to do the elpy completion and in my python buffer I can use another venv defined via the buffer-local variable pyvenv-activate.

In some python buffer i would like to have no virtual env active not the elpyone nore another one. How can I set the local pyvenv-activate up for this?

I tried setting pyvenv-activate buffer locally to "", but this gives me a mysterious [tmp] venv and slows the editing in that buffer down to a crawl. So what's the right way to do it?