Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Doesn't support quantized model via bitsandbytes (8bit, 4bit) #108

Open
Dianagle2 opened this issue Mar 26, 2024 · 0 comments
Open

Doesn't support quantized model via bitsandbytes (8bit, 4bit) #108

Dianagle2 opened this issue Mar 26, 2024 · 0 comments

Comments

@Dianagle2
Copy link

Bug

ValueError: `.to` is not supported for `4-bit` or `8-bit` bitsandbytes models. Please use the model as it is, since the model has already been set to the correct devices and casted to the correct `dtype`.

To Reproduce
Quantize a HuggingFace model with bitsandbytes (for example 8-bit).
Draw_graph has:

ValueError                                Traceback (most recent call last)
File /usr/local/lib/python3.8/dist-packages/torchview/torchview.py:256, in forward_prop(model, x, device, model_graph, mode, **kwargs)
    255 if isinstance(x, (list, tuple)):
--> 256     _ = model.to(device)(*x, **kwargs)
    257     # _ = model

.to is not supported for this kind of model.

Expected behavior
To be able to draw the quantized graph.
Another point: I can successfully generate the graph using torchviz's make_dot.
Both repositories produce Graphviz representation of PyTorch autograd graph, so right now I use make_dot for quantized models.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant