Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Question]: Does vertical_slash_sparse_attention supported to concatenate all batches into a single row for operation like flash_attn_2_cuda.varlen_fwd? #46

Open
Amanda-Barbara opened this issue Jul 17, 2024 · 1 comment
Open

[Question]: Does vertical_slash_sparse_attention supported to concatenate all batches into a single row for operation like flash_attn_2_cuda.varlen_fwd? #46

Amanda-Barbara opened this issue Jul 17, 2024 · 1 comment
Assignees
@Starmys
Labels
question Further information is requested

Comments

@Amanda-Barbara
Copy link

Describe the issue

Does vertical_slash_sparse_attention/block_sparse_attention/streaming_forward supported to concatenate all batches into a single row for operation like flash_attn_2_cuda.varlen_fwd?
@Amanda-Barbara Amanda-Barbara added the question Further information is requested label Jul 17, 2024
@iofu728
Copy link
Contributor

iofu728 commented Jul 18, 2024

Hi @Amanda-Barbara, thanks for your question.

Our three kernels supports multi-batch input, but currently, it does not support variable-length sequences. However, this is something that can be implemented.

@polarispw polarispw mentioned this issue Jul 23, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@Starmys Starmys

Labels
question Further information is requested
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@iofu728 @Starmys @Amanda-Barbara