Linear layer adds bias term

[zadaianchuk]

self.key = nn.Linear(input_size, num_input_heads * input_query_size).to(self.device)
self.value = nn.Linear(input_size, num_input_heads * input_value_size).to(self.device)

The linear layer adds bias term by default, however, in formula 4 of the paper, it is no bias term. Is this intended? Or should I use

self.key = nn.Linear(input_size, num_input_heads * input_query_size, bias=False).to(self.device)
self.value = nn.Linear(input_size, num_input_heads * input_value_size, bias=False).to(self.device)

[dido1998]

Hi, I was able to get similar results as the paper with the bias term added so I don't think it makes that much difference. To exactly match the official implementation, these linear layers should be implemented using group operations, For example, see https://github.com/anirudh9119/RIMs/blob/master/event_based/attention.py#L111.

[pvsnp9]

In GroupLSTM cell, you have used GroupLinear as Linear Layer that does not use bias. Is there any reason that you choose not to use bias in grouped LSTMCells or grouped GRUCell ?