PCC drop in detr detection variant

[meenakshiramanathan1]

Description

• notable pcc drop of 0.9707041228441168 is observed in the object detection variant in tvm. Encoder block and decoder block is running separately without any pcc drop but when encoder and decoder are implemented together, pcc drop is noticed.

Reproduce

git checkout mramanathan/detr_data_mismatch

# to run whole model 
pytest forge/test/models/pytorch/vision/detr/test_detr.py::test_detr_detection 

Observed Behaviour

rame_outputs = [tensor([[-1.2648,  1.1114,  0.0108,  ...,  2.3999,  1.7646, -1.9175],
        [-1.0903, -0.4668,  0.8739,  ..., -1.03...20,  4.4414],
        [-1.7141,  0.9052,  0.6864,  ..., -1.3129,  0.0210,  2.6208]],
       grad_fn=<SelectBackward0>)]
forge_outputs = (Forge Tensor: tensor([[-0.9232,  0.9180,  0.3173,  ...,  2.7896,  1.8620, -1.8788],
        [-0.5872, -0.4872,  1.129...   [-1.8014,  0.9211,  0.5087,  ..., -1.4547,  0.0652,  2.4845]],
       grad_fn=<ViewBackward0>), DataFormat.Float32,)
verify_cfg = DepricatedVerifyConfig(graph_name='graph', enabled=False, intermediates=False, rtol={torch.float32: None, torch.float1...ent_checking=True, enable_parameter_gradient_checking=True, _input_gradient_queue=None, _parameter_gradient_queue=None)

    def verify_framework_vs_forge_codegen(frame_outputs, forge_outputs, verify_cfg):
        from forge.op.eval import compare_tensor_to_golden
    
        test_pass = True
        for i, (golden, output) in enumerate(zip(frame_outputs, forge_outputs)):
            test_pass &= compare_tensor_to_golden(
                f"Framework vs. Forge codegen output {i}", golden, output.value(), is_forge=False, verify_cfg=verify_cfg
            )
    
>           assert test_pass, f"Data mismatch on output {i} between framework and Forge codegen"
E           AssertionError: Data mismatch on output 0 between framework and Forge codegen

Logs

detr.log