Some Car files don't contain Metadata of transactions

[ring0-rootkit]

I've downloaded car file using tool provided in repo, created index for that file and validated index.
And some transactions doesn't seem to conatin Data field in its Metadata when they clearly should

Examples (tx_signatures):
2mLw3iS91aiSfd1a3j5nsfePv2VeVyqpv9RghCd4NqWPey5M2s33RHtzervBYW6UV7RUaTJx4WQeA7MPB3gWe6gC
solana explorer link

3HSEKYnB6PyDBBsZuyzE4R1ZFK5oDYLJbiSbvCXa3zgvj3H65X8Yh19bv9kaPaLqsnx4TBb2ZniCCRm7crhnd8NM
solana explorer link

Why is that hapening?

Commented that examples with more details for that issue.

[ring0-rootkit]

How do i know if this it is an error and not supposed to happen?
Let's look at 2 examples:

1. tx_signature = 2mLw3iS91aiSfd1a3j5nsfePv2VeVyqpv9RghCd4NqWPey5M2s33RHtzervBYW6UV7RUaTJx4WQeA7MPB3gWe6gC
   In my callback I recieve

&{Kind:0 Data:{Kind:6 Hash:0xc000158168 Index:0xc000158170 Total:0xc000158178 Data:AWQnnP5xsFig5yn6wmh0Wr/pkyaI7L1O7pUgA
POpkFnv6xMRayRd1NzUu2jgCkywYw+vkZNxLl/Dq5Z5z2tPZgKAAQADBhUY2hXRqCNVgwoR+eTvjGwL+SNDDNXdkYo1bhmKR8GVFQiXGT/AvUvnZhjIUkXJI
IyPyiB74unLSNptVCP/9pVFrT8c7oqC+Eve5TyR35BhqFF7RT6z/dVav/ZpVjNPiQMGRm/lIRcy/+ytunLDm+e8jOW7xfcSayxDmzpAAAAABHnVW/IxwG7ud
MVuzmgVB/2xst6j9I5RArHNola8E4+0P/on9df2SnTAmx8pWHneSwmrNt/J3VFLMhqns4zl6Isl6u3ovis/eIy+PDbaOOwlQH+oUO+KNI5swp7fwjihAwMAB
QJODwMAAwAJA9jWAAAAAAAABC4QAAEBBA8EBQQSEAYRBgcIBgYGBgYGBgYBAgATChMLDAIBFA8JEwAQEBUTDg0EJ+UXy5d6460qAgAAAAdkAAEmZAEAAKPhE
QAAAAAzGOIRAAAAAAEAAALhq80glKCAdE4s+U+ACQLJV1ezuAEN86ob4PvguR4bfgMhIB4EDAsaGPGqTKodlyHiIbL3aotX41zEfbc5RbNFjB9iHRBsNSFIB
hgfHR4ZGwMaFRY= Next:<nil>} Metadata:{Kind:6 Hash:0xc000158180 Index:0xc000158188 Total:0xc000158290 Data: Next:<nil>} S
lot:273731829 Index:0xc0001582d0}

and if we take a look at that tx in solana explorer we can clearly see that this tranaction contains both inner instructions and LUT that are both stored in metadata of transaction yet no Metadata from car file
2) tx_signature = 3HSEKYnB6PyDBBsZuyzE4R1ZFK5oDYLJbiSbvCXa3zgvj3H65X8Yh19bv9kaPaLqsnx4TBb2ZniCCRm7crhnd8NM

&{Kind:0 Data:{Kind:6 Hash:0xc0001584b8 Index:0xc0001584c0 Total:0xc0001584c8 Data:AW2BV/OcBcVHMCCwWM/4Nahtii+GfDjrKISOR
MS6udI5LxFTz5BSti1vD+Sw/cbfduK1cc6zXzCvyjoF5tijYgeAAQAECRUY2hXRqCNVgwoR+eTvjGwL+SNDDNXdkYo1bhmKR8GVFQiXGT/AvUvnZhjIUkXJI
IyPyiB74unLSNptVCP/9pUeaN5kus68AjKYrGm54MhR9j1Qa5Hpk8ARYe66qKvNIGg9zt1QlnFzYIs0OKvHEklbWA05MNuRwQBgOh1kAVup2hvFUbT8vylPG
rxdH9Q5Jq+IYVVPitkB1JAPEVHMn/QDBkZv5SEXMv/srbpyw5vnvIzlu8X3EmssQ5s6QAAAAAR51VvyMcBu7nTFbs5oFQf9sbLeo/SOUQKxzaJWvBOPBpuIV
/6rgYT7aH9jRhjANdrEOdwa6ztVmKDwAAAAAAG0P/on9df2SnTAmx8pWHneSwmrNt/J3VFLMhqns4zl6EsBt3589++WEPovMf30lOyxr1+O9aGU0qFbC0wgg
lwFAwUABQL1mgIABQAJA1DDAAAAAAAABicQAAEBBgcGCAYREAANAQwDDw4EAhQTEAoSCgkLCgoKCgoKCgoDAQAo5RfLl3rjrSoCAAAAEQFkAAEHZAEAgLLmD
gAAAAAqFOcOAAAAAAEAAALpW+6TP+PER/gghD/WvWKh/sOFMm1OHFr39tPB69VzdgPo7PAEC10MENcXVBFNkEuwsL4NFsBUTRk07rXDQvtPQgLA5TlOIor9B
NPS0M8B1Q== Next:<nil>} Metadata:{Kind:6 Hash:0xc0001584d0 Index:0xc0001584d8 Total:0xc0001584e8 Data: Next:<nil>} Slot:
273814910 Index:0xc0001584f0}

[ring0-rootkit]

More info about what I'm trying to do:
Currently I'm writing custom tool to parse that files to my custom format, in order to do that I pass function for parsing to main.FindTransactions as a callback of sorts and at that point I'm encountering an error, when passing callback I recieve *ipldbindcode.Transaction type and (in my case) after approx 300mil tx metadata of at least 50 mil transactions is nil
only 50 mil, 'cause It've been taking long time to parse and i thougt that 50 mil is enough (633 epoch)

[ring0-rootkit]

Here is a non failed transaction example (I mean it still doesn't have metadata but it succeeded as you can see in solana explorer)
2eHTmyH8cxrKbyHez8FKpc5KBpEjw3BgZtDW6mbs3QjBTyRvUiXW2UV5wFj1A9crYQx3Xjc9fDYia3o1Nx6LVh3C
solana explorer link

TX:
{[2eHTmyH8cxrKbyHez8FKpc5KBpEjw3BgZtDW6mbs3QjBTyRvUiXW2UV5wFj1A9crYQx3Xjc9fDYia3o1Nx6LVh3C] {1 [SRKsvE3LUK3E44rCbh3tiP5h
BWkkxmH6NrX7qe5RFdA 8DdB5hHSZtPT3oqbsiHUytCrrodApNC31k3MuZhxJH61 DTvGR2jRB8xKumc4Vc6mFSJN6hS2cQCFfoqFG9GFiVVA A2JcM4nEHx
15osc9mb86Ca3DdTDNr8ABWjQRGFdbtDuW FDFaHQHLzvNhVemLo9B14W8M1EKu1kK6MXsByVsZGv6C 9o8RDYsJdANgyPaVLFmTmEScA79S8LcDkPutgTgxhrtV ComputeBudget111111111111111111111111111111 dRiftyHA39MWEi3m9aunc5MzRF1JYuBsbn6VPcn33UH g6eRCbboSwK4tSWngn773RCMexr
1APQr4uA9bGZBYfo SkykkEdfPbNNZK2gYz7vK7XmXp8qeAeUuHSsA76kke1 PeNpQeGEm9UEFJ6MBCMauY4WW4h3YxoESPWbsqVKucE 9Qq91rsgDVNHahb
bejHFonQ9FtUvzyVbH3xkVj2wSvG9 7GGF99Et8AN8bjKug9nh6BJAQ3LKSziD2ceCps89zvQa] {1 0 7} 9F7SMXXQNNNZYGLhS1UGd6VZBcEj1hLmHrPJgauRRhYo [{6 [] JymE1m} {6 [] 3WBgs5fm8oDy} {7 [33 0 34 13] 9MYWPS5irMuzCgpv7P9Zufg6s} {7 [33 14 35 34 36 37 8 15 16 13 
17 18 1] eC7t7NtEmd8} {7 [33 19 35 34 36 37 8 15 16 13 17 18 1] eC7t7NtEmd8} {9 [0 0 38 39 33 19 20 21 14 22 15 40 16 13 41 42 34 43 7 44 45 14 23 24 25 2 3 26 27 22 28 28 28 4 5 29 30 46 46 46 46 46 46 46 46 46 46 46 46 46 46 46 46 31 32 3
4 35 37 36 8 10 47 48 49 50 15 16 11 51 52 53 54 55 12 13 18 17 1 34 35 37 36 8 10 47 48 49 50 15 16 11 51 52 53 54 55 12 13 18 17 1 56 40] 4FwWdsfVGqXBdJHp5L37mtthmiZEGBjbKnnRKPECJAhFuQreydiiRPia4fUKjRvwRAztfhBUMdHJHgbcKB2yocErW5E9bARMKZjN
7UeP8QebJvrCKQTkYXSJMizizN7cCwvkq12Ygda6qSEViVWBvejtF3F12GH67xgPHVfgqGcXyn1kB226S8wiXY5KJ}] [{EJ5jQ9LswYuUZdhMdDcpiuJEWU
NUzgZ2KERLMGSzSg1h [27 46 48 49 28 29 63 64 65 47 117 53 126 6 52 10 9 58 12 13] [45 50 14 17 18 43 44 31 73 33 2 4 5 41 21 19 92 20 24 26 22 94 25 30]}] map[] false}}
META:
<nil>

This one is parsed a little bit, I just devided meta from tx for the sake of readability

[anjor]

Sorry I am a bit confused. In the example with tx_signature 2mLw3iS91aiSfd1a3j5nsfePv2VeVyqpv9RghCd4NqWPey5M2s33RHtzervBYW6UV7RUaTJx4WQeA7MPB3gWe6gC the response does have some metadata:
Metadata:{Kind:6 Hash:0xc000158180 Index:0xc000158188 Total:0xc000158290 Data: Next:<nil>}

[ring0-rootkit]

Sorry I am a bit confused. In the example with tx_signature 2mLw3iS91aiSfd1a3j5nsfePv2VeVyqpv9RghCd4NqWPey5M2s33RHtzervBYW6UV7RUaTJx4WQeA7MPB3gWe6gC the response does have some metadata: Metadata:{Kind:6 Hash:0xc000158180 Index:0xc000158188 Total:0xc000158290 Data: Next:<nil>}

Yes it does contain metadata but Data field is empty and this is the field that actually contains data we need for parsing.
And for reference before I think 300mil+ transaction all transactions have something in it's Metadata : Data field but after that all (or almost all) don't

EDIT: specified it in main issue

[ring0-rootkit]

if I run following code:

        sig, err := solana.SignatureFromBase58("2T5DUh6D7XqQZDCH5xr4iWuTNijQMVaqsW2M7mkfaFHJooNouwd3M5A5VTWoDDDgegiEveWfoxM7ZfC
NQkRjVGS2")
        if err != nil {
                panic(err)
        }

        tx, _, err := epoch.GetTransaction(c.Context, sig)
        if err != nil {
                panic(err)
        }
        spew.Dump(tx)

(*ipldbindcode.Transaction)(0xc0004ec3c0)({                                                                                    
 Kind: (int) 0,                                                                                                                
 Data: (ipldbindcode.DataFrame) {                                                                                              
  Kind: (int) 6,                                                                                                               
  Hash: (**int)(0xc000654328)(<nil>),                                                                                          
  Index: (**int)(0xc000654338)(<nil>),                                                                                         
  Total: (**int)(0xc000654348)(<nil>),                                                                                         
  Data: (ipldbindcode.Buffer) (len=705 cap=705) AUiARv6TmGYUwpY+ki9TpiHZqzgJjolHZ1HyoIjl/UFbYmWaPQsQwuugpjFCoIB5f6WvRiYsv6rQfMR
NZvOuvAeAAQAFC+Q+o3nRL1bDDXh4mC3lul2qtsCNNVU9fRyvdMC0nfUG+lp+4sDBErYqlAL835IwTJzFnDPs4AjyMlw8IFh+BfND1Ls4l0LRtVDFHyH2vNzjSUaz5w
FJWh0TYtjSa5/TVMjEmidnzXi+bQccpIYcI8UhoOeV1+iKdACxAs5kmJdL2Mj7wtgZDqbuThmUpS8uVUL+UQkgsKrAkrEKceQ5nJaLZ8D5V4tBetN3ROx8UH298h+Rn
KH/YtFVaek3nqvtAgMGRm/lIRcy/+ytunLDm+e8jOW7xfcSayxDmzpAAAAAKuycb/NWpIJ+p+pTCyrAFCy3OhJBMNEcwSrg4Zu41HYAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAR51VvyMcBu7nTFbs5oFQf9sbLeo/SOUQKxzaJWvBOPtD/6J/XX9kp0wJsfKVh53ksJqzbfyd1RSzIap7OM5ehPehcJ857IhJtRmgLbwIYHZq6
hQQPV0wleXDqcMXUolwUGAAkD4jADAAAAAAAGAAUCoKkDAAcEAAECCBAnkOwTE0DXqM7ZBgAAAAAACS8UAAEBCRUJCgkWFAsXCwwNCwsLCwsLCwsBAwAYDhgPEAMBFR
kRGAAUFBoYBBITCSflF8uXeuOtKgIAAAAHZAABJmQBABxAxXQAAAAA7pIcdwAAAADhIAAHBQABAgUUGSjWm6w7tTYAztkGAAAAAAAFAAAAAAAAAAgCzsFfL1w4gH+ER
wOoaVC2tAGwawJIERqlrR63X+284SQDREhJBAIPCQom/qAPWIfjQpS+e2paLO8JXd5Xx2STA17cV0gSjOAjfwYwMjEvNjcDOTg0,                           
  Next: (**ipldbindcode.List__Link)(<nil>)                                                                                     
 },
 Metadata: (ipldbindcode.DataFrame) {
  Kind: (int) 6,
  Hash: (**int)(0xc000654358)(<nil>),
  Index: (**int)(0xc000654368)(<nil>),
  Total: (**int)(0xc000654378)(<nil>),
  Data: (ipldbindcode.Buffer) ,
  Next: (**ipldbindcode.List__Link)(<nil>)
 },
 Slot: (int) 273755926,
 Index: (**int)(0xc000654390->0xc0005935f0)(398)
})

for transaction with signature: 2T5DUh6D7XqQZDCH5xr4iWuTNijQMVaqsW2M7mkfaFHJooNouwd3M5A5VTWoDDDgegiEveWfoxM7ZfCNQkRjVGS2

[ring0-rootkit]

Some new info:
I just wanted to run the same code with 634 epoch and it started failing from the very start. I mean metadata is absent from very begining of 634 epoch

[gagliardetto]

Confirmed. Looking for root cause.