Is it possible to forward files already stored on Telegram to the web?

[2640yang]

Since a large number of files have been stored in Telegram, it is inconvenient to download and upload them again.

[musahi0128]

I assume to the web is to the web UI. You can do it like this.

1. Forward the files to your TelDrive channel if the files is in another chat/group/channel
2. Get the message id. One way to do it is to forward the files from your TelDrive channel to @JsonDumpBot and note the forward_from_message_id
3. Add entry for the file in the files table on the TelDrive db

I believe your request was out of scope and therefore will not get a response from the developer. So here you have the basic idea on how to do it manually. I once imported 20k~ of files with similar approach.

Thanks

[wowdd1]

need this feature

[wes1993]

Hello @musahi0128 ,
Thanks for the suggestion, could you please explain better the procedure?
I have seen in the table files that there are many column but no message_id.. there is ID but seems a random generated value..
I hope in your help,
Best regards
Stefano

[musahi0128]

@wes1993 show me what you are looking at and I will do my best to clarify. Thanks

[wes1993]

@musahi0128
Thanks for yoru fast reply :-D
I have many files in the channel used by teldrive (Uploaded via Telegram API not with TelDrive), if is possibile i want to add this files to TelDrive.

My question is there is a script that reads all the information needed and INSERT in the TelDrive Files table?

Best Regards
Stefano

[orangejx]

I assume to the web is to the web UI. You can do it like this.

1. Forward the files to your TelDrive channel if the files is in another chat/group/channel
2. Get the message id. One way to do it is to forward the files from your TelDrive channel to @JsonDumpBot and note the forward_from_message_id
3. Add entry for the file in the files table on the TelDrive db

I believe your request was out of scope and therefore will not get a response from the developer. So here you have the basic idea on how to do it manually. I once imported 20k~ of files with similar approach.

Thanks

I can deduce other information according to the rules and fill in the data table files, but I do not know how to fill in the primary key ID in the files table. There is no associated information either in the data database or in the information given by the Json Dump Bot. How should I fill in the primary key ID in files?

[phanirithvij]

There is a way to add an existing file to teldrive if you have a beginner level of programming knowledge namely making http post calls

# To make a folder
curl 'http://localhost:8080/api/files' \
-X POST \
-H 'content-type: application/json' \
-H 'Cookie: user-session=eyJhbGciOiJQQkVTMi1IUzI1NitBMTI4S1ciLCJlbmMiOiJBMTI4Q0JDLUhTMjU2IiwicDJjIjo2MDAwMDAsInAycyI6Ii1DT0tSWV96eTFoY3NRWHVYTmlwVHcifQ.Pph_DMX9Lh7BvJXM5eWtQTYgirWaoGLoqU2JieDvoRU1UgaOsYjTiA.RpPlMU2aQ3JtQY0PwuJqBQ.t6uMB5xzHmSjFKUgw-SllzNA_QxCtjvnAFy3WXyLQe6Owe2xv6Oz-ZwLwYWYsRVZ6nihojYlfWY7StV0iTBpDhviC83EuJilACqSJpxqNr6kEsrQpnv46SnoxIK7J8ZHZLyeJgKYICGcoKtAwJFgJPPnimouixYEPT32M94LRMOo4hvG73Dkv9HFgkW8ELB6Gi2f3S71XFGMzooCJaLWjo7vDAQyxi6QSCQpGiD3OlMH2_FjRpIkJOzQPVFpp8yWasaFxkaoeQuEDQHZ1WSDsf0An4f3htSDtvMkICmVPfScy5uMryywHVXtlJ5XiM_yKIiFdkfswweIko3BrmJbhIzLADzjgORic4O4Cn0XnbNXRouTITBwan7FRRW-XN89Cvi2bIdl-sRCdXf4Yin-uHLZoGuXLl8HzHFqkAVcAh_BbjkMamatqaXgl-A7mnq7Rtvu3es2uEdT6yRswEwrk2CnN4pMvgRwuOUvPXPHhvS0Nygw6erNi3VJh7QIC_3oD-G32aLJMabOEJv7JTeVX7YVIi4WZw8YAF5FE5sdGi59I_JGI7Dfm1O-cS6plAiF6mdWBVEOZ6N-lG_E8m6dtpDiaU-COE7SF3RJ8tjx8oAqFnqXDQpx5ACH6I_kivfNZ4i1LaQeUZmvlOT1bhAFrnAW0XazSbdzw5k49PaWEKgdKcOUBqcGAx6UEw8SxQ3ETUCkq9oULZQaKO_b8N1zMA.3Zm3XIZScz7wsON0OkQ9Gw' \
--data-raw '{"name":"testfolder","type":"folder","path":"/"}'

# to add a file to that folder
curl 'http://localhost:8080/api/files' \
-X POST -H 'content-type: application/json' \
-H 'Cookie: user-session=eyJhbGciOiJQQkVTMi1IUzI1NitBMTI4S1ciLCJlbmMiOiJBMTI4Q0JDLUhTMjU2IiwicDJjIjo2MDAwMDAsInAycyI6Ii1DT0tSWV96eTFoY3NRWHVYTmlwVHcifQ.Pph_DMX9Lh7BvJXM5eWtQTYgirWaoGLoqU2JieDvoRU1UgaOsYjTiA.RpPlMU2aQ3JtQY0PwuJqBQ.t6uMB5xzHmSjFKUgw-SllzNA_QxCtjvnAFy3WXyLQe6Owe2xv6Oz-ZwLwYWYsRVZ6nihojYlfWY7StV0iTBpDhviC83EuJilACqSJpxqNr6kEsrQpnv46SnoxIK7J8ZHZLyeJgKYICGcoKtAwJFgJPPnimouixYEPT32M94LRMOo4hvG73Dkv9HFgkW8ELB6Gi2f3S71XFGMzooCJaLWjo7vDAQyxi6QSCQpGiD3OlMH2_FjRpIkJOzQPVFpp8yWasaFxkaoeQuEDQHZ1WSDsf0An4f3htSDtvMkICmVPfScy5uMryywHVXtlJ5XiM_yKIiFdkfswweIko3BrmJbhIzLADzjgORic4O4Cn0XnbNXRouTITBwan7FRRW-XN89Cvi2bIdl-sRCdXf4Yin-uHLZoGuXLl8HzHFqkAVcAh_BbjkMamatqaXgl-A7mnq7Rtvu3es2uEdT6yRswEwrk2CnN4pMvgRwuOUvPXPHhvS0Nygw6erNi3VJh7QIC_3oD-G32aLJMabOEJv7JTeVX7YVIi4WZw8YAF5FE5sdGi59I_JGI7Dfm1O-cS6plAiF6mdWBVEOZ6N-lG_E8m6dtpDiaU-COE7SF3RJ8tjx8oAqFnqXDQpx5ACH6I_kivfNZ4i1LaQeUZmvlOT1bhAFrnAW0XazSbdzw5k49PaWEKgdKcOUBqcGAx6UEw8SxQ3ETUCkq9oULZQaKO_b8N1zMA.3Zm3XIZScz7wsON0OkQ9Gw' \
--data-raw '{"name":"inside_bo_burnam_2022_outtakes.jpg","mimeType":"image/jpeg","type":"file","parts":[{"id":373,"salt":""}],"size":36321,"path":"/testfolder","encrypted":false,"channelId":0}'

First make directory then add the file. To make a directory /x/y/z need to make /x then /x/y then /x/y/z the api calls. If a directory already exisits no worries it will give 409 back.

Then to upload a file f.ext from channel channelId and size s to the folder /x/y/z you need to do '{"name":"f.ext","mimeType":"mimetype","type":"file","parts":[{"id":id,"salt":""}],"size": s,"path":"/x/y/z","encrypted":false,"channelId":channelId}'. id is the message id of a file. for eg. https://t.me/c/channelid/5213 id is 5213.

• By default channelid of 0 means the registered channel in teldrive settings.
• You can pass multiple values in the parts array which are of size 2000MB or 4000MB (prem) split files which will join together when downloading. i.e. "parts":[{"id":idofpart1,"salt":""},{"id":idofpart2,"salt":""},{"id":idofpart3,"salt":""},..].
• encrypted is always false unless you would like to add an exisiting ecrypted file in which case go through the code and figure out what to add as salt for each part.
• mimetype is video/x-matroska or vide/mp4 or image/jpeg or image/png etc. per the file type mkv/mp4/jpg/png. Easy way to determine, inspect element upload your file.flac see the post call for /api/files and check the mimetype.
• my cookie leaking here is fine because I am not running my teldrive publicly
• I obtained the cookie by inspect element in the teldrive web UI
• I obtained the channelid and size of a file from plus messenger or you can use tdl export which I am not going to go into details.
• The next step is to make two functions make_dirs and add_file in your programming language of choice which could even be bash if you like to use the curl command as is.
• I will not be providing any scripts please don't ask.
• I don't think teldrive will ever have this feature but it can be added, importing existing channel
• A better approach inserts to db directly but this approach is good enough I'd say.

[orangejx]

There is a way to add an existing file to teldrive if you have a beginner level of programming knowledge namely making http post calls

# To make a folder
curl 'http://localhost:8080/api/files' \
-X POST \
-H 'content-type: application/json' \
-H 'Cookie: user-session=eyJhbGciOiJQQkVTMi1IUzI1NitBMTI4S1ciLCJlbmMiOiJBMTI4Q0JDLUhTMjU2IiwicDJjIjo2MDAwMDAsInAycyI6Ii1DT0tSWV96eTFoY3NRWHVYTmlwVHcifQ.Pph_DMX9Lh7BvJXM5eWtQTYgirWaoGLoqU2JieDvoRU1UgaOsYjTiA.RpPlMU2aQ3JtQY0PwuJqBQ.t6uMB5xzHmSjFKUgw-SllzNA_QxCtjvnAFy3WXyLQe6Owe2xv6Oz-ZwLwYWYsRVZ6nihojYlfWY7StV0iTBpDhviC83EuJilACqSJpxqNr6kEsrQpnv46SnoxIK7J8ZHZLyeJgKYICGcoKtAwJFgJPPnimouixYEPT32M94LRMOo4hvG73Dkv9HFgkW8ELB6Gi2f3S71XFGMzooCJaLWjo7vDAQyxi6QSCQpGiD3OlMH2_FjRpIkJOzQPVFpp8yWasaFxkaoeQuEDQHZ1WSDsf0An4f3htSDtvMkICmVPfScy5uMryywHVXtlJ5XiM_yKIiFdkfswweIko3BrmJbhIzLADzjgORic4O4Cn0XnbNXRouTITBwan7FRRW-XN89Cvi2bIdl-sRCdXf4Yin-uHLZoGuXLl8HzHFqkAVcAh_BbjkMamatqaXgl-A7mnq7Rtvu3es2uEdT6yRswEwrk2CnN4pMvgRwuOUvPXPHhvS0Nygw6erNi3VJh7QIC_3oD-G32aLJMabOEJv7JTeVX7YVIi4WZw8YAF5FE5sdGi59I_JGI7Dfm1O-cS6plAiF6mdWBVEOZ6N-lG_E8m6dtpDiaU-COE7SF3RJ8tjx8oAqFnqXDQpx5ACH6I_kivfNZ4i1LaQeUZmvlOT1bhAFrnAW0XazSbdzw5k49PaWEKgdKcOUBqcGAx6UEw8SxQ3ETUCkq9oULZQaKO_b8N1zMA.3Zm3XIZScz7wsON0OkQ9Gw' \
--data-raw '{"name":"testfolder","type":"folder","path":"/"}'

# to add a file to that folder
curl 'http://localhost:8080/api/files' \
-X POST -H 'content-type: application/json' \
-H 'Cookie: user-session=eyJhbGciOiJQQkVTMi1IUzI1NitBMTI4S1ciLCJlbmMiOiJBMTI4Q0JDLUhTMjU2IiwicDJjIjo2MDAwMDAsInAycyI6Ii1DT0tSWV96eTFoY3NRWHVYTmlwVHcifQ.Pph_DMX9Lh7BvJXM5eWtQTYgirWaoGLoqU2JieDvoRU1UgaOsYjTiA.RpPlMU2aQ3JtQY0PwuJqBQ.t6uMB5xzHmSjFKUgw-SllzNA_QxCtjvnAFy3WXyLQe6Owe2xv6Oz-ZwLwYWYsRVZ6nihojYlfWY7StV0iTBpDhviC83EuJilACqSJpxqNr6kEsrQpnv46SnoxIK7J8ZHZLyeJgKYICGcoKtAwJFgJPPnimouixYEPT32M94LRMOo4hvG73Dkv9HFgkW8ELB6Gi2f3S71XFGMzooCJaLWjo7vDAQyxi6QSCQpGiD3OlMH2_FjRpIkJOzQPVFpp8yWasaFxkaoeQuEDQHZ1WSDsf0An4f3htSDtvMkICmVPfScy5uMryywHVXtlJ5XiM_yKIiFdkfswweIko3BrmJbhIzLADzjgORic4O4Cn0XnbNXRouTITBwan7FRRW-XN89Cvi2bIdl-sRCdXf4Yin-uHLZoGuXLl8HzHFqkAVcAh_BbjkMamatqaXgl-A7mnq7Rtvu3es2uEdT6yRswEwrk2CnN4pMvgRwuOUvPXPHhvS0Nygw6erNi3VJh7QIC_3oD-G32aLJMabOEJv7JTeVX7YVIi4WZw8YAF5FE5sdGi59I_JGI7Dfm1O-cS6plAiF6mdWBVEOZ6N-lG_E8m6dtpDiaU-COE7SF3RJ8tjx8oAqFnqXDQpx5ACH6I_kivfNZ4i1LaQeUZmvlOT1bhAFrnAW0XazSbdzw5k49PaWEKgdKcOUBqcGAx6UEw8SxQ3ETUCkq9oULZQaKO_b8N1zMA.3Zm3XIZScz7wsON0OkQ9Gw' \
--data-raw '{"name":"inside_bo_burnam_2022_outtakes.jpg","mimeType":"image/jpeg","type":"file","parts":[{"id":373,"salt":""}],"size":36321,"path":"/testfolder","encrypted":false,"channelId":0}'

First make directory then add the file. To make a directory /x/y/z need to make /x then /x/y then /x/y/z the api calls. If a directory already exisits no worries it will give 409 back.

Then to upload a file f.ext from channel channelId and size s to the folder /x/y/z you need to do '{"name":"f.ext","mimeType":"mimetype","type":"file","parts":[{"id":id,"salt":""}],"size": s,"path":"/x/y/z","encrypted":false,"channelId":channelId}'. id is the message id of a file. for eg. https://t.me/c/channelid/5213 id is 5213.

• By default channelid of 0 means the registered channel in teldrive settings.
• You can pass multiple values in the parts array which are of size 2000MB or 4000MB (prem) split files which will join together when downloading. i.e. "parts":[{"id":idofpart1,"salt":""},{"id":idofpart2,"salt":""},{"id":idofpart3,"salt":""},..].
• encrypted is always false unless you would like to add an exisiting ecrypted file in which case go through the code and figure out what to add as salt for each part.
• mimetype is video/x-matroska or vide/mp4 or image/jpeg or image/png etc. per the file type mkv/mp4/jpg/png. Easy way to determine, inspect element upload your file.flac see the post call for /api/files and check the mimetype.
• my cookie leaking here is fine because I am not running my teldrive publicly
• I obtained the cookie by inspect element in the teldrive web UI
• I obtained the channelid and size of a file from plus messenger or you can use tdl export which I am not going to go into details.
• The next step is to make two functions make_dirs and add_file in your programming language of choice which could even be bash if you like to use the curl command as is.
• I will not be providing any scripts please don't ask.
• I don't think teldrive will ever have this feature but it can be added, importing existing channel
• A better approach inserts to db directly but this approach is good enough I'd say.

Thank you so much for your response & your method; And your solution completely solved my question. I dont know this repo have these APIs If you don't talk it

[bolavefasfas]

if the teledrive channel and the rclone upload channel are same shoudnt it show files in the teldrive webui and rclone in both places ?

havent tried asking ?

[akzzy]

There is a way to add an existing file to teldrive if you have a beginner level of programming knowledge namely making http post calls

# To make a folder
curl 'http://localhost:8080/api/files' \
-X POST \
-H 'content-type: application/json' \
-H 'Cookie: user-session=eyJhbGciOiJQQkVTMi1IUzI1NitBMTI4S1ciLCJlbmMiOiJBMTI4Q0JDLUhTMjU2IiwicDJjIjo2MDAwMDAsInAycyI6Ii1DT0tSWV96eTFoY3NRWHVYTmlwVHcifQ.Pph_DMX9Lh7BvJXM5eWtQTYgirWaoGLoqU2JieDvoRU1UgaOsYjTiA.RpPlMU2aQ3JtQY0PwuJqBQ.t6uMB5xzHmSjFKUgw-SllzNA_QxCtjvnAFy3WXyLQe6Owe2xv6Oz-ZwLwYWYsRVZ6nihojYlfWY7StV0iTBpDhviC83EuJilACqSJpxqNr6kEsrQpnv46SnoxIK7J8ZHZLyeJgKYICGcoKtAwJFgJPPnimouixYEPT32M94LRMOo4hvG73Dkv9HFgkW8ELB6Gi2f3S71XFGMzooCJaLWjo7vDAQyxi6QSCQpGiD3OlMH2_FjRpIkJOzQPVFpp8yWasaFxkaoeQuEDQHZ1WSDsf0An4f3htSDtvMkICmVPfScy5uMryywHVXtlJ5XiM_yKIiFdkfswweIko3BrmJbhIzLADzjgORic4O4Cn0XnbNXRouTITBwan7FRRW-XN89Cvi2bIdl-sRCdXf4Yin-uHLZoGuXLl8HzHFqkAVcAh_BbjkMamatqaXgl-A7mnq7Rtvu3es2uEdT6yRswEwrk2CnN4pMvgRwuOUvPXPHhvS0Nygw6erNi3VJh7QIC_3oD-G32aLJMabOEJv7JTeVX7YVIi4WZw8YAF5FE5sdGi59I_JGI7Dfm1O-cS6plAiF6mdWBVEOZ6N-lG_E8m6dtpDiaU-COE7SF3RJ8tjx8oAqFnqXDQpx5ACH6I_kivfNZ4i1LaQeUZmvlOT1bhAFrnAW0XazSbdzw5k49PaWEKgdKcOUBqcGAx6UEw8SxQ3ETUCkq9oULZQaKO_b8N1zMA.3Zm3XIZScz7wsON0OkQ9Gw' \
--data-raw '{"name":"testfolder","type":"folder","path":"/"}'

# to add a file to that folder
curl 'http://localhost:8080/api/files' \
-X POST -H 'content-type: application/json' \
-H 'Cookie: user-session=eyJhbGciOiJQQkVTMi1IUzI1NitBMTI4S1ciLCJlbmMiOiJBMTI4Q0JDLUhTMjU2IiwicDJjIjo2MDAwMDAsInAycyI6Ii1DT0tSWV96eTFoY3NRWHVYTmlwVHcifQ.Pph_DMX9Lh7BvJXM5eWtQTYgirWaoGLoqU2JieDvoRU1UgaOsYjTiA.RpPlMU2aQ3JtQY0PwuJqBQ.t6uMB5xzHmSjFKUgw-SllzNA_QxCtjvnAFy3WXyLQe6Owe2xv6Oz-ZwLwYWYsRVZ6nihojYlfWY7StV0iTBpDhviC83EuJilACqSJpxqNr6kEsrQpnv46SnoxIK7J8ZHZLyeJgKYICGcoKtAwJFgJPPnimouixYEPT32M94LRMOo4hvG73Dkv9HFgkW8ELB6Gi2f3S71XFGMzooCJaLWjo7vDAQyxi6QSCQpGiD3OlMH2_FjRpIkJOzQPVFpp8yWasaFxkaoeQuEDQHZ1WSDsf0An4f3htSDtvMkICmVPfScy5uMryywHVXtlJ5XiM_yKIiFdkfswweIko3BrmJbhIzLADzjgORic4O4Cn0XnbNXRouTITBwan7FRRW-XN89Cvi2bIdl-sRCdXf4Yin-uHLZoGuXLl8HzHFqkAVcAh_BbjkMamatqaXgl-A7mnq7Rtvu3es2uEdT6yRswEwrk2CnN4pMvgRwuOUvPXPHhvS0Nygw6erNi3VJh7QIC_3oD-G32aLJMabOEJv7JTeVX7YVIi4WZw8YAF5FE5sdGi59I_JGI7Dfm1O-cS6plAiF6mdWBVEOZ6N-lG_E8m6dtpDiaU-COE7SF3RJ8tjx8oAqFnqXDQpx5ACH6I_kivfNZ4i1LaQeUZmvlOT1bhAFrnAW0XazSbdzw5k49PaWEKgdKcOUBqcGAx6UEw8SxQ3ETUCkq9oULZQaKO_b8N1zMA.3Zm3XIZScz7wsON0OkQ9Gw' \
--data-raw '{"name":"inside_bo_burnam_2022_outtakes.jpg","mimeType":"image/jpeg","type":"file","parts":[{"id":373,"salt":""}],"size":36321,"path":"/testfolder","encrypted":false,"channelId":0}'

First make directory then add the file. To make a directory /x/y/z need to make /x then /x/y then /x/y/z the api calls. If a directory already exisits no worries it will give 409 back.

Then to upload a file f.ext from channel channelId and size s to the folder /x/y/z you need to do '{"name":"f.ext","mimeType":"mimetype","type":"file","parts":[{"id":id,"salt":""}],"size": s,"path":"/x/y/z","encrypted":false,"channelId":channelId}'. id is the message id of a file. for eg. https://t.me/c/channelid/5213 id is 5213.

• By default channelid of 0 means the registered channel in teldrive settings.
• You can pass multiple values in the parts array which are of size 2000MB or 4000MB (prem) split files which will join together when downloading. i.e. "parts":[{"id":idofpart1,"salt":""},{"id":idofpart2,"salt":""},{"id":idofpart3,"salt":""},..].
• encrypted is always false unless you would like to add an exisiting ecrypted file in which case go through the code and figure out what to add as salt for each part.
• mimetype is video/x-matroska or vide/mp4 or image/jpeg or image/png etc. per the file type mkv/mp4/jpg/png. Easy way to determine, inspect element upload your file.flac see the post call for /api/files and check the mimetype.
• my cookie leaking here is fine because I am not running my teldrive publicly
• I obtained the cookie by inspect element in the teldrive web UI
• I obtained the channelid and size of a file from plus messenger or you can use tdl export which I am not going to go into details.
• The next step is to make two functions make_dirs and add_file in your programming language of choice which could even be bash if you like to use the curl command as is.
• I will not be providing any scripts please don't ask.
• I don't think teldrive will ever have this feature but it can be added, importing existing channel
• A better approach inserts to db directly but this approach is good enough I'd say.

Your method actually working. i can create folder and upload files. but the problem comes when i download those files. teldrive showing these errors when downloadin the files i upload using this method.

2024/09/08 14:49:07 [Recovery] 2024/09/08 - 14:49:07 panic recovered: runtime error: index out of range [1] with length 1 runtime/panic.go:115 (0x91b373) github.com/divyam234/teldrive/internal/reader/reader.go:131 (0x1c3a14f) github.com/divyam234/teldrive/internal/reader/reader.go:112 (0x1c39c05) github.com/divyam234/teldrive/internal/reader/reader.go:78 (0x1c39c13) github.com/divyam234/teldrive/pkg/services/file.go:785 (0x1c769a6) github.com/divyam234/teldrive/pkg/services/file.go:809 (0x1c75d81) github.com/divyam234/teldrive/pkg/controller/file.go:203 (0x1c82fad) github.com/gin-gonic/[email protected]/context.go:185 (0x1d56f79) github.com/divyam234/teldrive/cmd/run.go:273 (0x1d56f67) github.com/gin-gonic/[email protected]/context.go:185 (0x102e10a) github.com/gin-contrib/[email protected]/zap.go:68 (0x1d56130) github.com/gin-gonic/[email protected]/context.go:185 (0x103b4ce) github.com/gin-gonic/[email protected]/recovery.go:102 (0x103b4bb) github.com/gin-gonic/[email protected]/context.go:185 (0x103a271) github.com/gin-gonic/[email protected]/gin.go:633 (0x1039ce0) github.com/gin-gonic/[email protected]/gin.go:589 (0x1039811) net/http/server.go:3210 (0xcb2dad) net/http/server.go:2092 (0xc922cf) runtime/asm_amd64.s:1700 (0x95d920)

[phanirithvij]

Maybe something changed in the recent version. You can upload a file from the browser and see the network tab to see what's being sent.

[akzzy]

Maybe something changed in the recent version. You can upload a file from the browser and see the network tab to see what's being sent.

Everything looks normal when uploading from browser and downloading the same file from browser.
uploading log

[doomwithdon]

Is this feature being considered for the project or any easier way to automate it? Apart from the message id, other values needed for the api call or DB insert are difficult to find.