Keep name of the document in a pdf converter pipeline

[A-Goud]

Hello everyone,
I'm currently using project-lakechain to transform pdf or docx documents into txt document.
So I have S3 -> trigger -> pdfconverter/pandoc -> S3 connector -> S3
My problem is that the name of the output is kind of random while I want it like "entry_document_name.txt".
Moreover, is it possible to control where will go the document in the bucket ?

I didn't find any documentation on the subject

[HQarroum]

Hi @A-Goud! Thanks for getting back to us with your question.

You are right that there is no way to specify a "stable" name for the objects stored by the S3 Storage Connector. We've thought about allowing users to specify a template to apply to the naming of objects, but haven't pushed this feature yet due to security concerns.

What we would like to do is allow users to specify a prefix to the S3 Storage Connector controlling where to store the object in the bucket :

const connector = new S3StorageConnector.Builder()
  .withScope(this)
  .withIdentifier('S3StorageConnector')
  .withCacheStorage(cache)
  .withSource(source)
  .withDestinationBucket({
    bucket,
    prefix: 'images/'
   })
  .build();

For your use-case, since you need to know the name of the objects, would using S3 event notifications on the destination bucket allow you to store the objects on another system and fulfill your use-case? Or alternatively, using the SQS Storage Connector instead and control the writes to S3 yourself with a Lambda consuming from the SQS queue?

[A-Goud]

Hi HQarroum !

Thank you for your answer, I will try your solution with a lambda