Question: Managing File Retention and Deletion in Workflow

[IDmedia]

Self Checks

• I have searched for existing issues search for existing issues, including closed ones.
• I confirm that I am using English to submit report (我已阅读并同意 Language Policy).
• [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:）
• Please do not modify this template :) and fill in all the required fields.

Provide a description of requested docs changes

I have created a workflow that processes multiple PDF files uploaded via the "Start" node. These files are used in a Doc Extractor node. However, I’ve noticed that files accumulate quickly over time.

According to the Dify documentation:
"File Upload: Typically for temporary use, not stored long-term in the system."

Questions:
Is there a way to configure a file upload retention policy to automatically delete files after a certain period?
When files are uploaded and the workflow is executed via the API, I couldn’t find a DELETE endpoint to remove uploaded files using their file_id. Is such an endpoint available, or is there an alternative solution?

[dosubot[bot]]

Hello @IDmedia! I'm here to assist you with any bugs, questions, or contributions. Let's work together to resolve any issues you're facing.

To manage file retention and deletion in Dify workflows, you can use the clean_unused_datasets_task function, which is designed to clean up unused datasets by checking the age of documents and datasets, and then removing those that are no longer needed based on specified retention periods. This function uses SQL queries to identify datasets that have not been updated recently and are eligible for deletion, and it removes the associated indexes and updates the document status to disabled ^[1].

Regarding the deletion of uploaded files via the API, there isn't a specific DELETE endpoint for removing files by their file_id. However, you can delete documents using the following DELETE endpoint, which requires specifying the dataset_id and document_id:

curl --location --request DELETE 'https://api.dify.ai/v1/datasets/{dataset_id}/documents/{document_id}' \
--header 'Authorization: Bearer {api_key}'

This endpoint allows you to manage and delete documents effectively ^[2].

To continue talking to Dosu, mention @dosu.

[crazywoola]

According to the Dify documentation:
"File Upload: Typically for temporary use, not stored long-term in the system."

This is because the preview url is signed with token, and this token may not live for a very long time. This ensures the safety of the file itself. And this is a common practice as well.

If you want to remove files in s3 or some local machines. This can be done call this method.

dify/api/extensions/ext_storage.py

Lines 114 to 119 in 51db596

	def delete(self, filename):
	try:
	return self.storage_runner.delete(filename)
	except Exception as e:
	logging.exception(f"Failed to delete file {filename}")
	raise e

[wy96f]

@crazywoola We also have a similar issue. A workflow may generate a tool file, which is a temporary file used only for the current invocation. The file should be deleted after the invocation; otherwise, the user would have to write a tool to delete it and incorporate it into the workflow orchestration.