Title
Purging Log Files from Local Path Based on Retention Date
Description
As a developer, I need to retain table-level log files in the local path only for a configured retention period so that disk space usage is optimized.
Currently, whenever the producer job runs, it creates a separate log file for each table. Over time, this results in excessive disk space consumption. Implementing a date-based log purging mechanism will ensure that outdated logs are removed automatically while retaining only the required logs.
Acceptance Criteria
-
Log files in the configured local path are retained only for the specified number of days.
-
Log files older than the configured retention period are automatically deleted.
-
The purging logic runs without impacting the producer job execution.
-
Only table-level log files are considered for purging; no other files are affected.
-
The retention period is configurable via application configuration (no hard-coding).
-
Proper logging is added to indicate successful and failed purge operations.
-
The solution is tested to ensure disk space usage is reduced after purging.