Archiving Policy

Forwarder Retention Rule

The forwarder will hold data for a predefined number of days. By default this is set to indefinite. If this parameter is set, the forwarder will delete the data after N number of days. The minimum data retention period for the forwarder is 2 days. If a file is deleted within the 2 day period the Index Store will retain a timestamped copy recording an unauthorized delete operation.

Index Store Retention

The Index Store will apply an archiving operation after the defined number of days. The Index Store can be configured to compress the data using

• gzip compression - Slow compression and slow decompression with a high compression ratio of about 10% of the original size for text files. Consequently, searches are slower on gzipped data.
• snappy compression - Fast compression and decompression but results in larger compressed files about 20% of the original file size. Provides faster search of compressed data.

The archive operation can be set to [DELETE]. Which will delete the data of the Index Store after a number of days. Care must be taken with this option since they can lead to data loss if this is set incorrectly.

Settings

The Forwarder and Index Store retention rule can be enabled and configured on each data source.

Setting a value of 0 disables the archiving rule for that component.

It is important to note that your filemask much capture the extension after any archiving rules are applied to prevent loss of data.

The first number refers to the Forwarder. When this is set any files belonging to this data source will be deleted off the Forwarder. The second number refers to the Index Store. After the set number of days the Index Store will perform an operation on the data. The operation is configured with the drop down menu next to it. The data can either be compressed or deleted.