Data Sources

Logscape can source its data from many different locations: your applications, Syslog and Logscape apps, as well as a variety of other sources. The key characteristic each source has in common is that the data is written to file and ingested. A Datasource specifies a directory, tag and file mask; which determines the files it should contain. Once declared, every Agent will begin to monitor the location within their file system for any matching files. Once a matching file is found, any existing data will be ingested and Indexed, and the file tailed to capture additional data in real time, once ingested it is forwarded to and indexed by an indexstore, and becomes searchable by the end user.

Indexing

All data that is searchable within logscape is ingested using the indexing process. Indexing is configured through Data sources. The process is to

1. On every indexing host (Manager, IndexStore) - track all Data Source Directory paths for existing or newly created files that match the filename masks.
2. For every file, monitor/tail it so as to detect changes, Every delta is processed for timestamp, discovered fields and timeseries indexing.
3. When files roll or are deleted the Tailing process detects the event and updates the index accordingly.
4. While indexing is occurring the data is immediately available for searching