Data Flow Overview

Sources

Data Sources pull in log data from various locations , from your filesystems, network storage and applications.

Services

Services collect data from using application apis. Services are executed by each Forwarder or Indexer collecting OS KPIs, Database metrics or Applicatin Service API data

Learn More

Syslog

The Syslog format and protocol is supported by several applications, network devices and network disks to report system health.

Learn More

File Systems

Logs are imported from the file system

Learn More

Indexing

Logscape ingests the data and indexes, optimizing for search. As the data is ingested Logscape learns the structure of the data and extracts known Key Value patterns



Ingestion

When a data source is created files are ingested. This is a process where data is imported and indexed. During this process Logscape will extract common Key Value patterns and making these fields available at search. Typically json and xml data is recognised but other types of attributes are supported.

Learn More

Data Types

Types are assigned to incoming data. A type is a loose schema that is applied on top of log or application data. The schema exposes fields which can be used at search time.

Learn More.

Data Sources

Data Sources describe the location of incoming data. Each data source in the system will have at least on 'tag' assigned to it. Think of a tag as a categorization or a label for your data. Multiple tags can be assigned to each data sources allowing a rich taxonomy to be modelled around the data.

Learn More



Using the search example above, we can see that the search is filtered to the datasource tagged iis.This tag points to data located in C:\inetpub\logs\
or wherever the data for IIS web access logs have been configured.

The iis web access log data has been assigned the type www The www Datatype extracts fields such as

  • request
  • ,httpCode
  • ,userAgent
  • ,host
  • ,bytes
  • ....
from the web log data.

The UserAgent   field is in the example filters the results to consider only the user agents that contain the substring 'Safari'..
The _tag field narrows the search to data defined by theiis  tag. Any text before the '|' symbol is interpreted as a keyword filter.
In this example we look for failed web requests