Getting data in

Vigilfield stores your security data in tables and fills them from sources. You query tables with VFQL.

The two pieces

• A source is an ingestion feed. It knows where the logs come from (an S3 bucket, a CloudWatch Logs log group, an EventBridge bus or an SQS queue), how to parse them, and which columns to extract. See Sources.
• A table is where the rows land. It holds a storage schema: column names, types and whether each column may be null. See Tables.

A source feeds at most one table. A table can be fed by many sources, so you can send CloudTrail from several AWS accounts into one table.

Attach means on, detach means off

A source only ingests while it is attached to a table. There is no separate pause switch:

• Attach a source to a table to start ingestion.
• Detach it to stop ingestion. The source and its settings stay, and you can attach it again later.

When you attach, Vigilfield checks that the table can store what the source produces. See Schema compatibility.

The usual path

1. Go to Sources and click New Source.
2. Pick a template, such as CloudTrail or VPC Flow Logs, or pick Custom.
3. Enter where the logs live.
4. On the Destination step, choose Create a new table. Vigilfield creates a table that matches the source and attaches the source to it in one step.
5. For S3, CloudWatch Logs and SQS sources, deploy the role Vigilfield shows you in the AWS account that holds the logs, then click Verify.
6. Query the table:

cloudtrail | take 100

Where the data lives

Ingested data is stored in a dedicated AWS account that Vigilfield operates for your organization. By default it is encrypted under a per-organization key that Vigilfield manages. You can switch to your own AWS KMS key. See Storage key.