Sources

A source is one ingestion feed. It owns the whole read pipeline: where the data comes from, how it is parsed, and which columns are extracted. It writes into the table it is attached to.

Open Sources in the sidebar to see every source you can read. Each card shows the source's transport, its template and version, and the table it is attached to, or Detached. The counters at the top show Total Sources, Attached, Detached and Provisioning Issues (sources that are not yet active).

Templates

A template is a ready-made source definition for a common AWS log type. It supplies the parser and the output columns, so you only enter where the logs live.

The wizard only offers the transports a template supports. SQS Queue is available for Custom sources only.

Pick Custom when no template fits. You then choose the input format and declare the output columns yourself.

The template catalog is also available through the API: list templates.

Transports

For the three role-based transports, each source gets its own IAM role in the AWS account that holds the logs. Its permissions cover exactly the bucket (and prefix), log group or queue you named, and nothing wider.

Create a source

1. Go to Sources and click New Source.
2. Type. Choose a Source Template. A template pre-fills the source name, the description and the new table name.
3. Pipeline. Fill in:
   • Source Name (required). Names are unique within your organization, ignoring case.
   • Owner Team (required). All members of the owner team can manage this source.
   • Description (optional).
   • Ingestion Mechanism, then the fields for that transport (see below).
   • For Custom sources: Input Format and On Extraction Failure.
   • For VPC Flow Logs: an optional Custom format string. Paste the format string from your VPC Flow Log configuration. Leave it blank if you use the AWS default field order.
4. Output Schema (Custom only). Declare each column: name, type, whether it may be null, and the VRL expression that extracts it from the raw record, for example .field_name. You need at least one column, and every column needs a name and an extraction expression.
5. Destination. Choose where the data lands:
   • Create a new table. Vigilfield creates a table whose schema matches the source's output and attaches the source to it. Enter a Table Name.
   • Attach to an existing table. Pick a table. Vigilfield checks compatibility when it attaches.
   • Leave detached. The source is created, but nothing is ingested until you attach it.
6. Click Create Source. The app opens the new source's page.

API: create a source.

Fields per transport

Amazon S3, CloudWatch Logs and SQS Queue all ask for:

• AWS Account ID: the 12-digit account the source's role is created in.
• AWS Region: the region the resource lives in, for example us-east-1.

Then:

• Amazon S3: S3 Bucket (the bucket name, not an ARN) and an optional Prefix Filter. The role is scoped to that prefix. The prefix cannot contain *, ? or $.
• CloudWatch Logs: Log Group ARN, in the form arn:aws:logs:<region>:<account>:log-group:<name>. A trailing :* is optional.
• SQS Queue: Queue ARN, in the form arn:aws:sqs:<region>:<account>:<name>.

The account and region inside a log group or queue ARN must match the AWS Account ID and AWS Region you entered.

EventBridge asks for:

• Publishing Account ID: the 12-digit AWS account whose events these are. Vigilfield checks it against the sender of every message.
• Event Bus ARN: the bus your rules publish to.

Input format for Custom sources

A template picks its own parser. A Custom source uses Input Format, which defaults to JSON. The list of formats comes from list deserializers.

Finish setup: deploy the role and verify

An S3, CloudWatch Logs or SQS source starts out waiting for its role, with a Provisioning badge. Its page shows Finish setting up this source with these documents, each with Copy and Download:

• Identity provider: deploy once per AWS account, before any Vigilfield role. It is shown only when the account does not have one yet.
• Role template: a CloudFormation template that creates the source's role.
• Trust policy and Permissions policy: the same role as JSON, for review. The permissions policy is exactly what the role grants.

Deploy the role in the AWS account that holds the logs, then click Verify. Vigilfield assumes the role and makes the read calls the transport needs. When they succeed, the source becomes active and ingestion can start. If a check fails, the page shows the reason. Fix it and click Verify again. Verifying a source that is already active changes nothing.

API: get the role template, verify.

Set up EventBridge delivery

An EventBridge source needs no role. Instead, its page shows an EventBridge Setup card. Create a rule on your event bus that targets Vigilfield's queue, using the values on the card:

• Target queue ARN
• Source marker
• Input paths map
• Input template

API: get the EventBridge setup.

Attach and detach

The Attachment card on the source page controls ingestion.

• To attach, choose a Table and click Attach. Only tables filled by sources are offered. Vigilfield runs the compatibility check. If it fails, the reasons appear under the button.
• If the source produces fields the table has no column for, the attach succeeds with Attached with warnings. Those fields are dropped on ingest. Click Add columns to table to add each one to the table as a nullable column instead.
• To detach, click Detach. Ingestion stops. The source keeps its settings.

A source attached to one table cannot be attached to a different one. Detach it first.

You can also attach a detached source from a table's page, under Sources.

API: attach, detach.

Monitor a source

Ingestion Activity shows three hourly charts for the last 7 days: Ingested bytes, Ingested records and Ingestion errors. API: source metrics.

Advisories appear at the top of the page when Vigilfield has something to tell you about the source's setup.

If setup failed, a Provisioning failed message shows the reason.

Freshness (CloudWatch Logs)

CloudWatch Logs sources have an Ingestion Freshness card. It shows how far behind real time the source is, for example 3m behind, and a status:

A Quarantined source stays paused until you Detach it and attach it again. Fix the cause first, or it will be quarantined again.

The card also shows the log group's observed retention, when known.

API: source freshness.

Replay a window (CloudWatch Logs, admins)

Admins can re-read a time range from a CloudWatch Logs source. On the Ingestion Freshness card, click Replay window, enter From (UTC) and To (UTC), and click Replay.

• Times are UTC.
• The source must be attached and active.
• The range is split into 5-minute slices. One request can cover at most 8,640 slices, which is 30 days.
• The end must be after the start.
• The app states that replaying overlapping ranges is safe, because duplicate events are absorbed.

API: replay.

Template upgrades

A template-based source is pinned to the template version it was created with, shown on its badge, for example CloudTrail v2. When a newer version ships, the source page shows Template upgrade available with a column summary.

• Click Review changes to see added, removed and changed columns, and any OCSF mapping change.
• Tick Keep my customizations to keep columns you added to the source. Template columns are still upgraded. Without it, the source's columns are replaced by the new template's.
• Click Apply upgrade. If columns would be removed or the OCSF mapping changes, you must confirm first. This cannot be undone.

Upgrades are never applied automatically. Custom sources have no template and never show this banner.

API: check for an upgrade, apply an upgrade.

Edit a source

Click Edit to change the Name and Description.

You cannot change the transport settings (account, region, bucket, log group, queue or event bus) in place. Detach the source and create a new one instead.

API: update a source.

Delete a source

Click Delete and confirm. The source's pipeline settings are deleted permanently. Data already ingested into tables is kept.

API: delete a source.

Extraction failures

When a record cannot be extracted, the default setting, Skip, drops it.

Custom sources let you pick another value for On Extraction Failure. If you choose Nullify, Vigilfield assumes any field may be null when it checks compatibility. Such a source cannot attach to a table with a required (non-nullable) column that it fills.

The source page shows the current setting under Pipeline, as On extraction failure.

Who can do what

• Members of a source's owner team can view and manage it. Access can also be granted to other teams.
• Replay window is for admins only.
• On a table's page, the per-source Detach button is shown to admins only. Anyone who can manage the source can detach it from the source's own page.