Export your organization

An organization export copies everything out of Vigilfield into an S3 bucket you own:

• Every table's data, as the Parquet files it is stored in: your tables, query results that have not expired, and the audit trail.
• All metadata, as JSON: one file per kind of resource.

There is nothing to choose. An organization export always contains everything. To copy one table, in a format of your choice, use a table export instead.

Only one organization export can run at a time. Starting a second while one is queued, running or cancelling is refused with 409.

Start an export from the appCopy Anchor Link

1. Open Administration → Organization, tab Data export. Only the organization owner can open this page.
2. Check the estimate at the top: the data size, the number of tables and the estimated cost.
3. Destination is Amazon S3, the only kind today. Fill in the fields described below.
4. Click Review export. Every required field must be filled in.
5. The confirmation shows where the export will be written and its estimated cost. Click Start export.

Export history lists your exports, newest first, with their status, where each was written, when it was created and finished, how much was written and how many tables. It refreshes on its own while an export is in progress. A failed export shows the error.

Set up the S3 destinationCopy Anchor Link

Vigilfield reaches your bucket through AWS web-identity federation. It assumes a role you create, using a short-lived identity token. No long-lived Vigilfield credentials are stored there. This is the same setup as an S3 table export; if you already have that identity provider, reuse it.

1. Register Vigilfield as an identity providerCopy Anchor Link

In IAM in the AWS account that owns the bucket, add an OpenID Connect identity provider:

• Provider URL: https://<your-org>.vigilfield.com/api/v1/federation
• Audience: sts.amazonaws.com

<your-org> is your organization's slug, the first part of your Vigilfield address.

2. Create the export roleCopy Anchor Link

The trust policy admits only your organization's export identity. The token's subject is export:<org-id>, where <org-id> is your organization's id. The Data export tab shows it, and so does List organizations.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::<account-id>:oidc-provider/<your-org>.vigilfield.com/api/v1/federation"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "<your-org>.vigilfield.com/api/v1/federation:aud": "sts.amazonaws.com",
          "<your-org>.vigilfield.com/api/v1/federation:sub": "export:<org-id>"
        }
      }
    }
  ]
}

Large files are uploaded in parts, so the role needs the multipart actions as well as s3:PutObject:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["s3:PutObject", "s3:AbortMultipartUpload", "s3:ListMultipartUploadParts"],
      "Resource": "arn:aws:s3:::<bucket>/<prefix>/*"
    }
  ]
}

If you name a KMS key, the role also needs kms:GenerateDataKey and kms:Encrypt on that key:

{
  "Effect": "Allow",
  "Action": ["kms:GenerateDataKey", "kms:Encrypt"],
  "Resource": "arn:aws:kms:<region>:<account-id>:key/<key-id>"
}

The role needs no read or delete permission. Vigilfield only writes.

3. Fill in the destinationCopy Anchor Link

Vigilfield checks only that the fields are filled in when you start. It does not try the role until the export runs, so a wrong trust policy or a missing permission shows up as a failed export with an error message.

What is writtenCopy Anchor Link

Each export is written under its own folder, <prefix>/<export-id>/:

<prefix>/<export-id>/
  tables/<table-name>/schema.json
  tables/<table-name>/data/*.parquet
  metadata/<kind>.json
  manifest.json

• tables/<table-name>/data/ holds the table's data files, as stored.
• tables/<table-name>/schema.json describes the table's columns.
• metadata/<kind>.json holds every resource of one kind.
• manifest.json is written last. An export folder without it is incomplete: the export failed, was cancelled, or is still running.

CostCopy Anchor Link

An organization export is charged per GiB written. The Data export tab shows the estimated size and cost before you start, and asks you to confirm. The estimate uses the tables' current sizes, so the final amount follows what is actually written.

It is not limited by your plan's monthly exported volume, and it runs even when that limit has been reached. See Plans and billing.

Cancel an exportCopy Anchor Link

A queued or running export has a Cancel button. It moves to Cancelling, and then to Cancelled once it has stopped. Files already written stay in your bucket, and there is no manifest.json. A cancelled export is billed for the data it had already written. Cancelling a finished export changes nothing.

When an export failsCopy Anchor Link

The history shows the error from the call that failed. The usual causes:

• The role cannot be assumed. The identity provider is missing, or the trust policy's aud or sub does not match. The subject must be exactly export:<org-id>.
• Access denied writing to the bucket. The role lacks s3:PutObject or a multipart action, or its resource does not cover <bucket>/<prefix>/*. A bucket policy can also deny the write.
• Access denied on the KMS key. The role lacks kms:GenerateDataKey or kms:Encrypt, or the key policy does not allow the role.
• The bucket does not exist, or is in another region than the one you entered.

Fix the cause and start a new export. Files from the failed one are not removed; you can delete them yourself.

Through the APICopy Anchor Link

Estimating, starting, getting and cancelling an export need an organization admin.

• Estimate an export returns total_bytes, table_count, a tables list with each table's bytes, and cost_usd.
• Start an export takes one field, destination, with the same S3 fields as above. A missing field is refused with 400 naming it. A second export while one is active is refused with 409.
• List exports and Get an export return its status (queued, running, cancelling, succeeded, failed or cancelled), its location (s3://<bucket>/<prefix>/<export-id>/), bytes_written, table_count and, for a failed export, error_message.
• Cancel an export stops a queued or running export.