Supported Data Types

A project's data folder is uploaded and processed according to its specification (poq.toml). This page describes the file types the ingestion process understands.

Supported file types

Ingestion recognizes several source types, each covering one or more underlying file types, with examples of supported and unsupported cases below.

CSV (.csv)

One row per item with optional header row.

Supported:

• Single CSV file, like labels.csv, with a header row and a row for each item.
• Sapien PoQ automatically determines columns and formatting.

Unsupported:

• Combining multiple CSV files into a single ingestion source. Each CSV source reads exactly one file.
• Files with unusual or inconsistent formatting (odd quoting, mixed encodings) that cannot be deterministically parsed.

JSON (.json)

Each JSON file, or each element in a JSON array, becomes one datapoint, with fields read from its object keys.

Supported:

• One datapoint per file, for example finding-001.json, finding-002.json, etc.
• One file that contains a list of datapoints (a JSON array). Ingestion can split it into one datapoint per array element.
• One level of nested object fields, for example a severity value nested inside a finding object (finding.severity).

Unsupported:

• Object fields nested more than one level deep, or arrays nested inside arrays.

Binary or opaque files (.jpg, .pdf, ...)

The file_collection upload type can be used for various binary file formats for ingesting objects like images and PDFs contained within a folder. The contents of these files are not parsed, meaning multiple datapoints cannot be created from one file's contents.

Supported:

• A folder of images, PDFs, videos, or many other binary file types, matched by a pattern such as images/*.jpg.
• Mixing file types in the same pattern, such as images/*.{jpg,png}.

Unsupported:

• Pointing at a single file. This ingestion specifier is for folders containing many files.

Markdown files (.md)

A long document that gets divided into separate sections.

Supported:

• One or more text or report files, divided wherever a heading such as "## Finding 1" appears.
• Pulling a few extra labeled facts out of each section.

Unsupported:

• Dividing a report into sections, then dividing those sections into smaller pieces again. Only one level of division is supported.

If the dividing rule does not match anything, this is not reported right away — it surfaces later, once ingestion produces no datapoints.

Limits

• The largest supported file size is 5 GB.
• A single project's data produces at most 1,000,000 datapoints.
• File paths must stay inside the project's own folder; paths cannot reach outside of it.

If more data formats need to be supported, contact the Sapien team.

Continue