Pipelines are recurring workflows that run automatically. Each pipeline pairs a trigger, which decides what makes it run, with a workflow, which defines what runs.
What Pipelines Offer
Orchestrate data tables
Combine data from sources like SEC filings, news, prices, and company metadata. Filter, join, and add LLM-generated classification columns, then output rows into persistent tables called LiveArtifacts that accumulate over time.
Build live dashboards
Visualize a pipeline's accumulated output as charts/tables on a dashboard so trends are visible at a glance.
Monitor websites
Describe in plain English what to watch, such as FDA press releases or a competitor's pricing page, and have a workflow run when new matching content appears.
Triggers
- Data - fires when new rows match your filters in a source table
- Time - runs on a schedule, such as every weekday at 9am
- Web - fires when monitored sites publish new matching content
- Manual - only runs when you start it
Example Pipelines
- "Daily 9am summary of overnight earnings reports for my watchlist" (Time trigger)
- "Score every new 8-K filing from S&P 500 companies for materiality" (Data trigger -> LiveArtifact -> dashboard)
- "Notify me when the FDA announces a new drug approval" (Web trigger)
Use via the API
Pipelines are created and edited through the Pipeline Builder agent. Create a conversation with POST /api/v1/conversations and set agentType to pipeline-builder:
{
"prompt": "Build a pipeline that runs every weekday at 9am and summarizes overnight earnings reports for AAPL, MSFT, and NVDA.",
"agentType": "pipeline-builder"
}
To modify an existing pipeline, including its dashboards, workflows, or LiveArtifacts, start a new pipeline-builder conversation with pipelineId set to that pipeline and describe the change you want. The agent has the tools to edit the pipeline's components in place.
You can also read and run pipelines directly:
GET /api/v1/pipelineslists the pipelines you created. Read one withGET /api/v1/pipelines/{pipeline_id}. Rename it withPATCH, and archive or restore it withPOST .../archiveandPOST .../unarchive.POST /api/v1/pipelines/{pipeline_id}/runsruns the whole pipeline once, from the entry points of its active version that have work, and returns the startedworkflowRunIds(none when its sources have no new event).POST /api/v1/pipelines/{pipeline_id}/triggers/pauseand.../triggers/resumepause and resume its automatic triggers, as Pause and Resume do in the app.GET /api/v1/pipelines/{pipeline_id}/workflowslists the workflows in the pipeline's active version.POST /api/v1/workflows/{workflow_id}/runsstarts a run; passentryPointto start from a specific trigger. Follow the run withGET /api/v1/workflow-runs/{workflow_run_id}.GET /api/v1/pipelines/{pipeline_id}/live-artifactslists the pipeline's tables. Read rows withGET /api/v1/live-artifacts/{live_artifact_id}/data(limitup to 10,000, paged withoffset).GET /api/v1/live-artifacts/{live_artifact_id}/streamsends Server-Sent Events when the table changes: eachlive_artifactevent carries the table's metadata, such asupdated_atandrow_count, not its rows, so read new rows from/datawhen it arrives.GET /api/v1/pipelines/{pipeline_id}/artifactsandGET /api/v1/pipelines/{pipeline_id}/dashboardslist its reports, files, and dashboards.