Docs > Platform Observability > Getting Started with Log Pipelines

Getting Started with Log Pipelines

Overview

This guide introduces AppStatus Log Pipelines and covers:

Process log lines as they arrive, before they are stored
Drop rules for health checks and other high-volume noise
Redaction rules so sensitive values are never retained
Normalisation for services with inconsistent formats
Preview against real recent lines before enabling

What are Log Pipelines in AppStatus?

A pipeline is a short, ordered list of rules applied to log lines at ingestion. Rules run top to bottom: drop first, then redact, then reshape whatever remains.

Because this happens before storage, a dropped line is never stored and a redacted value never lands anywhere. That makes pipelines the right place for retention hygiene — and a place to be careful, since dropped data cannot be recovered.

Key capabilities:

Scope a pipeline to specific services rather than everything at once
Drop, redact, and reshape rules with explicit ordering
Preview against recent real lines before the pipeline goes live
Route surviving lines to the retention tier you choose
Volume comparison before and after a change

What you will see

What a pipeline looks like once configured

app.appstatus.ioSample data

#	Rule	Match	Effect
1	Drop health checks	path = /healthz or /readyz	Discarded — not stored
2	Redact auth header	field: request.headers.authorization	Replaced before storage
3	Normalise level	level in (warn, WARNING, W)	Rewritten to WARN
4	Route debug	level = DEBUG	Sent to archive tier

Scroll the table sideways to see every column.

Every drop rule is a decision to be blind to something. Name the rule so the reason survives you.

Troubleshooting

A pipeline dropped logs I needed during an incident

Dropped lines are not stored and cannot be recovered. Review pipelines after every incident and narrow any rule that removed something useful — that review is part of running them safely.

Redaction is not taking effect

Check the field path matches exactly, including nesting. Preview against recent lines to confirm the rule matches before enabling; a rule that matches nothing silently does nothing.

Volume did not fall after enabling a pipeline

Confirm the pipeline’s scope actually covers the noisy services, and check rule order — a reshape rule placed before a drop rule does work that is then thrown away.

Operational Guidance

Every drop rule is a decision to be blind to something — write down why in the rule name.
Review pipelines after an incident; the line you needed may have been dropped.
Redaction here is about retention hygiene; it is not a substitute for not logging secrets.

Step-by-Step Setup

A pipeline is short and ordered, and it runs before storage. That makes it powerful and worth building carefully — preview every rule against real lines before you enable it, because a dropped line is gone.

Before you start

Logs already arriving in Log Management
A known noisy source worth trimming

1
Find the noise
Open Log Management and sort sources by volume. Health checks, readiness probes and per-request debug lines are the usual top offenders, and none of them are lines you would ever read.
WhereLog Management → Sources
2
Create a scoped pipeline
Create a pipeline and scope it to the specific services you identified. A pipeline scoped to everything is difficult to reason about and easy to regret.
WhereLog Pipelines → Create pipeline
3
Add rules in order
Drop rules first, then redaction, then reshaping. Order matters: a reshape rule placed before a drop rule does work that is immediately thrown away.
WherePipeline editor → Add rule
Tip
Name each rule with the reason for it. Six months on, the reason is the part nobody remembers.
4
Preview against real lines
Run the preview against recent traffic and read what each rule matched. A rule that matches nothing silently does nothing; a rule that matches too much silently costs you visibility.
WherePipeline editor → Preview
5
Enable, then re-check volume
Enable the pipeline and come back after an hour. Compare volume before and after to confirm the change did what you expected.
WhereLog Pipelines → pipeline detail

Configuration Options

Every option you can set, what each choice means, and what to pick. Use this as a reference while you fill in the form.

Rule types

Field	Options	What it does	Recommended
Drop	Match expression	Discards matching lines before storage.	Irreversible. Preview first, always.
Redact	Field path	Removes a sensitive value before storage.	Redact at ingestion so the value is never stored anywhere.
Normalise	Field + mapping	Rewrites inconsistent values to one convention.	Use for services that disagree about level names.
Route	Hot / archive	Sends surviving lines to a retention tier.	Route debug to archive rather than dropping it outright.

Feature Reference

Every feature, where to find it in the app, and what it does. Use this when you know what you want to do but not where it lives.

Feature	Where in app	Description
Scoped pipelines	Log Pipelines	Apply rules to named services rather than globally.
Ordered rules	Pipeline editor	Explicit top-to-bottom evaluation you can reason about.
Preview	Pipeline editor → Preview	See exactly what each rule matches before enabling.
Volume comparison	Pipeline detail	Confirm the change had the effect you intended.

Next Steps

Continue building your monitoring stack:

Log Management

Find the sources worth shaping first.

Ingest Keys

Control what each source may send.

Cloud Cost

See what ingestion volume is costing.