Creative Pulse — methodology

29 April 20268 min

The Creative Pulse is a regular set of posts about what is shifting in performance creative across Meta, Google, and TikTok — anchored in our own analysis data and corroborated against trusted external sources.

This page describes how each post is built. It exists for two reasons. First, every claim in a Pulse post should be traceable back to either an internal database query or a verbatim quote from a published source page; that traceability needs a public reference. Second, the editorial discipline that keeps the posts honest also benefits from being written down and signed.

Linked from the footer of every Pulse post. Updated whenever the pipeline changes.

Three commitments

These are load-bearing. If any of them turns out to be untrue for a published post, that post is retracted.

Every number traces to a query. Every numeric claim in a Pulse post — every percentage, every count, every dollar amount, every multiplier — comes from a specific SQL query against the platform database. The query identifier is recorded against the post in the audit trail. We do not estimate, round dramatically, or insert numbers from elsewhere.

Every external claim is verified by a separate fetch. When a post cites an external development — a Meta announcement, a Google policy change, a TikTok-for-Business note, a piece in a trade publication — the citation includes a verbatim quote from the source page. A separate verification stage, independent from the model that proposed the citation, re-fetches the URL and confirms the quote appears word-for-word. Citations that fail re-verification are dropped, silently, from the post.

No per-account data, ever. Every statistic in a Pulse post applies to a minimum of twenty distinct customer accounts. Smaller cohorts are discarded at the data-extraction layer, before any prose is written. We do not name customers. We do not publish single-account performance figures. We do not publish vertical or segment cuts narrow enough to identify a specific account by elimination.

How a post is built

Six gates, run in order. A post that fails any one of them does not publish.

Gate 1 — Privacy-gated extraction. A weekly query rolls up creative-intelligence data into candidate facts. Each fact carries a sample size, a time window, and a p-value where applicable. Facts that apply to fewer than twenty distinct customers are dropped. Facts where a single customer accounts for more than 30% of the underlying activity are dropped. Facts that don't reach a 10-percentage-point effect or a 0.05 p-value are dropped. What survives is a small set of statistically meaningful, k-anonymous patterns.

Gate 2 — External research. For each surviving pattern, an Anthropic-hosted web search runs against a fixed allowlist of trusted domains (platform first-party announcements, established trade press) looking for one external development from the past fourteen days that explains or contradicts the pattern. The model returns proposed citations with verbatim quotes from search results. If no credible external link exists for a pattern, the citation is null — the model is instructed to never invent a connection.

Gate 3 — External verification. Every proposed citation is independently re-fetched. The page text is normalised (HTML stripped, whitespace collapsed, smart quotes converted to straight) and the verbatim quote is searched for. If the quote isn't found, the citation is dropped. If the URL redirects to a domain outside the allowlist, the citation is dropped. If the page returns an HTTP error, the citation is dropped.

Gate 5 — Composition. A separate model formats the verified bundle into a markdown post. The system prompt is explicit and narrow: only use claims that appear in the bundle, omit any section requiring a fact not in the bundle, return an "insufficient" response if the bundle cannot support a coherent post. Banned phrases are listed inline in the prompt. The model's output includes a list of every fact identifier it referenced.

Gate 6 — Numerical verification. Every percentage, currency value, multiplier, p-value, and sample size in the draft is independently extracted by regex and matched against the bundle. A typed number that doesn't have an exact match in the bundle blocks the post.

Gate 7 — Pattern lint. A deterministic check passes the draft through banned-phrase detection (cliché copy, predictive language about reader outcomes, methodology leaks, per-account references, defamatory comparative claims about named competitors with statistics).

Gate 8 — Cross-check. A separate model, with no awareness of how the composer worked, reads the draft alongside the source bundle and classifies every factual claim as supported, unsupported, weakened, or extrapolated. Anything other than "supported" blocks the post.

Gate 9 — Human review. The draft, with its full provenance trace, is reviewed by Oliver. Every numerical claim is hyperlinked to the source query. Every external citation is hyperlinked to the verified URL and the matched quote. Sign-off is the final action before publish.

Six gates against fabrication. Four are deterministic (no language model involved); two are language models running against the bundle; one is a human. Sign-off requires all of them to pass.

How we talk about the data

Language is calibrated to effect size. A delta of fewer than 3 percentage points is noise; we don't write about it. A delta of 3 to 10 points is "a modest change." A delta of 10 points or more is "a shift." We describe correlations as correlations and explicitly hedge causal language ("may be related to," "appears to coincide with") because most of what we report is observational, not experimental.

We describe sample sizes inline. A claim about "47 DTC accounts" is exactly 47, not approximately, and the post will say so. A reader who needs to evaluate confidence should be able to do that without leaving the page.

We do not predict your account's outcomes. Performance numbers that appear in a Pulse post describe what was observed in the cohort being studied. We do not write that adopting a pattern will boost ROAS, lift CTR, or improve conversion for the reader. Those forward-looking claims are not what the data supports, and we are not in the business of making them.

We name all three platforms when discussing cross-platform observations. Meta, Google, and TikTok are mentioned together where relevant — the data is unified across them and the posts treat them as one canvas.

What we don't publish

By design, a Pulse post will never:

Quote a specific customer.
Show a per-vertical statistic where the vertical contains fewer than twenty active customers.
Publish methodology details that would let a competitor reproduce the scoring model — feature weights, training-corpus composition, internal coefficients.
Make a quantitative comparative claim against a named competitor based on our own data ("Hawky's grading is X% less accurate"). Independent comparison work belongs on the comparison pages, where the framing is editorial, not data-journalism.
Claim statistical significance without disclosing the underlying methodology in the post.

When we get something wrong

Errors are inevitable. The retraction process exists so that getting one wrong does not compound.

If a published Pulse post is identified as containing an error — internally, or via a reader who emails the address at the end of this page — the pipeline pauses for new runs while the issue is investigated. The pause is automatic and operationally enforced: the weekly autonomous run checks for any retraction in the past 30 days and skips its run if it finds one. The pause auto-resumes 30 days after the most recent retraction, by which time the post-mortem and any added safety check should be in place. (An admin can still manually trigger a run during the pause if the investigation has concluded — the autonomous half stays asleep.)

The post is updated with a correction footer that names the affected claim, the corrected text, and the date. If the underlying error is material, a separate retraction post is published and linked from the original. The investigation includes a post-mortem: which gate should have caught this, why didn't it, what additional check should be added.

The retraction itself is recorded against the post's audit row, alongside the reason and the correction URL. Retractions are visible in the public archive, not quietly removed — they appear with a strike-through marker and a link to the corrected version.

Audit trail

Every published Pulse post has a permanent record. The record contains:

The candidate fact set produced at Gate 1, including every query identifier and result row.
The verified external citations, including each fetch timestamp and the matched quote position on the source page.
The composer model output, with the list of fact identifiers the model self-reports as referenced.
Every verifier result (numerical, pattern, cross-check), with full breakdowns of any near-misses.
The reviewer's sign-off, including the reviewer's identifier and signoff timestamp.
Total cost (in USD) of the LLM calls that produced the post.

The audit row is append-only. Updates create new rows linked to the original. Retractions add a row but never delete the original.

Found an error?

Email relations@omniscia.app with the post URL and the claim you believe is wrong. Include a source if you have one. The pipeline will pause for new runs while we investigate.

We don't promise to agree with every report. We do promise to investigate every report that names a specific claim, log the outcome, and update the post if the claim was in fact wrong.

This page describes the methodology as of {{publishedAt}}. Material changes to the pipeline are documented here and dated.