Test every competitive intelligence vendor on your shortlist with the same brief

The checklists we read in September 2026 either list what a CI program should track (1up's competitive intelligence checklist, RivalBeam's PMM checklist) or describe proof-of-concept discipline for any software (Worthwhile's nine keys to a POC). Worthwhile's discipline transfers: a defined problem, measurable outcomes, verified data, criteria set in advance. None of the three applies it to the sources and jobs a CI tool is bought for.

Fix the competitors, the window and the sources first

Use the smallest set of competitors you already track that covers contrasting change patterns. Give every vendor the same start and end date, agreed in advance with each vendor. A competitor change cannot be scheduled; if no material change occurs during the window, record the monitoring result as not observed.

List the sources you may point a tool at: specific competitor pages, a review site, a pricing page, a news source, and any internal material such as call recordings or CRM notes. For every nonpublic source, record the role or process authorized under company policy and applicable contract or law to approve this vendor use, plus any customer, data-subject or recording restriction. Separately record whether the vendor contract permits storage or other use, for how long, and by which subprocessors. Exclude the source if you cannot confirm those conditions. Vendors differ in what they can read and what permission they require, so the source list is part of the test; a source a vendor cannot reach is a finding.

Keep the brief, the source list and the scorecard in a Claude Project set up for product marketing, so every vendor's output lands next to the same rubric. Anthropic's Projects article confirms that a Project holds uploaded files and its own instructions. A shared document works too. The brief has to exist in one place before the first test starts.

Give every vendor the same three jobs

One monitoring alert. Name the competitor and the kind of change that matters to you, such as a pricing change or a new integration page. You score the alert itself: when it arrived, what it said, and whether it linked the changed page.

One battlecard section. Choose one objection sellers hear now, about the same competitor in every test. You score the section the tool produces and the citations it attaches. The battlecards lesson describes the shape sales reads, if you need a standard.

One digest. Ask each tool for its summary of that competitor over the window, delivered as a seller or executive would receive it, in the channel the vendor proposes.

Write the three requests once and paste them into each vendor's test unchanged. If a vendor proposes a fourth job that shows its strength, add it for every vendor or leave it out.

Decide what proof each output needs, before the outputs exist

For each job, write down what the output must contain to count. Our recommendation for all three: a link to the source page, the date the source was read, and wording you can check against that page. A battlecard claim with no source fails however well it reads. A digest that does not name its pages cannot be tested.

Then list the ways a CI tool fails quietly, because a demo never shows them:

• A stale claim carried forward. The competitor removed a feature or changed a price, and the card still states the old fact.
• A real change missed. A monitored page changed inside the window and no alert arrived. Only someone who also checks the pages notices, which is the control run below.
• A citation that does not support the claim. The link exists, the page loads, and the sentence beside it says something the page does not.

Name who checks each failure case for each vendor, and write into the brief how often they open the monitored pages, since missed changes take the most time to catch.

Agree the pass and fail criteria in writing

Write the acceptance criteria before the first test and get the purchase approvers to agree. Keep them fixed so every vendor is judged against the same test. If you use weights, set them before results are visible.

• The monitoring alert reached the agreed channel within the agreed time of a real change, and linked the changed page.
• The battlecard section cites a source for every factual claim, and each cited page supports the sentence beside it.
• The digest names every source it summarised, and a reader can open each one.
• Every source on the permission list was reachable by the tool, or the vendor stated in writing which were not and why.
• None of the three quiet failures occurred, or the tool disclosed each one before the checker found it.
• A seller who did not take part in the test could use the battlecard section without asking a question.
• The setup a person had to do, in hours, is recorded for each vendor.

Score each item pass or fail, with a sentence of evidence. You can still buy a vendor that fails an item; the decision then records which item and why that was acceptable.

Run a DIY control with the same brief

Before or alongside the vendor tests, run the same three jobs yourself: check the monitored pages on a schedule, write the battlecard section from what you find, and produce the digest by hand or with Claude. The competitive research plan, competitor alerts and competitor digest lessons cover each job without a vendor.

The control run tells you what changed during the window, which is a direct way to know whether a tool missed something. It also puts hours on the manual version, so the decision compares each vendor with the work you would otherwise do yourself.

Score every output against the identical rubric

1. Paste each vendor's output into the Project with its date and job label

   Include the alert text, the battlecard section and the digest exactly as delivered, with citations. A citation lost in copying becomes a failure the vendor did not cause.

2. Ask Claude to grade the output against the acceptance list

   Have it return, per item, pass or fail and the sentence in the output that decides it. Read every fail and open the cited page yourself. In this paste-back method, Claude can compare a claim with a linked passage only when you provide both.

3. Check the failure cases against your control run

   Compare each vendor's alerts with the changes your control run recorded. A change the control saw and the vendor did not is a missed change. Record it with the page and the date.

4. Write one line per vendor per job, then decide

   Write a compact scorecard for each vendor that summarizes the acceptance results, output evidence and setup hours. Use more space when the evidence requires it, and keep the scorecard as the baseline for the renewal conversation.

Claims that survive the test can move into a competitive claim ledger with their source and date, whichever tool you choose. If two vendors tie, the build-versus-buy guide covers the ongoing workload the test did not measure.

If Steve is on your shortlist, ask the vendor for a demo and bring the same brief, rubric and control run. Steve's competitive intelligence product page describes the product area.