Release testing for pharma · Veeva Vault CRM and the Vault platform
Three Vault releases a year.
Every country you rolled out to.
One replayable proof that nothing moved.
Dedicated regression suites for Vault CRM, PromoMats, Clinical and RIM, parametrised per market and replayed against the prerelease Vault. Coverage is measured from what the runs actually touched — flow by flow, layer by layer, country by country — and every artifact stays inside your environment.
| Flow · once | DE | FR | IT | ES | UK | JP |
|---|---|---|---|---|---|---|
| Login & home | enforce | enforce | enforce | enforce | enforce | enforce |
| Account & territory | enforce | enforce | enforce | enforce | enforce | enforce |
| Call reporting | enforce | enforce | enforce | enforce | enforce | log |
| Samples + signature | enforce | enforce | log | enforce | n/a | gap |
| Consent capture | enforce | enforce | enforce | gap | enforce | gap |
| Disclosure (ToV) | enforce | enforce | enforce | enforce | enforce | n/a |
| Approved content | enforce | enforce | enforce | enforce | enforce | enforce |
Why multi-country rollouts fail at release time
One Vault, N configurations. The release lands on all of them at once.
A country rollout is never a copy. Germany records calls differently from France; Japan’s sample rules are not Italy’s; the disclosure fields, the consent channels, the language packs and the territory model all diverge. Each market was validated on its own timeline. Then a general release arrives on Veeva’s date and changes the platform under every one of them in the same weekend.
Re-execute the scripts by hand, market by market
Someone maps the RIA onto each country’s configuration, books local key users who have a day job, and walks the scripts in the prerelease Vault. Evidence is pasted into documents. The markets that finish in time are “covered”; the ones that don’t are covered on paper.
- Effort scales with markets — the sixth country costs what the first did, every release
- Coverage is asserted in a tracker, not measured from anything that ran
- Evidence quality tracks fatigue — the last market tested is the worst documented
- Auto-on features arrive regardless, on a date you did not choose
Record the flow once. Replay it per country. Measure what ran.
Start from the dedicated Vault CRM suite, point it at your configuration, and add one spreadsheet column per market. When the prerelease Vault opens, every flow replays for every column and each screen is compared to the baseline approved for that market. The coverage matrix tells you — from the run data — what has and has not been exercised.
- Markets are data — a new country is a column, not a new suite
- Coverage is a fact — flow × layer × country, computed from artifacts
- Evidence is generated — timestamped, per step, per market
- Deterministic — same build, same script, same evidence
“Although Veeva Vault provides a general RIA, it is not tailored to your organization’s policies, Vault configuration, or system usage.”
ProPharma Group, on Vault upgrade and release management
How we validate · what evidence you get
One call report, recorded once on the iPad. Every step captured, compared, kept.
A rep’s call-recording flow is replayed against the prerelease Vault. At each step the screen is captured, compared to the baseline your validation lead approved, and written down as a timestamped artifact with a verdict per check layer. The screens below are generic call-report wireframes, not Veeva’s interface.
1 · Record once
A key user walks the call report on a tablet against your Vault. What is saved is a Playwright script plus an approved capture of every screen — the ideal flow.
2 · Replay per market
The same script runs against the prerelease Vault for each country column of the scenario spreadsheet. No model in the replay path; the same build gives the same evidence.
3 · Compare each step
Every captured screen is checked against its baseline on nine layers — visual, text, DOM, network, console, performance, URL, API, storage — each enforced or logged as you configured.
4 · Keep the evidence
Per step: the capture, the baseline, the diff, the timestamp and the layer verdicts, in a folder on your disk. Differences are held for review and filed as issues with everything attached.
Dedicated suites, per Vault application
Built for the Vault you run. Vault CRM first.
Each suite already knows its application: the auth bridge, the object pages, the flows that carry GxP or commercial-compliance risk, and the one gotcha per area that would otherwise cost a week. You start from a working, recorded suite and point it at your own configuration and scenario data. The output is plain Playwright in your repository.
Vault CRM
Accounts, territories, call reporting, samples and consent - per country, with local rules for what a rep may record and to whom. The largest surface and the one that changes most between markets.
How we test Vault CRM →Vault PromoMats / MedComms
The rendition is the product. A status of Approved says nothing about what the HCP actually sees, so these suites compare the rendered document, per market and per language.
How we test Vault PromoMats / MedComms →Vault Clinical (CTMS, eTMF, EDC)
One platform, N study configurations. The suite tests the invariants every study shares and then the per-study configuration the release could have moved.
How we test Vault Clinical (CTMS, eTMF, EDC) →Vault RIM
The health authority owns the oracle. Suites check submission structure and metadata against the validation criteria you do not control, before a dossier goes out.
How we test Vault RIM →Data-based coverage measurement
Coverage you can point at. Not coverage you assert.
A traceability matrix records intent: someone ticked a box saying a script covers a requirement. Lastest records execution. When run #412 replays the German face-to-face call report, the artifacts say 14 screens were reached and 126 layer checks produced a verdict there. The ledger on the right is summed from those artifacts — nobody fills it in.
- Per market-specific flow. “DE · call report — face-to-face” and “FR · sample drop + hand-signed receipt” are separate rows with their own baselines and verdicts, because they are different flows — not one flow with a locale switch.
- Counted, not ticked. Screens reached, checks executed, and which of the nine layers — visual, text, DOM, network, console, performance, URL, API, storage — enforced or only logged, all read from the run’s artifact folder.
- Gaps are facts. “Consent capture reached 0 of 6 screens for Japan” is a query result, not an opinion, and it surfaces weeks before the general release.
- Trend per build. Coverage and a weighted health score (pass rate, non-flaky rate, route coverage) are kept per run, so an auditor can see the history rather than the last screenshot.
- Readable by people and agents. The same ledger drives what gets authored next and is exposed over the MCP server.
Read the Vault CRM recipe for how the scenario spreadsheet and area tree feed the ledger.
How a release cycle runs
From the RIA landing to the general release, without re-clicking a script.
Veeva publishes the validation package and opens prerelease Vaults weeks before each general release. The suites are built to use that window: baseline on the current release, replay on the prerelease, review the differences, and keep the evidence.
Adopt the suite, add your markets
Point the Vault CRM suite at your configuration, fill the scenario spreadsheet with one column per country, record any flow that is unique to you. Baselines are approved per market.
Scope from the RIA
Read the feature list and GxP ratings, mark which flows and markets are in scope. The matrix shows you where the current suite is thin before anyone runs anything.
Replay per country
Every flow runs for every market against the prerelease Vault. Each screen is compared to its approved baseline with three engines; the text and DOM layers check Part 11 manifestations literally.
Review differences, not stacks
A short list of changed screens per market, with the evidence attached. Approve intended changes, file the rest as issues, and export the artifacts into your own records.
Run again
The same suite is a scheduled check after the general release, after each limited release and after your own configuration changes. The third run costs a scheduled job, not a week.
What a validation and security reviewer will ask
Four properties, each checkable
before you buy anything.
01 · Locality
It runs inside your environment
Self-hosted with Docker on infrastructure you control. Runner, captured screens, baselines and reports stay inside your network boundary, and the install can run air-gapped. Scenario data is synthetic, so no real HCP or patient record is anywhere near the suite.
02 · Evidence
Generated, not assembled
Every step of every flow, for every market, produces a timestamped, machine-generated artifact of the screen as it rendered, tied to the run that produced it. Retention, review, approval and signature stay in the systems and SOPs you already use for them.
03 · Ownership
Playwright, in your repository
The dedicated suites, your recorded additions and the scenario spreadsheet are files under your version control. Reviewable line by line, diffable in a change record, runnable by an auditor, a CRO or whoever replaces us. Leaving does not mean starting over.
04 · Determinism
AI at authoring time. None at replay.
A model helps record or repair a flow, against an endpoint you choose. What it emits is a script. In the regulated profile nothing probabilistic produces a verdict: replay executes the script with no inference in the path, so the same build and the same script produce the same evidence.
Questions from validation, QA and IT security
- What does "dedicated tests for Veeva" actually mean?
- Suites that already know the shape of Vault CRM, PromoMats, Clinical and RIM: the login bridge, the object pages, the call-reporting and sample flows, the rendition viewer, the study and submission structures. You start from a recorded, working suite for your application instead of a blank recorder, then point it at your own configuration and your own scenario data. What comes out is plain Playwright in your repository - the starting suite removes weeks of authoring, it does not replace your PQ scripts or decide your regression scope.
- How does one suite cover several countries?
- Each country is a column in the scenario spreadsheet that feeds the suite: the local call-reporting rules, sample and consent regulations, disclosure fields, language and territory model. The same recorded flow replays once per column against the prerelease Vault, and every replay is compared to the baseline that was approved for that market. Adding a market is adding a column, not recording a new suite.
- What is "data-based coverage measurement"?
- Coverage computed from what the runs actually did, not from a test plan. Every run records which screens it reached, which of the nine check layers produced a verdict there, and for which market. The result is a matrix of flow x layer x country where a gap is a fact - "consent capture was never executed for Japan" or "the text layer is only logging for France" - visible before the general release date rather than discovered in an audit.
- Does Lastest replace our validation process or our QMS?
- No. Lastest is a test runner that produces objective, timestamped evidence inside your existing process. Your URS defines intended use, your PQ and UAT scripts define scope, your change control and approvals stay where they are. Lastest changes how execution is performed and how evidence is captured - it is not an electronic-signature system, not a Part 11 records system, and not a quality management system, and we will not claim otherwise.
- Where does our Vault data live when we run this?
- Inside your network. Lastest self-hosts via Docker on infrastructure you control; the runner, screenshots, baselines and reports never leave it, and the install can run air-gapped. Scenario data is synthetic and generated from the spreadsheet, so real HCP or patient records have no reason to be anywhere near the suite. Prerelease Vault screens carry regulated content, which is why this matters more here than anywhere else.
- Does AI run while the tests are executing?
- No. A model is used when a flow is authored or a broken step is repaired, against an endpoint you choose - hosted or local. What it produces is a Playwright script. Replay executes that script with no inference in the path, so the same build and the same script produce the same evidence every time. In the regulated profile nothing probabilistic is allowed to produce a verdict.
- What happens to the tests if we stop using Lastest?
- You keep them. The suites, the scenario spreadsheet and the baselines are files in your own repository - readable TypeScript, not rows in a vendor database. An auditor, a CRO, an internal team or whichever vendor replaces us can run them without Lastest installed.
- How does this relate to the Veeva Release Impact Assessment?
- It sits downstream of it. The RIA tells you which features changed, their GxP risk rating and which are auto-on. It does not know your Vault configuration, your country-specific rules or how your users actually work. You read the RIA to decide scope; the suites are how that scope gets executed against every market you have rolled out to, with the differences captured screen by screen.
Book a release-readiness review
Bring the RIA for your next release. Thirty minutes.
Bring the Release Impact Assessment you are working from, the list of countries you have rolled out to, and the scripts you re-execute every time. We will walk through:
- Which of your flows the dedicated Vault suites already cover, and what is unique to you.
- What the coverage matrix would show for your markets on the first replay.
- What your IT security review needs to sign off a self-hosted install.
If it doesn’t fit, we will say so. Veeva is one of several vendors setting release dates for their customers — the shared pattern is on release testing for vendor-set dates.