Output comparison belongs in every serious Exstream upgrade. Run the old and new package-and-engine combinations against controlled inputs, then compare the PDF, AFP, PostScript, HTML, reports, and operational side effects that matter.
Output comparison proves what the selected inputs rendered. It cannot cover design paths those inputs never selected or identify whether the expected package was tested.
Samples leave design paths untouched
An Exstream application can contain wording and logic reached only for a particular product, jurisdiction, language, channel, campaign, effective date, or error condition.
If the regression data never selects that path, a changed rule or missing paragraph produces no output difference. The output comparison is correct. The test coverage is incomplete.
Add a design-state comparison beside the output test.
Package rebuilds change more than visible pages
An upgrade can rebuild a package with different variable settings, formula source, data-file mappings, fonts, object versions, or packaged resources. Some changes affect output only under a condition absent from the regression population. Others affect operations rather than page appearance.
A visual PDF comparison will not tell you that:
- A data field now populates another variable.
- A formula changed compute time.
- A rule body changed in an untested branch.
- A customer-facing sentence disappeared from a language variant that was not generated.
- The package was built from latest work instead of approved objects.
- The new package contains a different application revision despite producing the same small sample.
These are design and artifact questions.
Add a design-state comparison
Read both packages into the same structured design state and compare them at detailed depth:
exstream-package-cli diff before-upgrade.pub after-upgrade.pub \
--json-mode detailed --format html --out package-review.html
If the upgrade also changes the Design repository, compare the new package with the intended approved repository state:
exstream-package-cli diff approved-state.json after-upgrade.pub \
--lineage same-lineage \
--population intersecting-populations \
--json-mode detailed \
--format html --out approved-package-review.html
The result identifies added, removed, moved, property-changed, and content-changed objects. It also states which fields could not be compared, which matters when package generations expose different detail.
Use four layers of upgrade evidence
For most classic upgrades, I would use four distinct layers.
Artifact identity
Record the package digests, application versions, Design Manager versions, approval selections, engine builds, control files, and output driver settings. Without this, a clean regression run may be attached to the wrong artifacts.
Design and package comparison
Compare old package with new package and approved repository with new package. Review wording, code, mappings, settings, object versions, and comparison coverage.
Output regression
Run representative production conditions, not one preview document. Include every output type, important data branch, language, sorting and bundling path, accessibility mode, and downstream handoff the upgrade can affect.
Operational verification
Check reports, queue behavior, resource resolution, performance, delivery handoff, restart behavior, and failure handling. A page can look identical while the production job around it is broken.
Disagreement between layers is useful
Suppose the design-state diff reports a formula change, but output stays identical. That is not noise. It tells you the current regression data did not distinguish the formulas, or the change is behaviorally neutral for the tested population.
Suppose output changes but the two packages have no comparable design difference. Look at the engine, compatibility switches, output driver, fonts, resources, and environment.
Suppose the approved repository differs from the new package while old and new output match. Stop and establish which state the release intended to deploy.
The layers narrow the investigation because they fail for different reasons.
Keep the claim as narrow as the evidence
A clean output comparison supports this statement: no difference was observed for these inputs, artifacts, engines, configurations, and output checks.
Do not broaden that result into "the upgrade changed nothing."
A clean package diff has its own boundary. It says no difference was found in the fields that both package readers could compare. Read the coverage before making the claim broader.
Upgrade evidence becomes credible when each result states exactly what it tested and what it could not test.
The existing Exstream upgrade strategy and regression automation guide covers the package-and-engine test unit. CCMForge Inspect adds the repository and package evidence around that run.