Reproducing a quantum experiment is more than rerunning the same circuit. You also need to preserve the software environment, circuit definition, compilation settings, randomness, shot count, noise assumptions, backend details, raw measurement data, and evaluation method. This checklist gives you a practical workflow for making quantum experiments repeatable and comparisons fair across simulators, SDKs, and quantum hardware.
Overview
Quantum experiments can produce different results even when the source code appears unchanged. Some differences are expected: measurement is probabilistic, simulators may sample from a distribution, and hardware introduces calibration and noise variation. Other differences come from the development workflow, such as an SDK upgrade, a changed transpiler pass, a different qubit mapping, or an omitted random seed.
Reproducibility does not mean that every run must produce identical counts. It means that another developer—or your future self—can identify the inputs, rerun the experiment under the same conditions, and understand why any remaining variation occurred.
A useful reproducibility record has four layers:
- Program: the circuit, parameters, measurements, and classical post-processing.
- Environment: SDK, language, dependency, operating system, and runtime versions.
- Execution: backend, device configuration, transpilation, shots, seed, and noise model.
- Evaluation: raw results, derived metrics, tolerances, and comparison methodology.
Keep the circuit definition and metadata together where possible. A repository can contain source code, locked dependencies, serialized circuit representations, configuration files, raw result files, and a short run manifest. For guidance on moving circuits between frameworks, see the Quantum Circuit Portability Guide.
Checklist by scenario
1. Deterministic statevector or exact simulation
Exact simulation can be highly repeatable because it calculates amplitudes or probabilities without sampling. That does not remove the need for metadata. Record:
- SDK, simulator, programming language, and dependency versions.
- Number and order of qubits, classical bits, gates, and measurements.
- Initial state and any custom state preparation.
- Parameter values, parameter ordering, and precision settings.
- Whether the output is a statevector, probability distribution, expectation value, or another representation.
- Numerical tolerances and the method used to compare floating-point values.
For deterministic outputs, compare values with an explicit tolerance rather than requiring byte-for-byte equality. Also preserve the circuit diagram or an unambiguous serialized form. Two circuits that look similar in a notebook may differ in gate order, qubit indexing, or implicit measurement behavior.
2. Stochastic simulation
Shot-based simulation samples measurement outcomes. Reproduction therefore depends on both the probability model and the sampling configuration. Record:
- Total shots and how shots are divided across batches or parameter values.
- All random seeds, including separate seeds used by the SDK, simulator, noise model, or optimization loop.
- Simulator type and noise model, including channel parameters and where noise is applied.
- Measurement basis, readout processing, and whether results are mitigated or post-processed.
- Seed behavior: whether a seed is accepted, ignored, generated automatically, or applied only to part of the workflow.
A seed can make a particular simulator run repeatable, but it is not a substitute for recording the probability model. If a framework changes its random-number implementation, the same seed may not reproduce the same sequence. Treat the seed as one input among many, not as a universal guarantee.
3. Real quantum hardware
Hardware runs require a broader record because the device is an active execution environment. Capture:
- Backend or device identifier and the date and time of execution.
- SDK, provider, runtime, and dependency versions.
- Requested circuit and the final transpiled or compiled circuit, if available.
- Qubit mapping, gate set, optimization level, routing choices, and scheduling settings.
- Shot count, queue or job identifier, and any execution options.
- Calibration, noise, error-mitigation, dynamical-decoupling, or readout-processing settings.
- Raw counts or samples, not only a plotted result or final score.
Hardware is not expected to return identical counts across runs. A fair report should distinguish repeatability of the workflow from stability of the physical result. When comparing devices, use the same logical circuit, shot policy, result-processing method, and clearly stated acceptance criteria. The guide to running a quantum circuit on real hardware provides useful context for separating local development from cloud execution.
4. Parameterized or hybrid experiments
Variational circuits and hybrid quantum-classical workflows add another source of variation: the classical optimizer. Record the initial parameter vector, parameter bounds, optimizer name and configuration, stopping conditions, number of iterations, batch structure, objective function, and any gradient-estimation settings. Save the parameter history and objective values, not just the final parameters.
If you are training a parameterized circuit, preserve the dataset split, preprocessing steps, initialization rule, and data order. A quantum machine learning result cannot be reproduced from the circuit alone. For implementation details, see Parameterized Quantum Circuits in Python.
5. Regression tests and CI
For automated tests, prefer small circuits and stable invariants. Test circuit structure, parameter binding, gate counts, measurement keys, probability relationships, or expectation values within a tolerance. Avoid asserting one exact sampled-count dictionary unless the test deliberately controls the simulator and seed.
Store a compact experiment manifest with each benchmark run. A practical template is:
experiment_id: descriptive-name
commit: repository commit or release
language: runtime and version
sdk: package names and versions
backend: simulator or device identifier
circuit: source path and serialized circuit hash
transpilation: mapping, optimization, routing, scheduling
parameters: values or parameter-file hash
shots: integer
seeds: all applicable random seeds
noise_model: definition or configuration-file hash
mitigation: method and settings
raw_results: file path or artifact identifier
metrics: formulas, tolerances, and comparison baseline
timestamp: execution time and timezoneWhat to double-check
Circuit identity: Confirm that the circuit being executed is the circuit you inspected. Transpilation can change gate order, introduce swaps, and alter depth while preserving intended behavior. Save both the logical circuit and the executable form when the toolchain exposes it.
Qubit and bit ordering: Different frameworks and result formats may display bitstrings in different orders. Document the convention and test it with a simple basis-state circuit before comparing results.
Shots and aggregation: Check whether shots are total, per circuit, per parameter setting, or per repetition. Do not compare a high-shot run with a low-shot run without acknowledging the difference in sampling uncertainty.
Seeds: Verify which random processes are seeded. An optimizer, parameter initializer, noise sampler, and circuit simulator may each have separate randomness.
Noise and mitigation: Record whether noise is simulated, measured on hardware, or inferred from calibration data. A mitigated result should always be reported alongside the unmitigated result when possible, with the processing method identified.
Metrics: Define the metric before comparing runs. Depending on the task, this could be total variation distance, success probability, expectation-value error, approximation ratio, runtime, circuit depth, or another domain-specific measure. State whether lower or higher is better and include tolerances.
When debugging unexpected differences, the Quantum Circuit Debugging guide and practical debugging workflow can help isolate measurement, indexing, and visualization issues.
Common mistakes
- Saving only screenshots: Images show a result but not the inputs needed to reproduce it. Save raw results and machine-readable metadata.
- Recording only the SDK name: “Qiskit,” “Cirq,” or another framework is not enough. Versions, backend plugins, and compilation settings can affect outputs.
- Comparing logical circuits instead of executed circuits: Hardware usually runs a compiled form. Depth, routing, and native-gate decomposition matter.
- Assuming a seed controls everything: Seeds may cover only one random-number generator or one execution layer.
- Mixing exact and sampled results: An exact probability distribution and a finite-shot histogram should not be treated as equivalent data types.
- Reporting only the best run: Selective reporting hides variability. Keep repeated runs and explain the selection rule.
- Changing post-processing between runs: Make readout correction, thresholding, filtering, normalization, and aggregation explicit.
- Using unstable tests: Tests based on exact sampled counts can fail for legitimate statistical reasons. Use tolerances or test deterministic circuit properties instead. The quantum testing in CI guide covers this distinction.
When to revisit
Revisit your reproducibility record before sharing a benchmark, publishing a result, promoting a notebook into a service, or adding a quantum workflow to CI. Recheck it whenever you upgrade an SDK, simulator, compiler, provider integration, operating system, or dependency lockfile. Hardware experiments deserve an additional review when the target device, calibration context, error-mitigation configuration, or execution service changes.
At the start of a new planning or development cycle, run a small reference circuit and compare its structure, metadata, and baseline metrics with the previous record. If the result changes, do not immediately label it a regression. First compare the manifest: versions, seeds, transpilation, shots, backend, noise, and post-processing. This turns an unexplained difference into a diagnosable workflow change.
Before your next experiment, use this short final checklist:
- Freeze or record the code revision and environment.
- Save the logical and executed circuit representations.
- Record backend, compilation, shots, seeds, and noise settings.
- Store raw results with a timestamp and job identifier.
- Define metrics, baselines, and tolerances before comparing runs.
- Separate deterministic, stochastic, and hardware conclusions.
- Archive the manifest beside the code and results.
Reproducibility is a developer workflow, not a final formatting step. When every run carries its inputs and assumptions with it, quantum circuit benchmarking becomes easier to debug, compare, automate, and revisit as tools evolve.