Computational Reproducibility at Exascale Workshop (CRE2019) Report

Where Colorado Convention Center, Room 603
Part of SC19, Denver, CO
When Sunday, November 17, 2019, 9:00 am–12:30 pm
Deadline Tuesday, September 03, 2019
Notifications Monday, October 07, 2019
Full Papers Monday, December 30, 2019, at
Organized byWalid Keyrouz (NIST) and Michael Mascagni (FSU & NIST)


Experimental reproducibility is a cornerstone of the scientific method. As computing has grown into a powerful tool for scientific inquiry, computational reproducibility has been one of the core assumptions underlying scientific computing. With “traditional” single-core CPUs, documenting a numerical result was relatively straightforward. However, hardware developments over the past several decades have made it almost impossible to ensure computational reproducibility or to even fully document a computation without incurring a severe loss of performance. This loss of reproducibility started when systems combined parallelism (e.g., clusters) with non-determinism (e.g., single-core CPUs with out-of-order execution). It has accelerated with recent architectural trends towards platforms with increasingly large numbers of processing elements, namely multicore CPUs and compute accelerators (e.g., GPUs and FPGAs).

Programmers targeting these platforms rely on tools and libraries to produce codes or execute them efficiently. As a result, codes can run efficiently, but have execution details that can be impossible to predict and are often very difficult to understand after execution. Furthermore, parallel implementations often result in code with varying execution orders between runs, leading to non-reproducible computations. The underlying reasons are that (1) the hardware and system software allocate parallel work in ways that are not always specifiable at compile time and (2) the execution often proceeds in an opportunistic manner with the execution order changing between runs. As such, floating-point computations, which are not commutative and associative, can have different execution orders and execute on different processing elements between runs, leading to runs with varying results as a matter of fact. The predictability of systems is further complicated by two issues that are becoming more critical as systems grow in scale: (1) interconnect systems with latencies that are often outside the control of programmers and (2) reliability as the mean time between failure (MTBF) is now measured in hours on large systems. This situation seriously affects the ability to rely on scientific computations as a metrological substitute for experimentation!

Previous Offerings

This workshop combines the Numerical Reproducibility at Exascale workshops (conducted in 2015 and 2016 at SC) and the panel on Reproducibility held at SC16 (originally a BOF at SC15) to address several different issues in reproducibility that arise when computing at Exascale. The first combined workshop, Computational Reproducibility at Exascale 2017 (CRE2017), took place at SC17 and was a success; its web page can be found here; the following edition, CRE2018, took place at SC18 and its web page can be found here. Previous editions of the workshop were NRE2015 and NRE2016; their web pages can be found here and here respectively. We began to offer the International version of this workshop in 2019 as NRE2019 at the ISC-HPC conference in Germany; its website is here.

Workshop Scope

The workshop is meant to address issues of numerical reproducibility as well as approaches and best practices to sharing and running code and the reproducible dissemination of computational results. The workshop is meant to address the scope of the problems of computational reproducibility in HPC in general, and those anticipated as we scale up to Exascale machines in the next decade. The participants of this workshop will include government, academic, and industry stakeholders; the goals of this workshop are to understand the current state of the problems that arise, what work is being done to deal with these issues, and what the community thinks the possible approaches to these problems are.

The workshop is meant to address the scope of the problems of numerical reproducibility in HPC in general and those anticipated as we scale to Exascale machines in the next decade. We initially seek contributions of extended abstracts (two pages) in the areas of computational reproducibility in HPC from academic, government, and industry stakeholders. Areas of interest include, but are not limited to:

Workshop Format

The workshop will have: (1) two plenary talks (25 min + 5 min Q&A each), (2) four contributed talks (15 min + 5 min Q&A each), and (3) a 40-min panel discussion to summarize the problem, current research, and prospects on long-term solutions. The table below gives the workshop’s schedule:




9:00 am

Peter Coveney (UCL, UK)

Reproducibility, computability, and the scientific method (abstract, slides)

9:30 am

G. Kiar, P. de Oliveira Castro, P. Rioux, E. Petit, S.T. Brown, A.C. Evans, T. Glatard

Comparing perturbation models for evaluating stability of post-processing pipelines in neuroimaging (abstract, figshare-repo)

9:50 am

B. Neuman, A. Dubois, L. Monroe, R.W. Robey

Fast, good, and repeatable: summations, vectorization, and reproducibility (abstract, slides)

10:10 am


Food and coffee provided by SC

10:35 am

Howard Bushouse (STScI, USA)

Reproducibility and Provenance of James Webb Space Telescope Data Products (abstract, slides)

11:05 am

T. Imamura, D. Mukunoki, R. Iakymchuk, F. Jézéquel, S. Graillat

Numerical reproducibility based on minimal-precision validation (abstract, slides)

11:25 am

D.H. Bailey

Reproducibility and variable precision computing (abstract, slides)

11:45 am


Panel Discussion on Computational Reproducibility: Howard Bushouse, Peter Coveney, Torsten Hoeffler, Thomas Ludwig, Line Pouchard

All papers submitted to the workshop must be original and not simultaneously submitted to another journal or conference. They will reviewed and the referees will select the papers that will be at the workshop. The selected extended abstracts will be published on the workshop’s website.

Authors of the selected abstracts will be invited to submit full-length versions of their papers to be published in a special issue of the International Journal of High-Performance Computing and Applications (IJHPCA) devoted to Computational Reproducibility. Please note that papers submitted to IJHPCA for the CRE2019 special issue must fall within IJHPCA’s editorial scope; this primarily means that all papers for the special issue have relevance to high-performance computing. These papers will be limited to 20 pages and will be managed through IJHPCA’s website. The deadlines and logistics for IJHPCA submissions are as follows:


Submissions of two-page extended abstracts are sought. The format for the abstracts and full papers should follow the IEEE Conference Proceedings format. Templates are available at IEEE - Manuscript Templates for Conference Proceedings. The full papers should be 20 pages or less.

The abstracts are to be submitted as a PDF document using Easychair at

Important Dates

Tue., Sep. 03, 2019 submission deadline for two page abstracts via
Mon., Oct. 07 2019 notification of authors about their submissions based on rejection, acceptance as a paper, acceptance as a paper and presentation
Mon., Dec. 30, 2019submission of full-length papers
Mon., Feb. 03, 2020 first reviews
Mon., Mar. 30, 2020submission of revised manuscripts

Organizers and Co-Editors

Steering Committee


E-mail: (replace “.at.” by “@”)