SRA Workshop 7 Notes

Notes for SRA 2021 Workshop 7, “Monte Carlo simulation and probability bounds analysis in R with hardly any data (Instructors: Ferson & Grey)” held 2021-12-05.

Description

Description from https://www.sra.org/events-webinars/annual-meeting/program/annual-meeting-workshops/. Attend virtual session on Pathable.

Workshop #7: Full Day 8AM-5PM | Monte Carlo simulation and probability bounds analysis in R with hardly any data (Instructors: Ferson & Grey)

• Features hands-on examples worked in R on your own laptop, from raw data to final decision.
• Introduces and compares Monte Carlo simulation and probability bounds analysis for developing probabilistic risk analyses when little or no empirical data are available.
• You can use your laptop to work the examples, or just follow along if you prefer.
• The examples illustrate the basic problems risk analysts face: not having much data to estimate inputs, not knowing the distribution shapes, not knowing their correlations, and not even being sure about the model form.
• Monte Carlo models will be parameterized using the method of matching moments and other common strategies.
• Probability bounds will be developed from both large and small data sets, from data with non-negligible measurement uncertainty, and from published summaries that lack data altogether.
• Explains how to avoid common pitfalls in risk analyses, including the multiple instantiation problem, unjustified independence assumptions, repeated variable problem, and what to do when there’s little or no data.
• The numerical examples will be developed into fully probabilistic estimates useful for quantitative decisions and other risk-informed planning.
• Emphasis will be placed on the interpretation of results and on how defensible decisions can be made even when little information is available.
• The presentation style will be casual and interactive.
• Participants will receive handouts of the slides and electronic files with software for the examples.

Presentation

Notes from Slide Presentation:

Does maximum entropy maximize uncertainty? Arguably yes, but in fact it can throw out information, see slide 79.

Application of these methods requires judgment; choices of distributions and methods should be defensible, but ultimately are analyst opinions.

Monte Carlo assumes that each variable is independent, and different. Be careful about how values are represented, see slide 88, and be careful about variables that are not independent.

Probability bounds analysis (slide 111) is more rigorous than Monte Carlo, and works with even terrible data.

Robust Bayesian analysis (slide 115) addresses the problem of arbitrary priors by using multiple appropriate priors - this is the missing piece that has always bothered me about Bayesian math.

PBA is really interesting way to communicate both variability and ambiguity. Is PBA better at communicating ambiguity than MC? [Quite possibly]

What if we added a p-box to a Monte Carlo CDF? (or complementary CDF?) this should add insight into the level of ambiguity (epistemic uncertainty).

What we’re doing with FAIR and similar analyses is really just scratching the surface of what’s possible.

Second-order Monte Carlo analysis will be bounded within the p-box, and it’s highly unlikely it will show the complete p-box.

Are Monte Carlo and PBA combined? Yes, they are “nested” typically. This is what’s done in Info-Gap Analysis - first present a more deterministic solution with all assumptions, then a p-box showing when the assumptions are relaxed.

Summary of the “Neuroscience of Risk”: address people’s perception of incertitude and variability (which are different) in risk communication - see slides 231-234.

pba.r has the ability to constrain bounds based on dependence and correlation.

This is a bit off-topic, do you have general advice on applying these principles risk analysis and communication (for people with less math/statistics training)? Is it just as simple as “work to communicate both the variability and incertitude?”

“Fermi Estimates” is a pretty good description of FAIR and Hubbard-style risk quantification.

This paper and podcast make a good case to simply average the expert estimates - that there’s no clear evidence that there’s anything better. https://doi.org/10.1016/j.ssci.2017.02.018 - https://safetyofwork.com/episodes/ep40-when-should-we-trust-expert-opinions-about-risk

Paper on elliptical estimates from the Fermi estimation: https://scholarworks.utep.edu/cgi/viewcontent.cgi?article=1130&context=cs_techrep

Recommends Mark Burgman’s work on expert elicitation.

Overall, Alex and Scott are making the case for using p-boxes to make better decisions under variability and incertitude (risk).

Notes

This is the game on Alex’s shelf. Risk with hidden objectives: https://en.wikipedia.org/wiki/RisiKo!

This looks to be a pretty good introduction to basic Monte Carlo using R. https://www.countbayesie.com/blog/2015/3/3/6-amazing-trick-with-monte-carlo-simulations

Workshop Site

Additional links and downloads available at https://sites.google.com/site/hardlyanydata (files have been downloaded to this repository):

This tutorial explains how you can develop a fully probabilistic risk analysis even though there may be very little empirical data available on which to base the analysis. It compares the strengths and weakness of a traditional Monte Carlo assessment with probability bounds analysis in the R statistical computing environment which is freely available over the internet.

Overview of topics

Introduction

Welcome Case studies: civil and aerospace engineering, exposure analysis, and conservation biology Installation of R and workshop software

Monte Carlo simulation

Random values and replications Distributions Independent and perfect sampling Calculations in R Interpreting results: tails are where the action is

Probability bounds analysis

Kinds of uncertainty: the ‘open question’ Probability boxes Independent, perfect, and Fréchet Calculation in R Interpreting results: fully probabilistic answers

Approximation versus enveloping

Integrating Monte Carlo and probability bounding Fixed but unknown, or actually varying? Distributions, p-boxes, and interval ranges What you know and what you assume

Selecting input distributions

Moments and ranges Maximum likelihood and maximum entropy Confidence boxes Shape assumptions to refine estimates

Correlations and dependencies

Making no assumptions about dependence Perfect correlations Dispersive Monte Carlo dependence Independence maximizes entropy

Case studies in R

Civil engineering: dam safety Aerospace engineering: spacecraft design Environmental protection: contaminant exposure analysis Conservation biology: estimating endangerment

Model uncertainty

What-if studies Stochastic mixtures and Bayes model averaging Bounding methods Conservative methods for polynomial models

Sensitivity analyses

Bang-for-buck control analysis Value of information: what data to collect More samples or better measurements

Presenters

Alexander Wimbush, Ph.D. (2022), University of Liverpool, optimising medical diagnostic algorithms under uncertainty, communicating risks to patients, and calculation with confidence structures and possibility distributions; https://www.researchgate.net/profile/Alexander-Wimbush

Scott Ferson, Chair of Risk and Uncertainty at the University of Liverpool School of Engineering and director of the Institute for Risk and Uncertainty; developing reliable mathematical and statistical tools for risk assessments and on methods for uncertainty analysis when empirical information is very sparse

Related links

Society for Risk Analysis https://www.sra.org Society for Risk Analysis Annual Meeting https://www.sra.org/events-webinars/annual-meeting/ On-line workshop registration https://sra.membershipsoftware.org/ev_calendar_day.asp?eventid=22&evreg1=2&t=&testmtype=&pub=1 https://members.sra.org/ev_calendar_day.asp?eventid=22&evreg1=2&t=&testmtype=&pub=1 Society for Imprecise Probabilities https://sipta.org/ Liverpool Institute for Risk and Uncertainty https://riskinstitute.uk https://www.liverpool.ac.uk/risk-and-uncertainty/ Sandia National Laboratories’ Epistemic Uncertainty Project https://sites.google.com/site/uncertaintyprojection/ Applied Biomathematics uncertainty projects https://sites.google.com/site/abuncertainty/ NSF workshop on risk perception and communication https://sites.google.com/site/montaukriskcommunication/