Semi-parametric inference for the means of heavy-tailed distributions

Abstract

Heavy tailed distributions present a tough setting for inference. They are also common in industrial applications, particularly with Internet transaction datasets, and machine learners often analyze such data without considering the biases and risks associated with the misuse of standard tools. This article outlines a procedure for inference about the (possibly conditional) mean of a heavy tailed distribution that combines nonparametric inference for the bulk of the support with parametric inference – motivated from extreme value theory – for the heavy tail. We are able to derive analytic posterior conditional means and variances for the expected value of a heavy tailed distributivo. We also introduce a simple and novel independence Metropolis Hastings algorithm that samples from the distribution for tail parameters via small adjustments to a parametric bootstrap, and through this algorithm are able to provide comparisons between our framework and frequentist semiparametric inference. We also provide a modeling extension that shrinks tails across distributions to an overall background tail. We illustrate on two examples: treatment effect estimation on a set of 72 A/B experiments, and the fitting of regression trees for prediction of user spending. Both use data from tens of millions of users of eBay.com.