Perturbed dataset hypothesis testing methods

Classical hypothesis testing relies on assumptions about the distribution of the noise corrupting the measurements. Confidence regions based on such methods are only as reliable as the underlying assumptions.We created a family of hypothesis testing algorithms that has a few building blocks which can be customized according to the given prior knowledge. The Sign-Perturbed Sums method [1] belongs to this family with proper parameterization of the building blocks based on the symmetry assumption corresponding to the SPS method. We have created an exact confidence hypothesis test within this family that corresponds to independent and identically distributed noise samples. This replaces the symmetry condition with independent and identical distribution of the noise values. We have also shown the importance of accurately analyzing the shape of confidence regions corresponding to confidence sets. For example, these can be very misleading in the case of the SPS method for parameter estimates of linear dynamical systems.