The data-uninformed Parametric Bootstrap Cross-fitting Method (PBCM) generates synthetic data from two models of a phenomenon with given model parameter values, and then cross-fits the models to these data. The result is two distributions of the goodness of fit metric ΔGoF=GoF1−GoF2, where GoF1 is the fit of model 1 and GoF2 the fit of model 2.
pbcm.du( fun1, fun2, genfun1, genfun2, reps, args1 = NULL, args2 = NULL, genargs1 = NULL, genargs2 = NULL, print_genargs = TRUE, verbose = TRUE, GoFname = "GoF" )
| fun1 | First modelling function |
|---|---|
| fun2 | Second modelling function |
| genfun1 | Generator function for first model |
| genfun2 | Generator function for second model |
| reps | Number of Monte Carlo repetitions |
| args1 | List of arguments passed to |
| args2 | List of arguments passed to |
| genargs1 | List of arguments passed to |
| genargs2 | List of arguments passed to |
| print_genargs | Whether the generator argument values should be included in output (see Details) |
| verbose | If |
| GoFname | Name of the element returned by |
A data frame in long format with the following columns:
repMonte Carlo repetition number
generatorGenerating model
GoF1Goodness of fit of model 1
GoF2Goodness of fit of model 2
DeltaGoFEquals GoF1 - GoF2
In addition to these columns, if print_genargs == TRUE, each argument in the lists genargs1 and genargs2 is included as a column of its own, with the argument's name prefixed by "genargs1_" or "genargs2_".
Functions fun1 and fun2 must take an argument named data in addition to any arguments specified in args1 and args2; this is used to pass the synthetic data generated by genfun1 and genfun2. Moreover, these functions must return a list with at least one element carrying the goodness of fit; the name of this element may be specified through the GoFname argument, by default the string "GoF" is assumed.
Wagenmakers, E.-J., Ratcliff, R., Gomez, P. & Iverson, G. J. (2004) Assessing model mimicry using the parametric bootstrap. Journal of Mathematical Psychology, 48(1), 28–50. https://doi.org/10.1016/j.jmp.2003.11.004
x <- seq(from=0, to=1, length.out=100) mockdata <- data.frame(x=x, y=x + rnorm(100, 0, 0.5)) myfitfun <- function(data, p) { res <- nls(y~a*x^p, data, start=list(a=1.1)) list(a=coef(res), GoF=deviance(res)) } mygenfun <- function(a, p) { x <- seq(from=0, to=1, length.out=100) y <- a*x^p + rnorm(100, 0, 0.5) data.frame(x=x, y=y) } pbcm.du(fun1=myfitfun, fun2=myfitfun, genfun1=mygenfun, genfun2=mygenfun, reps=20, args1=list(p=1), args2=list(p=2), genargs1=list(a=1.1, p=1), genargs2=list(a=1.1, p=2))#> Initializing output data frame... #> Bootstrapping... #> | | | 0% | |==== | 5% | |======= | 10% | |========== | 15% | |============== | 20% | |================== | 25% | |===================== | 30% | |======================== | 35% | |============================ | 40% | |================================ | 45% | |=================================== | 50% | |====================================== | 55% | |========================================== | 60% | |============================================== | 65% | |================================================= | 70% | |==================================================== | 75% | |======================================================== | 80% | |============================================================ | 85% | |=============================================================== | 90% | |================================================================== | 95% | |======================================================================| 100%#> genargs1_a genargs1_p genargs2_a genargs2_p rep generator GoF1 GoF2 #> 1 1.1 1 NA NA 1 model1 25.86491 29.89399 #> 2 1.1 1 NA NA 2 model1 20.45596 24.54978 #> 3 1.1 1 NA NA 3 model1 21.16950 24.06770 #> 4 1.1 1 NA NA 4 model1 24.61148 28.81452 #> 5 1.1 1 NA NA 5 model1 26.83535 29.63272 #> 6 1.1 1 NA NA 6 model1 25.86201 27.19919 #> 7 1.1 1 NA NA 7 model1 19.87490 21.50302 #> 8 1.1 1 NA NA 8 model1 22.39677 25.41388 #> 9 1.1 1 NA NA 9 model1 27.36766 33.20927 #> 10 1.1 1 NA NA 10 model1 26.05621 30.13748 #> 11 1.1 1 NA NA 11 model1 24.17376 25.75219 #> 12 1.1 1 NA NA 12 model1 22.18600 25.31316 #> 13 1.1 1 NA NA 13 model1 19.42765 18.74827 #> 14 1.1 1 NA NA 14 model1 27.17106 30.18763 #> 15 1.1 1 NA NA 15 model1 17.47694 21.77938 #> 16 1.1 1 NA NA 16 model1 23.93401 29.26444 #> 17 1.1 1 NA NA 17 model1 33.63317 35.93311 #> 18 1.1 1 NA NA 18 model1 16.11678 19.69319 #> 19 1.1 1 NA NA 19 model1 18.44598 19.51856 #> 20 1.1 1 NA NA 20 model1 18.87330 18.72844 #> 21 NA NA 1.1 2 1 model2 28.20654 29.10725 #> 22 NA NA 1.1 2 2 model2 26.09868 25.19643 #> 23 NA NA 1.1 2 3 model2 23.74487 21.83687 #> 24 NA NA 1.1 2 4 model2 22.11130 18.85654 #> 25 NA NA 1.1 2 5 model2 31.61005 29.51508 #> 26 NA NA 1.1 2 6 model2 22.85500 21.27322 #> 27 NA NA 1.1 2 7 model2 24.01693 22.96493 #> 28 NA NA 1.1 2 8 model2 34.86692 28.69820 #> 29 NA NA 1.1 2 9 model2 20.26879 19.14675 #> 30 NA NA 1.1 2 10 model2 24.57482 23.02979 #> 31 NA NA 1.1 2 11 model2 30.97166 29.15267 #> 32 NA NA 1.1 2 12 model2 25.80460 24.28906 #> 33 NA NA 1.1 2 13 model2 24.63727 23.10712 #> 34 NA NA 1.1 2 14 model2 23.14107 20.51857 #> 35 NA NA 1.1 2 15 model2 24.94701 24.25816 #> 36 NA NA 1.1 2 16 model2 26.89643 27.15460 #> 37 NA NA 1.1 2 17 model2 29.92769 28.97220 #> 38 NA NA 1.1 2 18 model2 25.12921 23.41655 #> 39 NA NA 1.1 2 19 model2 25.15284 23.79519 #> 40 NA NA 1.1 2 20 model2 26.34246 23.62479 #> DeltaGoF #> 1 -4.0290759 #> 2 -4.0938204 #> 3 -2.8982000 #> 4 -4.2030392 #> 5 -2.7973627 #> 6 -1.3371830 #> 7 -1.6281171 #> 8 -3.0171122 #> 9 -5.8416030 #> 10 -4.0812684 #> 11 -1.5784317 #> 12 -3.1271680 #> 13 0.6793775 #> 14 -3.0165707 #> 15 -4.3024410 #> 16 -5.3304252 #> 17 -2.2999460 #> 18 -3.5764066 #> 19 -1.0725768 #> 20 0.1448584 #> 21 -0.9007126 #> 22 0.9022480 #> 23 1.9079996 #> 24 3.2547641 #> 25 2.0949655 #> 26 1.5817771 #> 27 1.0519948 #> 28 6.1687239 #> 29 1.1220317 #> 30 1.5450349 #> 31 1.8189916 #> 32 1.5155317 #> 33 1.5301444 #> 34 2.6224936 #> 35 0.6888477 #> 36 -0.2581709 #> 37 0.9554955 #> 38 1.7126611 #> 39 1.3576457 #> 40 2.7176621sweep <- lapply(X=seq(from=0.5, to=1.5, by=0.1), FUN=function(X) { pbcm.du(fun1=myfitfun, fun2=myfitfun, genfun1=mygenfun, genfun2=mygenfun, reps=20, args1=list(p=1), args2=list(p=2), genargs1=list(a=X, p=1), genargs2=list(a=X, p=2)) })#> Initializing output data frame... #> Bootstrapping... #> | | | 0% | |==== | 5% | |======= | 10% | |========== | 15% | |============== | 20% | |================== | 25% | |===================== | 30% | |======================== | 35% | |============================ | 40% | |================================ | 45% | |=================================== | 50% | |====================================== | 55% | |========================================== | 60% | |============================================== | 65% | |================================================= | 70% | |==================================================== | 75% | |======================================================== | 80% | |============================================================ | 85% | |=============================================================== | 90% | |================================================================== | 95% | |======================================================================| 100% #> Initializing output data frame... #> Bootstrapping... #> | | | 0% | |==== | 5% | |======= | 10% | |========== | 15% | |============== | 20% | |================== | 25% | |===================== | 30% | |======================== | 35% | |============================ | 40% | |================================ | 45% | |=================================== | 50% | |====================================== | 55% | |========================================== | 60% | |============================================== | 65% | |================================================= | 70% | |==================================================== | 75% | |======================================================== | 80% | |============================================================ | 85% | |=============================================================== | 90% | |================================================================== | 95% | |======================================================================| 100% #> Initializing output data frame... #> Bootstrapping... #> | | | 0% | |==== | 5% | |======= | 10% | |========== | 15% | |============== | 20% | |================== | 25% | |===================== | 30% | |======================== | 35% | |============================ | 40% | |================================ | 45% | |=================================== | 50% | |====================================== | 55% | |========================================== | 60% | |============================================== | 65% | |================================================= | 70% | |==================================================== | 75% | |======================================================== | 80% | |============================================================ | 85% | |=============================================================== | 90% | |================================================================== | 95% | |======================================================================| 100% #> Initializing output data frame... #> Bootstrapping... #> | | | 0% | |==== | 5% | |======= | 10% | |========== | 15% | |============== | 20% | |================== | 25% | |===================== | 30% | |======================== | 35% | |============================ | 40% | |================================ | 45% | |=================================== | 50% | |====================================== | 55% | |========================================== | 60% | |============================================== | 65% | |================================================= | 70% | |==================================================== | 75% | |======================================================== | 80% | |============================================================ | 85% | |=============================================================== | 90% | |================================================================== | 95% | |======================================================================| 100% #> Initializing output data frame... #> Bootstrapping... #> | | | 0% | |==== | 5% | |======= | 10% | |========== | 15% | |============== | 20% | |================== | 25% | |===================== | 30% | |======================== | 35% | |============================ | 40% | |================================ | 45% | |=================================== | 50% | |====================================== | 55% | |========================================== | 60% | |============================================== | 65% | |================================================= | 70% | |==================================================== | 75% | |======================================================== | 80% | |============================================================ | 85% | |=============================================================== | 90% | |================================================================== | 95% | |======================================================================| 100% #> Initializing output data frame... #> Bootstrapping... #> | | | 0% | |==== | 5% | |======= | 10% | |========== | 15% | |============== | 20% | |================== | 25% | |===================== | 30% | |======================== | 35% | |============================ | 40% | |================================ | 45% | |=================================== | 50% | |====================================== | 55% | |========================================== | 60% | |============================================== | 65% | |================================================= | 70% | |==================================================== | 75% | |======================================================== | 80% | |============================================================ | 85% | |=============================================================== | 90% | |================================================================== | 95% | |======================================================================| 100% #> Initializing output data frame... #> Bootstrapping... #> | | | 0% | |==== | 5% | |======= | 10% | |========== | 15% | |============== | 20% | |================== | 25% | |===================== | 30% | |======================== | 35% | |============================ | 40% | |================================ | 45% | |=================================== | 50% | |====================================== | 55% | |========================================== | 60% | |============================================== | 65% | |================================================= | 70% | |==================================================== | 75% | |======================================================== | 80% | |============================================================ | 85% | |=============================================================== | 90% | |================================================================== | 95% | |======================================================================| 100% #> Initializing output data frame... #> Bootstrapping... #> | | | 0% | |==== | 5% | |======= | 10% | |========== | 15% | |============== | 20% | |================== | 25% | |===================== | 30% | |======================== | 35% | |============================ | 40% | |================================ | 45% | |=================================== | 50% | |====================================== | 55% | |========================================== | 60% | |============================================== | 65% | |================================================= | 70% | |==================================================== | 75% | |======================================================== | 80% | |============================================================ | 85% | |=============================================================== | 90% | |================================================================== | 95% | |======================================================================| 100% #> Initializing output data frame... #> Bootstrapping... #> | | | 0% | |==== | 5% | |======= | 10% | |========== | 15% | |============== | 20% | |================== | 25% | |===================== | 30% | |======================== | 35% | |============================ | 40% | |================================ | 45% | |=================================== | 50% | |====================================== | 55% | |========================================== | 60% | |============================================== | 65% | |================================================= | 70% | |==================================================== | 75% | |======================================================== | 80% | |============================================================ | 85% | |=============================================================== | 90% | |================================================================== | 95% | |======================================================================| 100% #> Initializing output data frame... #> Bootstrapping... #> | | | 0% | |==== | 5% | |======= | 10% | |========== | 15% | |============== | 20% | |================== | 25% | |===================== | 30% | |======================== | 35% | |============================ | 40% | |================================ | 45% | |=================================== | 50% | |====================================== | 55% | |========================================== | 60% | |============================================== | 65% | |================================================= | 70% | |==================================================== | 75% | |======================================================== | 80% | |============================================================ | 85% | |=============================================================== | 90% | |================================================================== | 95% | |======================================================================| 100% #> Initializing output data frame... #> Bootstrapping... #> | | | 0% | |==== | 5% | |======= | 10% | |========== | 15% | |============== | 20% | |================== | 25% | |===================== | 30% | |======================== | 35% | |============================ | 40% | |================================ | 45% | |=================================== | 50% | |====================================== | 55% | |========================================== | 60% | |============================================== | 65% | |================================================= | 70% | |==================================================== | 75% | |======================================================== | 80% | |============================================================ | 85% | |=============================================================== | 90% | |================================================================== | 95% | |======================================================================| 100%sweep <- do.call(rbind, sweep) sweep$parameter <- ifelse(is.na(sweep$genargs1_a), sweep$genargs2_a, sweep$genargs1_a) if (FALSE) { library(ggplot2) g <- ggplot(sweep, aes(x=DeltaGoF, fill=generator)) + geom_density(alpha=0.5) g <- g + facet_wrap(.~parameter) print(g) }