Better GP benchmarks: community survey results and proposals

by David R. White, James McDermott, Mauro Castelli, Luca Manzoni, Brian Goldman, Gabriel Kronberger, Wojciech Ja’skowski, Una-May O’Reilly, Sean. Luke
Abstract:
We present the results of a community survey regarding genetic programming (GP) benchmark practices. Analysis shows broad consensus that improvement is needed in problem selection and experimental rigor. While views expressed in the survey dissuade us from proposing a large-scale benchmark suite, we find community support for creating a blacklist of “toy problems.” We provide a set of alternative problems named GPBench2012 to replace the blacklisted ones, a discussion on improving experimental rigor, and a listing of challenging problems in the hope of improving GP research.
Reference:
Better GP benchmarks: community survey results and proposals (David R. White, James McDermott, Mauro Castelli, Luca Manzoni, Brian Goldman, Gabriel Kronberger, Wojciech Ja’skowski, Una-May O’Reilly, Sean. Luke), In Genetic Programming and Evolvable Machines, volume 14, 2013.
Bibtex Entry:
@Article{White2012better,
  Title                    = {Better {GP} benchmarks: community survey results and proposals},
  Author                   = {White, David R. and McDermott, James and Castelli, Mauro and Manzoni, Luca and Goldman, Brian and Kronberger, Gabriel and Ja'skowski, Wojciech and Una-May O'Reilly and Luke, Sean.},
  Journal                  = {Genetic Programming and Evolvable Machines},
  Year                     = {2013},
  Number                   = {1},
  Pages                    = {3-29},
  Volume                   = {14},
  if                       = {[IF 2013: 1.065]},

  Abstract                 = {We present the results of a community survey regarding genetic programming (GP) benchmark practices. Analysis shows broad consensus that improvement is needed in problem selection and experimental rigor. While views expressed in the survey dissuade us from proposing a large-scale benchmark suite, we find community support for creating a blacklist of ``toy problems.'' We provide a set of alternative problems named GPBench2012 to replace the blacklisted ones, a discussion on improving experimental rigor, and a listing of challenging problems in the hope of improving GP research.},
  Doi                      = {10.1007/s10710-012-9177-2},
  Keywords                 = {Genetic programming; Benchmarks; Community survey},
  Url                      = {http://www.cs.put.poznan.pl/wjaskowski/pub/papers/White2013BetterGP.pdf}
}

This entry was posted by . Bookmark the permalink.