Genetic Programming Needs Better Benchmarks | dr hab. Wojciech Jaśkowski

by J. McDermott, D.R. White, S. Luke, L. Manzoni, M. Castelli, L. Vanneschi, W. Ja’skowski, K. Krawiec, R. Harper, K. De Jong, Una-May O’Reilly

Abstract:

Genetic programming (GP) is not a field noted for the rigor of its benchmarking. Some of its benchmark problems are popular purely through historical contingency, and they can be criticized as too easy or as providing misleading information concerning real-world performance, but they persist largely because of inertia and the lack of good alternatives. Even where the problems themselves are impeccable, comparisons between studies are made more difficult by the lack of standardization. We argue that the definition of standard benchmarks is an essential step in the maturation of the field. We make several contributions towards this goal. We motivate the development of a benchmark suite and define its goals; we survey existing practice; we enumerate many candidate benchmarks; we report progress on reference implementations; and we set out a concrete plan for gathering feedback from the GP community that would, if adopted, lead to a standard set of benchmarks.

View PDF

Reference:

Genetic Programming Needs Better Benchmarks (J. McDermott, D.R. White, S. Luke, L. Manzoni, M. Castelli, L. Vanneschi, W. Ja’skowski, K. Krawiec, R. Harper, K. De Jong, Una-May O’Reilly), In Proceedings of the fourteenth international conference on Genetic and evolutionary computation conference (Terence Soule, ed.), ACM, 2012.

Bibtex Entry:

@InProceedings{McDermott2012genetic,
  Title                    = {Genetic Programming Needs Better Benchmarks},
  Author                   = {McDermott, J. and White, D.R. and Luke, S. and Manzoni, L. and Castelli, M. and Vanneschi, L. and Ja'skowski, W. and Krawiec, K. and Harper, R. and De Jong, K. and Una-May O'Reilly},
  Booktitle                = {Proceedings of the fourteenth international conference on Genetic and evolutionary computation conference},
  Year                     = {2012},
  Editor                   = {Soule, Terence},
  Organization             = {ACM},
  Pages                    = {791--798},
  Publisher                = {ACM},

  Abstract                 = {Genetic programming (GP) is not a field noted for the rigor of its benchmarking. Some of its benchmark problems are popular purely through historical contingency, and they can be criticized as too easy or as providing misleading information concerning real-world performance, but they persist largely because of inertia and the lack of good alternatives. Even where the problems themselves are impeccable, comparisons between studies are made more difficult by the lack of standardization. We argue that the definition of standard benchmarks is an essential step in the maturation of the field. We make several contributions towards this goal. We motivate the development of a benchmark suite and define its goals; we survey existing practice; we enumerate many candidate benchmarks; we report progress on reference implementations; and we set out a concrete plan for gathering feedback from the GP community that would, if adopted, lead to a standard set of benchmarks.},
  Keywords                 = {Genetic Programming, Benchmarks},
  Url                      = {http://gpbenchmarks.org/wp-content/uploads/2012/08/gpbenchmarks-GECCO2012.pdf}
}