[Bbob-discuss] Request for BBOB algorithms.

Nikolaus Hansen Nikolaus.Hansen at lri.fr
Tue Feb 28 13:55:16 CET 2012

Hi Petr,

On Tue, 28 Feb 2012 09:54:41 +0100, Petr Pošík <posik at labe.felk.cvut.cz>  

> Hi Olivier.
> Regarding the running time: you are right that the graphs produced by  
> postprocessing do not show the running time. Nevertheless, the BBOB
> methodology suggests (requires?) to run also a timing experiment to

yes, it is a requirement, not a suggestion.  

> indicate
> the wall-clock time needed to find a solution. (See e.g. the  
> exampetiming.m
> file). In the first BBOB issue in 2009, almost all papers actually  
> contained
> the information about the time requirements of the algorithms. In 2010,  
> I am
> not that sure...
> So, the questions are:
> - Should the BBOB methodology be modified to really REQUIRE the info  
> about
> the time demands of the algorithms?

+1 for me. Well, it's not a modification, it's just keeping the demand.

> - Should/can the COCO framework be modified to do this more  
> automatically?

undecided. Looks pretty automatic to me already. Some authors actually  
presented results for larger dimensions than requested, which we might  
discourage making it more formal.

> - Should the time demands be more emhasized in the latex templates?

undecided. There could be an empty paragraph with two sentences explaining  
the experiment just such that it is harder to forget.


> Cheers, Petr
> From: bbob-discuss-bounces at www.lri.fr
> [mailto:bbob-discuss-bounces at www.lri.fr] On Behalf Of Olivier Teytaud
> Sent: Tuesday, February 28, 2012 12:21 AM
> To: Daniel Molina
> Cc: Nikolaus Hansen; Francisco Herrera; bbob-discuss at www.lri.fr
> Subject: Re: [Bbob-discuss] Request for BBOB algorithms.
> Hi Daniel + thanks for this interesting link.
> In particular I see that this site includes higher dimensions than BBOB
> (without downloading, changing parameters and rerunning :-) ). For a
> starting project (using optimization, but not focusing on it) I must  
> choose
> which algorithms downloading/implementing/testing, and this test bed  
> might
> be the best approximation for my needs (I do not consider 40-D as a good
> approximation of 40000D).
> Another (related) remark on BBOB is that computational costs are usually
> given in terms of number of iterations; computation times are not given  
> in
> the usual graphs; whereas it is an important issue; if you run 40-D only  
> and
> don't show the evolution of the running time, you might not see that some
> algorithms can run in dimension 10 000 and some others can not. This is
> certainly not a negligible issue.
> Best regards,
> Olivier
> The source code of my proposal in previous BBOB, MA-CMA-LS Chains, is
> available in
> http://sci2s.ugr.es/EAMHCO/#Software
> Also, the source code of several other algorithms (with its reference, of
> course), like MOS, is also
> available in the same website.
> There is also a section on the website in
> http://sci2s.ugr.es/EAMHCO/#SS

Science is a way of trying not to fool yourself.

The first principle is that you must not fool yourself, and you
are the easiest person to fool. So you have to be very careful
about that. After you've not fooled yourself, it's easy not to
fool other[ scientist]s. You just have to be honest in a
conventional way after that.
                                     -- Richard P. Feynman

Nikolaus Hansen
INRIA, Research Centre Saclay – Ile-de-France
Machine Learning and Optimization group (TAO)
University Paris-Sud (Orsay)
LRI (UMR 8623), building 490
91405 ORSAY Cedex, France
Phone: +33-1-691-56495, Fax: +33-1-691-54240
URL: http://www.lri.fr/~hansen

More information about the bbob-discuss mailing list