[Bbob-discuss] Request for BBOB algorithms.

Olivier Teytaud olivier.teytaud at gmail.com
Wed Feb 29 01:14:16 CET 2012


Hi Niko and others;
I try to answer all your emails, Niko, in one email for convenience; with a
short version first and details below.
In short, I agree that
there is some progress in new Bbob codes for noisy problems (compared to
old versions), but

- for the issue of computation
   time for high-dimension, Bbob is less convincing than other available
testbeds (you might
   just consider that this is not the purpose of Bbob...);

- and I think that changing details in the criteria and presentation
   completely change the overall results, and for checking this carefully
one would have
   to download all codes and rerunning, changing just a few details each
time for
   understanding exactly which assumption leads to which conclusion. You
might consider
   that adding the constraint of making codes fully downloadable including
the interfacing
   with Bbob is too much a requirement for Bbob participants - in my humble
opinion it's
   just "tar -zcvf myBbobExperiments.tgz myDirectory" so it should not be
that much a trouble.

Details below; best regards,
Olivier

================================
I disagree with the idea that running times would put the main focus on
implementation details.
Complexity is clearly an issue when working > 1000D. It's not an
implementation detail which
switches from something which is intractable in dim 100 to something which
is fast in dim 1000.
Yes, you're right, it's number of function-evaluations, not number of
iterations, but I guess that you know
that is not very important about the fact that this number hides the
running time.

I am not silently assuming that computing times are not at all available, I
just say that it's not what
is primarily visible. Importantly, testbeds which have provided results in
1000D are more satisfactory
for this; even if the detailed computation times are not presented, at
least it shows that it was possible
to run the algorithm in dimension 1000.

I don't say the graphs do not correspond to raw data, I just say that when
I downloaded raw data, I plotted
results without trying to follow the Bbob guidelines, and I got pictures
which give very different
feelings.

So it makes me think that results depend a lot on the precise methodology
chosen for plotting
results. So I'd find that interesting to download the exact codes,
including the interfacing, and to re-run
everything. But as this takes quite a lot of time if downloading,
interfacing included, is not available,
I will just not do it. After all there are other testbeds in which results
can be found. But that would
make sense for Bbob - it would make it possible to check quickly which
criteria leads to which conclusions.

 For the 32bits/64 bits: some years ago, one of the organizers told me that
>> some of the algorithms used by the organizers were run in 32bits, and
>> some in 64 bits.
>>
>
> And so what?
>

For very long runs, the difference 32bits / 64 bits matters a lot.
You mentioned that you don't want to focus on implementation details, but
this precisely introduces
a big implementation issue.

For examples of high-dimension problems: one of my favorite is sequential
decision making on networks.
Maybe I am an extremal point with my interest on problem with 10000s
dimensions and more, but I guess 200, 1000, 10000, is not rare; and I guess
that problems which are 10D and one which you have hundreds of millions of
function evaluations are not the main problem of nonlinear optimization.

For many problems, just the expected fitness for a fixed time budget
(possibly approximated by the number of function-evaluations if you hate
computation times...) is meaningful. One can not just consider that this
criterion is a bad criterion. It's just another criterion. And with limited
budget it can be quite realistic, and focus on transient behavior
(something which is far from being uninteresting in evolutionary
algorithms) rather than on convergence rate.

Best regards,
Olivier
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lri.fr/pipermail/bbob-discuss/attachments/20120229/ec064973/attachment-0001.html>


More information about the bbob-discuss mailing list