Skip to content

WorkShop13_HackfestNotes

Jump to bottom
Vitalii Koshura edited this page Apr 12, 2023 · 2 revisions

Welcome to the BOINC'13 hackfest notepad

  • Misc ** Tools for hackfest communication all year long Two days of hackfest was probably more productive. How could we keep doing this for the whole year ? Matt suggested to use Jira and Tristan made a demo/presentation of it: https://www.atlassian.com/software/jira Jira is maintained by the bitbucket company. Maybe the same kind of things could be done with github. Then the community behind the tool is also important to take into account. ** Suggestions and requests
  • Christian: deadline extension for workunits. This requires to update the client. ** Next BOINC workhop Hawai`i ? More seriously Budapest may be an option.
  • BOINC on Android , making Android Apps

Joachim, Matt, Keith, Uwe ** Expected Making Android app is not a group thing so maybe not for today, although Uwe is interested in learning this... There is a wiki page about this that could be tested.

What we should discuss instead:

  • Discussing the UI. What could be improved. An outside eye would help.
  • Helping with the registration, google account setup or OpenID. ** What was done
  • Uwe tried the tool chain, used the instructions on the BOINC wiki page. It mostly worked and he improved them.
  • Regarding google/openID registration: they completed a proof of concept that included testing website integration, android application integration and reviewed the requirements in order to make it work within using wxWidgets on the BOINC client. I could not follow all the technical works that remains for it to work but David and Kevin did so they should remember it....
  • Multi-user projects Arnaud, Lionel, Wenjing, Kevin. ** Points to take into account: *** Batch prioritization
    • multi-user
    • multi-app/project

  • better user contribution in term of volunteer machines may mean better machine share (coallitions ?) *** Batch completion

  • task granularity matching (GPU/CPU/Android)

  • machine availability and speed to accelerate batch completion

  • homogeneous redundancy

  • how to estimate the expected runtime of B given the project's entire resource

    A basic mechanism could be that batches are prioritized according to some fairness mechanism and whenever a volunteer requests work, we try to find a job suitable for him but with stricter "SLA". We would give him a job only if we have good confidence it will be able to process it in time, i.e. if there is a chance that the job completes before the expected deadline of the batch.

    Issue: a wrong estimation of batch completion will lead to starving jobs... Possible solution: keep a separate queue of high priority jobs with higher replication / which are sent to reliable fast machines

    There is a mechanism in the server that "forces" the client to contact the server every so often, to make sure the client reports a job as soon as it's completed instead of waiting for the deadline.

**** BOINC Brainstorming This is a follow up of the with David. ***** Wiki design documents. PortalFeatures MultiUser MultiUserPriority JobPrioritization ***** Policy in six steps ****** First step: compute batch priorities ******* Goal The goal of this first step serves two goals: 1. obtain priorities to order batches 2. give a fair share of resources (without necessarily taking into account the preferences of the volunteers) ******* Solution As a point of simplification, we don't take into account volunteers preferences in the sharing of the platform so we use the whole aggregated power of the platform to compute the estimated runtime of the batches.

   Based on user shares and runtime estimates of batches, we compute
   Logical Start Time of users and batches and Logical End Time of
   batches. Then we prioritize batches by increasing Logical Start
   Time.

   This is described here:
   [PortalFeatures](PortalFeatures)#Prioritizingbatches

   This is done at batch submission. Unfortunately, there is a
   problem with this approach, which we evoke now

******* What if LST of some users become incredibly large ? That happens if one or two users are inactive for a long time. Is it a problem ?

       Discussion about how much "burst" we could allow. It would be very
       nice to be able to say that a user does not have more "advantage"
       if he stays inactive for more than some time (say 1 week). One way
       of doing this is to compute the virtual schedule by assigning
       shares based on active users instead of all users (like done in
       http://rr.liglab.fr/research_report/RR-LIG-033_orig.pdf), and
       constraint Logical Start Times to be at least "now - 1 week".

       We need to take into account the problem of the computing cost
       of recomputing everything each time a user queue changes state
       from being empty or not (optimization concern here).

******* What if LET is badly estimated ****** Second step: compute batch deadlines ******* Goal The objective of this step is to provide an estimate which allows to perform "resource selection": those "batch deadlines" will be used to prevent slow machines to execute jobs from this batch. Too tight deadlines may incur starvation, so we will add a deadline extension mechanism. But even with this mechanism, too tight deadlines will slow the progress of the batch by excluding not so slow machines.

On the other hand, too loose deadlines will slow the progress
       of the batch by accepting slower machines that could/should
       have contributed to a lower priority batch. This may seem
       strange but there could be a very good reason for doing
       this. Systematically excluding slow workers will require to
       know about jobs a long time in advance, which may be
       problematic. For example, in WCG, they only load batches one
       day in advance and cannot load more than this. So this kind
       of projects should have much looser deadlines to make sure
       every volunteer gets a batch it can work on.

******* How to do it The server computes an "estimated" schedule (still based on a fluid view of the platform).

   *Recheck whether this makes sense or not*
   There are at least 2 ways of doing this:

  • Arnaud: Very optimistic one. If we exclude slow hosts, and perform a little bit of anticipated replication near the end of the batch, and if batches are large enough, estimated execution times are actually quite close to reality. I think this is somehow illustrated here: http://www.cs.technion.ac.il/~dang/conf_papers/SilbersteinSGS09.pdf‎ But it is actually also necessary to take the number of jobs in the batch into account (if it has too few jobs, then estimating its finishing time based on the power of the whole platform is overly optimistic -- even if actually, the slow machines have started working on this batch, before).

         Crude proposition :
     For batch $i$ (sorted by priority)
          T_i = (\sum_{j=0}^i C_j)/(Total platform power)
          R_j = estimated Computational Cost of Batch j

          T'_i = Expected turnaround time of the (\sum_j=0^i N_j)-fastest
          machines
            N_j = number of jobs in batch j

         Estimated finish time of Batch i = max(T_i, T'_i)

   Note that this is may be quite inaccurate when the system is
   not in steady state because the project ran out of work. The
   estimation for the first jobs is going to be completely
   of. We will have the same kind of trouble if the batches are
   relatively small, hence the next proposals.

  • Lionel: supposedly more accurate, you add the power of the fastest machines until you get to complete the whole batch. in order of priority, and based on distribution of turnaround times of machines, compute a deadline that would allow sufficiently many machines to finish all jobs. I'm not really sure how to do this without keeping track of the queue of each machine, which is something we do NOT want to do. Maybe by splitting the machines in a small number of groups with similar performance ? - Kevin: maybe smarter but complex whenever the system gets empty, Arnaud's estimation is way too optimistic. So we could try to estimate the aggregate power of the system as time goes before entering steady state and integrate this curve to evaluate the estimated completion time of the batch.

  • Kevin 2: divide the batch cost by the aggregated power of the say 50% fastest machine. Somehow, it's the same as option 1 but giving some slack, maybe just another way of thinking of it. - Kevin 3 (maybe the preferred one ?): if batches are much smaller than available host, use the median speed of machines, and compute home much time it would take to complete 1 job on this machine. Within such time bound, we would expect "90-95%" of the jobs to be completed. Some others will have to be resubmited but only on the fastest machines. So recompute the same value but with the 90%-fastest machine.

     In any case, it is important to set a "minimum value" for
 deadlines to avoid starving of batches.

 But apparently it is not necessary to give a very high slack at
 this step. Furthermore, these batch deadlines are likely to
 have to be extended as time goes if we realize they are too
 tight and exclude too many people. This could be done with
 thresholds but we don't have a nice idea of a general solution.

 Yet, after discussing with Kevin, we realized some "bad"
 situations may happen, and that it may call for some kind of
 advanced control loop between the batch deadline estimation and
 the regulator. If we have several ongoing high priority batches
 for which we made a too optimistic estimation of their
 deadline. Then all of them should see their deadline set to a
 minimum value that selects only the 10% fastest machines. As
 these batch do not complete and require replicas only on the
 10% fastest machines, the regulator will see the UNSENT job set
 filled with urgent jobs, which means the deadlines are too
 tight and need to be re-extended. So the regulator may actually
 be the right person to take care of batch deadline management.

****** Third step: populate the shared memory segment (feeder+regulator) ******* Goal When a host requests work, it will pick from the feeding array (shared memory segment) the jobs with higher priority that it can finish by the deadline.

It is thus important that *the feeding array contains enough
diversity* for the job/volunteer matchmaking to work: filling it in
order of priority may result in "slow" machines not getting any
work, because all jobs in the feeding array would have too tight
deadlines.

******* How to do it To do this, one way is to analyze the "expected performance" of machines (ie their speed modified by their average availability), divide it in quantiles, and make sure that there is a similar amount of work for each of these quantiles. Or (and it may be easier) respect minimal amounts for each quantile, and fill the rest with high priority jobs (e.g, replicated jobs that need to be resent because of an error on a client). This is a generalization of the "size matching" mechanism already in place in the regulator, and could be implemented in (an alternate version of) the regulator as well.

   We also need to ensure that there is some jobs available for each
   user/project, because some volunteers accept only some limited
   subset of users. This would be the job of the regulator, based on
   the current set of "UNSENT" jobs.

   In this case, probably the best implementation for the feeder would
   be to pick at random from the "UNSENT" jobs. This way, unless,
   there is an incredibly high diversity, and a large difference
   between the size of the UNSENT job set and the size of the shared
   memory array.

****** Fourth step: job/volunteer matchmaking ******* Goal Here, we want to enforce the priority values computed in the first step. ******* How to do it We scan the array and select jobs such that 1. expected completion on this particular volunteer is smaller than batch deadline 2. It fits the size constraints (i.e., not too large and not too small) 3. It respects the volunteer project preferences. 4. Make sure that the deadline the task is going to get will not create a too important slack (we may want to use volunteer provided information on how often they reconnect as an estimation of what cannot be accepted) as it may make some volunteers unhappy. Ideally, there would be a volunteer provided value regarding minimum slack.

   Among these jobs, we select the ones with the highest batch
   priority until we get the desired amount of work and jobs are
   still feasible.

   Issues:

  • Strictly smaller than batch deadline may create starvation of the batch. So we need a mechanism to re-extend batch deadlines but this was discussed earlier. ****** Fifth step: assign job deadlines Left open, we do not know yet how to do this.

     A lot of discussions but here is what we finally proposed:

 Let BD be the batch deadline. Let T_90 be the time a job of the
 batch would take on the 90% percentile machine. Say we aim to
 at most 3 series of resubmission.  Then we initially set the
 job deadlines to BD-3T_90. Whenever this deadline passed, we
 enter the resubmission mechanism and we only replicate with a
 tight deadline of T_90. This resubmission means you create an
 additional replica of the job because the deadline was passed.

****** Sixth step: trigger replication Instead of having only resubmission, we may want to have speculative job replication. This means that when deadline BD-3T_90 is passed, we resubmit not just one missed job, but several copies to decrease the failure probability and hence the potential re-resubmission. ****** Possible Client Mechanism Evolution

  • add a job state that says "report asap"

  • add a mechanism that says "run immediately" ?

  • allow volunteers to minimum slack ***** Other Concerns ****** Too tight job/batch deadlines ? This requires tuning I guess ****** How to do aggressive replication for the straggler jobs ? It depends on what we mean by aggressive. We can either use automatic resubmission or speculative replication or a combination. ****** Does there exist "processor affinity" ? I mean jobs which are really more efficient on GPU, and jobs more efficient on CPU ? Answer is yes but we may want to consider this kind of optimization. ****** Turnaround time estimation vs. Throughput estimation We're currently effective turnaround time and not potential turnaround time. In WCG, some very fast workers have a huge cash and take a lot of time to complete jobs, hence a poor effective turnaround time whereas they have a huge throughput. Actually, we do not keep track of this throughput but we keep track of the credit history, which may be a more accurate measure as job duration from a batch to another may vary whereas credit is bound the amount of work to be done. ****** Why do we have a notion of batch LET and batch deadline? The LET notion is used to ensure there is a "fair" sharing of resource. The batch deadline is here to exclude slow volunteers. So although the two notions have quite different usages, they are very related. Kevin provided a good reason for having two separate notions computed in a different way. If there is ever a batch B1 with a large number of small jobs and a batch B2 with a small number of large jobs, they will both incur the same resource usage. However the time it will actually take to complete will be quite different.

     Note that this means that if we ever compute batch deadline
 using Kevin 3 option, B1 will get rather tight deadline. It is
 still not clear to me whether it is a good or a bad
 thing. Somehow volunteers expect reasonable slacks. One option
 would be to enforce a minimum slack for volunteers so we added
 this constraint in the matchmaking.

****** Can we "predict" availability based on current uptime ? This is an interesting question but this seems very difficult. Maybe I could discuss with Jean-Marc and he would convince me it's just unfeasible. :) ****** Issues raised by Uwe On a project where the number of batches/jobs that are ready to run is very large relative to the size of the infrastructure, preloading all workunits into BOINC can cause a significant degredation of performance. There may need to be a way to load a batch so that it can be priortized and planned, but that the workunit/result records are not created and files are not copied to download until the system knows that those are required. A mechanism would need to exist to assess this condition and trigger that creation.

  • Application specific search space division Wenjie, Gerdus

    More or less the same scheme can be applied, maybe with some problem-specific tweaking, but not much. No BOINC server side change is needed, a few simple scripts can already implement the mechanism quite well. Maybe some kind of priority system can be implemented in the recycling. Splitting leftovers is also a way to deal with the long workunits that lasts, but maybe application-specific. Important to keep things simple, even at the cost of accuracy! This is an engineering problem.

  • TODO Integration with hubs, clouds, grids, and desktop grids Joszef and Adam talked about it but we don't know much. We should ask them.

  • Make project web sites translatable We should be careful about words with context-sensitive meaning. Having the context would be useful when translating. Maybe, SETI@home could be a good test-case.

  • Creating and deploying VM-based app versions Carlos, Christian, Uwe, Carlos Val. Wenjing, Francisco, dario The smallest VM size Christian achieved was ~680 MB (uncompressed) and 180 MB (gzip compressed). Getting lower would require specific kernels, and then there is a increased risk of bugs It would be great if we could get it under the 100MB range. In Sztaki, they simply compress the image and it shrinks from 700 to 100MB, which is sufficient.

    Christian explained Dario and Carlos how to create VM-based apps and how he deals with this in RNA World VmImage: http://www.rnaworld.de/rnaworld/download/rnaWorld2GB.vdi.gz Extract the VDI to your hard disk, create a new Virtual Machine using this VDI and create a shared folder called 'shared' and place an executable/script named 'boinc_app' inside. This will be executed by the startup script of the VM. You can do for example: #!/bin/bash sleep 10 echo "Hello World" > ../shared/hello.txt sleep 10 You can also interrupt the startup script by Ctrl-C in the VM and explore the inside.

    We identified the problem of getting serious errors from inside the VM in a generic way. One way would be to redirect the console output via a serial port to the vboxwrapper and look for kernel panic or similiar messages. One approach is the socat pipe (http://wiki.illumos.org/display/illumos/Serial+Console+in+VirtualBox).

    To solves these problems, Christian needs new features in the virtualbox wrapper, which could be based on socat.

    Nothing much was done on the small VM size side.

    Right now, although we have an installer for both BOINC and Vbox, the BOINC page mainly advertize for BOINC only as we don't want people to install useless things. David would like to change this to a big download BOINC+Vbox button and a small download BOINC explaining that it may restrict project participation. Christian raises the fact that to run 64bits apps in Vbox, you need the VTX feature to be activated in the bios, and there is no way to check whether it is activated or not without trying to execute workunits. Ideally in such a case, the client should post a notification to the volunteer to explain him what he should do to improve things.

    An alternative suggestion was to enable BOINC to initiate the download and installation of VirtualBox at the time the user attempts to connect to a project that requires VirtualBox

  • Php pages with twitter bootstrap Francisco, Dario, David, Christian

    Only works with bootstrap 2.3.2 and can be integrated into default BOINC with some effort. I also think it's possible to make it optional to have an easy upgrade path for projects. It will be soon put into BOINC.

  • How to automate the end-to-end testing of BOINC? Dario, Kevin, Joachim, Augustin, Adam

  • automate the database deployment, compiling, submission of a client

    Nothing was done here.

  • Remote job submission : unification Wenjin, Christian, David

    API-doc: RemoteJobs It would be nice to have a disk quota for each submitter to limit disk usage on the server A hook on the server side to allow application specific preparation of batches. Possible implementation: place a file appname.inc in html/project.inc and submit_rpc_handler.php will look for this and call a specific function (prepare_job or prepare_batch) that is returning an XML structure that get's passed back to the submitter. Attention, this preparation could take some time! Another hook should make it possible to prepare the output data of a batch. A project might want to zip output files for each submitter on a daily basis before the whole batch is finished so the submitter can download these partial files.

    Brainstormed on how name files should be managed. There is a beginning of a plan but it's not quite there yet. Maybe it will become more concrete next week.

  • Drupal/BOINC tutorial Tristan talked with whoever was interested (ClimatePrediction, LHC, CAS@home). He's going to write some documentations explaining how to do this, which should make it easy for everyone.

  • Abandonned tasks ** Sub-second CPU throttling. ** Further simplifying the BOINC install process and GUI ** Prototype a BOINC GUI using HTML5 ** Improving the BOINC server documentation ** GPU, multi-thread, VM-based, and Android applications ** Data-intensive applications

Home

Clone this wiki locally