Skip to content

Docker app cookbook

Jump to bottom
David Anderson edited this page Nov 14, 2024 · 5 revisions

This cookbook shows how to create and deploy Docker apps in BOINC. We assume you already have created a BOINC project.

For background, please read about how Docker apps work. This explains the two ways of packaging Docker apps:

  • Single-purpose app: there is one BOINC app per science app. You need to create new BOINC app versions each time your science app changes. This means your science app developers must have login access to the BOINC server, and must learn BOINC's command-line tools.

  • Universal app: there is a single BOINC app (and a set of per-platform app versions) that handles all science apps. Dockerfiles and science executables are included with each job. This provides flexibility. For example, it lets you create web-based job submission systems in which scientists use (and evolve) Docker apps without logging into the server (or indeed knowing anything about BOINC).

This page has cookbooks for both packaging methods. In both cases we're going to make a Docker app for Windows; the Linux and Mac OS cases are similar.

Docker app jobs are sent only to BOINC 9.0+ clients that have Docker. On Windows this requires that

  • WSL is enabled
  • The host has the 'BOINC WSL distro' installed from the Windows Store (or has another WSL distro with either Docker or Podman installed).

Both variants are going to use the same "science executable" - an x64/Intel program that runs inside a Docker container. This program is called worker; it converts a file to upper case and optionally uses some CPU time.

worker --nsecs 60 infile outfile

Get "worker"

You can download it:

wget https://boinc.berkeley.edu/worker_3_x86_64-pc-linux-gnu

or build it in the BOINC source tree:

cd samples/worker
make

and rename it to worker_3_x86_64-pc-linux-gnu. (The '3' is a version number; in general all files should have version numbers in case they change).

Get docker_wrapper

Download it:

wget https://boinc.berkeley.edu/docker_wrapper_1__windows_x86_64.exe

Other files

Make a file Dockerfile_worker_1:

FROM debian
WORKDIR /app
CMD ./main.sh

Make a file job_1.toml:

project_dir_mount = "/project"

Make a file main_2.sh:

#! /bin/bash

resolve () {
    sed 's/<soft_link>..\/..\/projects\/[^\/]*\//\/project\//; s/<\/soft_link>//' $1 | tr -d '\r\n'
}

$(resolve worker) --nsecs 60 $(resolve in) $(resolve out)

Single-purpose app

Create the app

  • Go to your project's admin web interface
  • Click on Manage applications
  • Add an application with name 'worker' and description 'Test app'.

Create input and output templates

In ~/projects/<proj_name>/templates/, create a file worker_in:

<input_template>
    <file_info>
    </file_info>
    <workunit>
        <file_ref>
            <open_name>in</open_name>
        </file_ref>
    </workunit>
</input_template>

and worker_out:

<output_template>
    <file_info>
        <name><OUTFILE_0/></name>
        <generated_locally/>
        <upload_when_present/>
        <max_nbytes>5000000</max_nbytes>
        <url><UPLOAD_URL/></url>
    </file_info>
    <result>
        <file_ref>
            <file_name><OUTFILE_0/></file_name>
            <open_name>out</open_name>
        </file_ref>
    </result>
</output_template>

Create an app version for Windows

Using the above files, create the following directory structure:

~/projects/<proj_name>/apps/
    worker/
        1.0/
            windows_x86_64__docker/
                Dockerfile_worker_1
                docker_wrapper_1.exe
                job_1.toml
                main_2.sh
                worker_3_x86_64-pc-linux-gnu
                version.xml

where version.xml contains

<version>
    <file>
        <physical_name>docker_wrapper_1.exe</physical_name>
        <main_program/>
    </file>
    <file>
        <physical_name>worker_3_x86_64-pc-linux-gnu</physical_name>
        <logical_name>worker</logical_name>
    </file>
    <file>
        <physical_name>main_2.sh</physical_name>
        <logical_name>main.sh</logical_name>
        <copy_file/>
    </file>
    <file>
        <physical_name>Dockerfile_worker_1</physical_name>
        <logical_name>Dockerfile</logical_name>
        <copy_file/>
    </file>
    <file>
        <physical_name>job_1.toml</physical_name>
        <logical_name>job.toml</logical_name>
        <copy_file/>
    </file>
    <is_wrapper/>
</version>

In your project's main directory:

bin/update_version --noconfirm

Set up a validator and assimilator

Edit your project's config.xml and add the following to the <daemons> section:

<daemon>
  <cmd>script_validator --app worker --init_script "validate_init.py" --compare_script "validate_compare.py"</cmd>
</daemon>
<daemon>
  <cmd>script_assimilator -d 3 --app worker --script "sample_assimilate.py wu_name batch_id files"</cmd>
</daemon>

Test

In your project's main directory:

bin/stop
bin/start

Create a file infile with some mixed-case text. Run

bin/submit_job worker infile

It will print a job name.

Go to a Windows computer that has Docker enabled (see above) and has a BOINC 9.0+ client running and attached to your project. Update the project. It should fetch the job you just created, run it (takes about 1 minute) and upload the output file.

When the job is done, run

bin/query_job <job_name>

This should show that the job is completed, and display its output file.

Universal app

Create the app

  • Go to your project's admin web interface
  • Click on Manage applications
  • Add an application with name 'buda' and description 'BOINC universal Docker app'.

Set up a validator and assimilator

Edit your project's config.xml and add the following to the <daemons> section:

<daemon>
  <cmd>script_validator --app buda --init_script "validate_init.py" --compare_script "validate_compare.py"</cmd>
</daemon>
<daemon>
  <cmd>script_assimilator -d 3 --app buda --script "sample_assimilate.py wu_name batch_id files"</cmd>
</daemon>

Create an app version for Windows

Using the above files, create the following directory structure:

projects/<proj_name>/apps/
    buda/
        1.0/
            windows_x86_64__docker/
                docker_wrapper_1.exe
                version.xml

where version.xml contains

<version>
    <file>
        <physical_name>docker_wrapper_1.exe</physical_name>
        <main_program/>
    </file>
    <is_wrapper/>
</version>

Run

bin/update_verions --noconfirm

Create a BUDA directory

'BUDA' stands for BOINC Universal Docker App. This app can handle jobs for multiple science apps. The files for these apps are store in a directory hierarcy with a particular structure.

In your project directory, create a directory tree

buda_apps/
    worker/
        cpu/

Copy these files to the cpu/ directory:

Dockerfile_worker_1
file_list
job_1.toml
main_2.sh
template_in
template_out
worker_3_x86_64-pc-linux-gnu

where file_list contains

Dockerfile_worker_1
job_1.toml
main_2.sh
worker_3_x86_64-pc-linux-gnu

and template_in contains

<input_template>
    <file_info>
    </file_info>
    <file_info>
    </file_info>
    <file_info>
    </file_info>
    <file_info>
    </file_info>
    <file_info>
    </file_info>
    <workunit>
        <file_ref>
            <open_name>Dockerfile</open_name>
            <copy_file/>
        </file_ref>
        <file_ref>
            <open_name>job.toml</open_name>
            <copy_file/>
        </file_ref>
        <file_ref>
            <open_name>main.sh</open_name>
            <copy_file/>
        </file_ref>
        <file_ref>
            <open_name>worker</open_name>
        </file_ref>
        <file_ref>
            <open_name>in</open_name>
        </file_ref>
    </workunit>
</input_template>

and template_out contains

<output_template>
    <file_info>
        <name><OUTFILE_0/></name>
        <generated_locally/>
        <upload_when_present/>
        <max_nbytes>5000000</max_nbytes>
        <url><UPLOAD_URL/></url>
    </file_info>
    <result>
        <file_ref>
            <file_name><OUTFILE_0/></file_name>
            <open_name>out</open_name>
        </file_ref>
    </result>
</output_template>

Test

Start the project:

bin/start

and run

bin/submit_buda worker infile

Update the project on your Windows BOINC client so that it fetches and runs the job. Then enter

bin/query_job <job_name>

to check the results.

Home

Clone this wiki locally