waymo dataset compilation - bug and tip

[maxiuw]

It's kind of a bug, kind of a tip for the people who want to work with Waymo data. Installing waymo datasets toolkit (the only available version is v11 or smt lik this) crashes all the other dependencies. I guess it is hard to make sure that all the dependencies are in place since it is really only needed for building pkl dataset and eventually for evaluation (but you can also eval with kitti eval type and you don't really need it).

How I dealt with that:
2 separate docker images (or conda env): one for building the waymo dataset only and the other one for running (you dont need waymo toolking for training).

My recommendation is to build the docker image.

1. Build the image (script below)
2. run dokcer using docker run -it -v "/mnt/yourworkdir:/mnt/yourworkdir" yourdockerimag
3. runs setup.py and install other dependencies that were missing, docker commit and save as separte image for trainig the models as yourdockerimag:trainig
4. run dokcer using docker run -it -v "/mnt/:/mnt" yourdockerimag install waymo tool kit, dont worry about all the dependencies errors, you only need it for waymo compilation.
5. docker comit yourdockerimag:waymo.

I wish i found a better way but it gave me so many headaches to work with that i decided to share it :) Docker provided in the repo didn't work for me. This docker that uses torch 2.1 works well :)

FROM nvidia/cuda:11.8.0-cudnn8-devel-ubuntu20.04


ENV PYTHONUNBUFFERED=1 
ENV FORCE_CUDA="1"
ENV CMAKE_PREFIX_PATH="$(dirname $(which conda))/../"
ENV TORCH_NVCC_FLAGS="-Xfatbin -compress-all"

# SYSTEM
RUN apt-get update --yes --quiet && DEBIAN_FRONTEND=noninteractive apt-get install --yes --quiet --no-install-recommends \
    software-properties-common \
    build-essential apt-utils \
    wget curl vim git ca-certificates kmod \
 && rm -rf /var/lib/apt/lists/*
RUN apt-get update 
RUN apt-get install ffmpeg libsm6 libxext6  -y
# PYTHON 3.10
RUN add-apt-repository --yes ppa:deadsnakes/ppa 
RUN add-apt-repository -y ppa:ubuntu-toolchain-r/test
RUN apt install -y g++-11
RUN apt install -y gcc-11
RUN apt install -y libsparsehash-dev
RUN apt-get update --yes --quiet
RUN DEBIAN_FRONTEND=noninteractive apt-get install --yes --quiet --no-install-recommends \
    python3.10 \
    python3.10-dev \
    python3.10-distutils \
    python3.10-lib2to3 \
    python3.10-gdbm \
    python3.10-tk \
    pip

RUN update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.10 999 \
    && update-alternatives --config python3 && ln -s /usr/bin/python3 /usr/bin/python

RUN curl -sS https://bootstrap.pypa.io/get-pip.py | python3.10



RUN pip3 install torch==2.1.0 torchvision==0.16.0 --index-url https://download.pytorch.org/whl/cu118



RUN pip3 install opencv-python==4.7.0.68
RUN pip3 install llvmlite numba tensorboardX easydict pyyaml scikit-image tqdm SharedArray open3d mayavi av2 kornia==0.6.5 pyquaternion
RUN pip3 install spconv-cu118

WORKDIR /usr/local/
COPY . PTv3_Det
WORKDIR /usr/local/PTv3_Det
RUN pip3 install -r requirements.txt

# next, run docker and
# python3 setup.py develop

[github-actions]

This issue is stale because it has been open for 30 days with no activity.