call to mapnik.register_default_input_plugins never returns

[zdila]

Call mapnik.register_default_input_plugins() never returns.

• mapnik module version: 4.2.0 (same problem with 4.0.2)
• node version: 10.14.2
• OS: alpine linux, version edge, running in docker

Last lines from strace output:

mprotect(0x25f844204000, 503808, PROT_READ|PROT_WRITE) = 0
mprotect(0x25f844204000, 503808, PROT_READ|PROT_EXEC) = 0
stat("/freemap-mapnik/node_modules/mapnik/lib/binding/lib/mapnik/input", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
stat("/freemap-mapnik/node_modules/mapnik/lib/binding/lib/mapnik/input", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
open("/freemap-mapnik/node_modules/mapnik/lib/binding/lib/mapnik/input", O_RDONLY|O_CLOEXEC|O_DIRECTORY) = 20
fcntl(20, F_SETFD, FD_CLOEXEC)          = 0
getdents64(20, /* 13 entries */, 2048)  = 432
getdents64(20, /* 0 entries */, 2048)   = 0
close(20)                               = 0
stat("/freemap-mapnik/node_modules/mapnik/lib/binding/lib/mapnik/input/csv.input", {st_mode=S_IFREG|0755, st_size=1362981, ...}) = 0
stat("/freemap-mapnik/node_modules/mapnik/lib/binding/lib/mapnik/input/csv.input", {st_mode=S_IFREG|0755, st_size=1362981, ...}) = 0
futex(0x7f8f8821b0cc, FUTEX_WAIT_PRIVATE, 2147483664, NULL

Removing /freemap-mapnik/node_modules/mapnik/lib/binding/lib/mapnik/input/csv.input makes it halt on another *.input file. If I remove all files from /freemap-mapnik/node_modules/mapnik/lib/binding/lib/mapnik/input/ then the call returns but obviously it fails later as necessary drivers are missing.

[zdila]

Note that I had no such issue with debian linux running in docker.

[zdila]

I've checked all threads of node process. All are waiting on futex/FUTEX_WAIT_PRIVATE but one was epoll_pwait(13, .

[zdila]

Some output from gdb:

(gdb) bt
#0  0x00007f8bfa29cfd1 in ?? () from /lib/ld-musl-x86_64.so.1
#1  0x00007fff49699508 in ?? ()
#2  0x00007f8bf6cd70cc in mapnik::CreateStatic<mapnik::datasource_cache>::create()::static_memory () from /freemap-mapnik/node_modules/mapnik/lib/binding/lib/libmapnik.so
#3  0xffffffff80000010 in ?? ()
#4  0x00007f8bfa29c561 in __timedwait_cp () from /lib/ld-musl-x86_64.so.1
#5  0x0000000000000000 in ?? ()

[talaj]

datasource_cache is the only place in Mapnik where std::recursive_mutex is used: src/datasource_cache.cpp#L163. I'm wondering whether this cannot be somehow connected to your problem. Maybe the mutex actually created is not recursive one.

but one was epoll_pwait(13,

You can take a look what is behind the file descriptor 13 by ls -l /proc/6482/fd with pid of your app.

[zdila]

It is anon_inode:[eventpoll].

[zdila]

I can prepare a testcase if it would help.

[talaj]

Are you building Mapnik directly in Docker? In that case it would be great if you can share the dockerfile.

[zdila]

Dockerfile:

RUN apk add --update nodejs npm gcompat
RUN mkdir /test
WORKDIR /test
RUN npm init -y
RUN npm i --save mapnik
RUN echo "require('mapnik').register_default_input_plugins();" > index.js

Steps:

1. docker build -t test .
2. docker run -it -w /test test node .

docker run should terminate but it doesn't.

[talaj]

Thanks, I can reproduce it. Will try to take a look deeper later today.

[talaj]

I still don't know what exactly causes that deadlock, but Mapnik package from Alpine works - no surprise - well:

My test app:

#include <mapnik/datasource_cache.hpp>
#include <iostream>
#undef NDEBUG
#include <cassert>

int main()
{
    auto & cache = mapnik::datasource_cache::instance();

    assert(cache.register_datasources("/usr/lib/mapnik/input/", true));

    for (auto const & name : cache.plugin_names())
    {
        std::clog << name << std::endl;
    }

    return 0;
}

Output:

$ ./test
csv
gdal
geojson
ogr
pgraster
postgis
raster
shape
sqlite
topojson

I would maybe just not use the prebuilt Mapnik from npm, rather build it from latest sources inside Docker. It is also way how to deploy patches quickly. To build Mapnik itself is simple, I would like to try also node-mapnik:

FROM alpine:latest

RUN echo "http://repository.fit.cvut.cz/mirrors/alpine/edge/main" > /etc/apk/repositories; \
    echo "http://repository.fit.cvut.cz/mirrors/alpine/edge/community" >> /etc/apk/repositories; \
    echo "http://repository.fit.cvut.cz/mirrors/alpine/edge/testing" >> /etc/apk/repositories

RUN apk update && apk upgrade

RUN apk add bash make git python gdal-dev cairo-dev \
    postgresql-dev harfbuzz-dev g++ boost-dev sqlite-dev \
    proj4-dev jpeg-dev tiff-dev libwebp-dev

RUN mkdir /build

WORKDIR /build

RUN git clone https://github.com/mapnik/mapnik.git

WORKDIR mapnik

RUN git checkout v3.0.x
RUN git submodule update --init

RUN ./configure
RUN JOBS=2 make
RUN make install

[talaj]

Building node-mapnik:

FROM mapy-sk-mapnik-build:latest

RUN apk add nodejs npm

WORKDIR /build

RUN git clone https://github.com/mapnik/node-mapnik.git

WORKDIR node-mapnik

RUN git checkout v3.0.x
RUN git submodule update --init

RUN make release_base

Test image:

FROM mapy-sk-mapnik-build-node:latest

RUN mkdir /test
WORKDIR /test
COPY package.json .
RUN npm i
RUN echo "var mapnik = require('mapnik'); mapnik.register_default_input_plugins();console.log(mapnik.datasources());" > index.js

package.json:

{                                                                                                                                                                                                                  
  "name": "test",
  "version": "1.0.0",
  "description": "",
  "dependencies": {
    "mapnik": "file:/build/node-mapnik"
  },
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "keywords": [],
  "author": "",
  "license": "ISC"
}

Output:

$ docker run --rm -i -t mapy-sk-mapnik-build-node-test:latest node .
[ 'csv',
  'gdal',
  'geojson',
  'ogr',
  'pgraster',
  'postgis',
  'raster',
  'shape',
  'sqlite',
  'topojson' ]

[zdila]

Great, thanks!

Anyway we already use debian-based containers but we'll definitively use your instructions if we decide to switch to alpine :-)

Not sure whether I should close this issue now. I'll leave it up on you.