Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Could you help me with adding multiple models to keras-serving #2

Open
jerminal opened this issue Jun 1, 2017 · 35 comments
Open

Could you help me with adding multiple models to keras-serving #2

jerminal opened this issue Jun 1, 2017 · 35 comments
Assignees
Labels

Comments

@jerminal
Copy link

jerminal commented Jun 1, 2017

Could you help me with adding multiple models for keras-serving ? Just reply we can discuss it via email or skype.

@krystianity
Copy link
Owner

Hi jerminal, I would like to help you on this. Are you talking about changing the model's feature vectors (input) and breaking the dimension apis in GRPC nodejs requests?

I will need a description of the input vector of your network to help you.

@krystianity krystianity self-assigned this Jun 1, 2017
@krystianity
Copy link
Owner

Or do you just want to add multiple versions of a model to the same serving-server?

@jerminal
Copy link
Author

jerminal commented Jun 2, 2017

I have both problems. I need to classify images (so to change the input vector). And I have multiple keras models that I want to apply to each image. So maybe you can add me to skype: jermuk_skype and I'll describe it with more details?

@jerminal
Copy link
Author

jerminal commented Jun 4, 2017

Or maybe you can tell me your skype or email?

@jerminal
Copy link
Author

jerminal commented Jun 7, 2017

For example to server models for face recognition.
Some examples from here: https://github.com/oarriaga/face_classification

@krystianity
Copy link
Owner

Hi @jerminal I will try to setup an example using the face_classification soon.
I am sorry that I cannot give you support through Skype or Email.

Serving multiple versions of a model at once should be no problem, you should only have to name the exports differently and send the version as paramter to the grpc service during requests.

@jerminal
Copy link
Author

jerminal commented Jun 7, 2017

Oh. Great. But is it possible to check 1 image for several models at 1 request to docker server?

And about support - ok, no problem. You can add me to twitter or anything else. And I can pay for such improvements of your model if you'd like.

@krystianity
Copy link
Owner

I would probably suggest to load multiple versions of the model into the serving-server and then use the nodejs service to make parallel requests to both/multiple models at the same time, its very easy to do this in node. Il try to setup the face-recog. example with such a parallel request endpoint.

Haha cheers! Il pm you on twitter, when I am working on it ;)

@krystianity
Copy link
Owner

Actually the question here is, are you actually using the node.js service simplified API? If yes (thats cool) how many requests are u going to run against it? (You could scale the services to unlimited instances,, but its still a valid question if you are trying to predict in realtime using web-services)

@jerminal
Copy link
Author

jerminal commented Jun 7, 2017

I have about 10-15 requests per minute. It's not very much :)

@jerminal
Copy link
Author

jerminal commented Jun 8, 2017

And yes - I use the node.js service simplified API.

@krystianity
Copy link
Owner

Cool, just hang on ;)

@jerminal
Copy link
Author

jerminal commented Jun 8, 2017

👍

@krystianity
Copy link
Owner

So I have already pushed my current state into the remote master, but I am not done yet.
I still have to finalise the grpc node communications so that both models are used to predict on behalf of an image that is uploaded to the node server via client. I think I can finish this until mid next week :)

@krystianity
Copy link
Owner

I have created a sub project as directory: https://github.com/krystianity/keras-serving/tree/master/face-recog

@jerminal
Copy link
Author

Thanks. Will be waiting for this very much!!!

@krystianity
Copy link
Owner

@jerminal alright. It is up an running :)
Enjoy the awesomeness 🍸 and drop me beer if you like https://www.paypal.me/5cf 🍺

Let me know if you need or want to try something else.

@jerminal
Copy link
Author

Hi. Thanks a lot!!! I'll check it in the evening. And of course I've sent drop for a beer for you.

And promise when I come to Germany again, I'll to treat you beer personally :)

@jerminal
Copy link
Author

Oh.. I have a strange problem. When follow the instructions:
https://github.com/krystianity/keras-serving/tree/master/face-recog#4-test-api-via-client

at p. 4 when I run: "npm start ./../images/cutouts/4.png" I got strange error :(

10 verbose lifecycle [email protected]~start: CWD: /Users/siarheiparashchanka/sites/keras-serving/face-recog/client
11 silly lifecycle [email protected]~start: Args: [ '-c', 'node request.js "./../images/cutouts/4.png"' ]
12 silly lifecycle [email protected]~start: Returned: code: 1  signal: null
13 info lifecycle [email protected]~start: Failed to exec start script
14 verbose stack Error: [email protected] start: `node request.js "./../images/cutouts/4.png"`
14 verbose stack Exit status 1
14 verbose stack     at EventEmitter.<anonymous> (/usr/local/lib/node_modules/npm/lib/utils/lifecycle.js:283:16)
14 verbose stack     at emitTwo (events.js:87:13)
14 verbose stack     at EventEmitter.emit (events.js:172:7)
14 verbose stack     at ChildProcess.<anonymous> (/usr/local/lib/node_modules/npm/lib/utils/spawn.js:40:14)
14 verbose stack     at emitTwo (events.js:87:13)
14 verbose stack     at ChildProcess.emit (events.js:172:7)
14 verbose stack     at maybeClose (internal/child_process.js:818:16)
14 verbose stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:211:5)
15 verbose pkgid [email protected]
16 verbose cwd /Users/siarheiparashchanka/sites/keras-serving/face-recog/client
17 verbose Darwin 16.6.0
18 verbose argv "/usr/local/bin/node" "/usr/local/bin/npm" "start" "./../images/cutouts/4.png"
19 verbose node v5.1.0
20 verbose npm  v5.0.3
21 error code ELIFECYCLE
22 error errno 1
23 error [email protected] start: `node request.js "./../images/cutouts/4.png"`
23 error Exit status 1
24 error Failed at the [email protected] start script.
24 error This is probably not a problem with npm. There is likely additional logging output above.
25 verbose exit [ 1, true ]

@krystianity
Copy link
Owner

No worries, your running node 5.10; (with npm 5.0.3 o.O) Can you upgrade you node version to at least 7.10 ?

@krystianity
Copy link
Owner

If its not possible for you to update node on the root system, I can wrap the client in a Docker container as well :)

@jerminal
Copy link
Author

Updated node. Now strange thing with running npm start ./../images/cutouts/4.png
I got response:


> [email protected] start /Users/siarheiparashchanka/sites/keras-serving/face-recog/client
> node request.js "./../images/cutouts/4.png"

Reading from path: /Users/siarheiparashchanka/sites/keras-serving/face-recog/images/cutouts/4.png
Loaded image with size: 2304
Making request to CNN GRPC wrapper.
{
  "error": {
    "code": 3,
    "metadata": {
      "_internal_repr": {}
    }
  }
}

@krystianity
Copy link
Owner

After upgrading nodejs, did you delete the node_modules folder in the face-recog/client dir and ran "npm install again" ? (just to make sure)

Additionally, I have a feeling that the models you servers are serving arent correct e.g. cached.
How are you starting the servers? When you are using my script (start-dual-servers.sh) it should make sure that nothing is cached.

To understand whats going wrong I need you to send me the error that is logged by the server, when your client retrieves the error.

@krystianity
Copy link
Owner

Additionally.
Are you executing via python 2.7?
Which docker and docker-compose versions are you running?
Could you dump the whole output of docker-compose when you start the dual servers and make a request with the client?

@jerminal
Copy link
Author

Oh. I'll check it now on different computer. A few hours and will write back.

@jerminal
Copy link
Author

jerminal commented Jun 16, 2017

I've checked. The same problem.
I recompiled all.
And after I run "npm start ./../images/cutouts/4.png"

Got the same error.
Looks like problem with input tensor size:

node-server_1     | npm info it worked if it ends with ok
node-server_1     | npm info using [email protected]
node-server_1     | npm info using [email protected]
node-server_1     | npm info lifecycle [email protected]~prestart: [email protected]
node-server_1     | npm info lifecycle [email protected]~start: [email protected]
node-server_1     | 
node-server_1     | > [email protected] start /usr/src/app
node-server_1     | > node server.js
node-server_1     | 
emotion-server_1  | 2017-06-16 13:55:14.421089: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:145] Restoring SavedModel bundle.
gender-server_1   | 2017-06-16 13:55:14.691086: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:145] Restoring SavedModel bundle.
node-server_1     | listening @ http://localhost:8080
emotion-server_1  | 2017-06-16 13:55:15.425129: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:180] Running LegacyInitOp on SavedModel bundle.
emotion-server_1  | 2017-06-16 13:55:15.459184: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:274] Loading SavedModel: success. Took 2047432 microseconds.
emotion-server_1  | 2017-06-16 13:55:15.460264: I tensorflow_serving/core/loader_harness.cc:86] Successfully loaded servable version {name: emotion_model version: 1}
emotion-server_1  | 2017-06-16 13:55:15.471674: I tensorflow_serving/model_servers/main.cc:298] Running ModelServer at 0.0.0.0:9000 ...
gender-server_1   | 2017-06-16 13:55:15.608845: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:180] Running LegacyInitOp on SavedModel bundle.
gender-server_1   | 2017-06-16 13:55:15.648949: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:274] Loading SavedModel: success. Took 2080879 microseconds.
gender-server_1   | 2017-06-16 13:55:15.650862: I tensorflow_serving/core/loader_harness.cc:86] Successfully loaded servable version {name: gender_model version: 1}
gender-server_1   | 2017-06-16 13:55:15.663076: I tensorflow_serving/model_servers/main.cc:298] Running ModelServer at 0.0.0.0:9000 ...
node-server_1     | { Error: input must be 4-dimensional[]
node-server_1     | 	 [[Node: image_array_2/convolution = Conv2D[T=DT_FLOAT, _output_shapes=[[-1,48,48,16]], data_format="NHWC", padding="SAME", strides=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_image_array_input_2_0_0, image_array_2/kernel/read)]]
node-server_1     |     at /usr/src/app/node_modules/grpc/src/node/src/client.js:442:17 code: 3, metadata: Metadata { _internal_repr: {} } }

@jerminal
Copy link
Author

jerminal commented Jun 16, 2017

And yes, I execute via python 2.7.

Which docker and docker-compose versions are you running?

Docker version 17.04.0-ce, build 4845c56
docker-compose version 1.12.0, build b31ff33

Could you dump the whole output of docker-compose when you start the dual servers and make a request with the client?

Yes, I've done it above. Hope it will help you.

@krystianity
Copy link
Owner

Alright,

the versions look fine, the output is strange as it relates to the dimensions defined for the interface.
Of which both are 4 dimensional for predictEmotion(1,48,48,1) and predictGender(1,48,48,3).

I have added additional logging to the node-server and the client, could please run the following and
send me the output of the docker-compose, as well as the client here again:

cd /keras-prod
git pull
rm -r ./export
python face-recog/export.py
./start-dual-servers.sh
#..
cd face-recog/client
npm start ./../images/cutouts/4.png

@krystianity
Copy link
Owner

krystianity commented Jun 17, 2017

image

This is how it should actually look like.

@jerminal
Copy link
Author

Here log:

node-server_1     | npm info it worked if it ends with ok
node-server_1     | npm info using [email protected]
node-server_1     | npm info using [email protected]
node-server_1     | npm info lifecycle [email protected]~prestart: [email protected]
node-server_1     | npm info lifecycle [email protected]~start: [email protected]
node-server_1     | 
node-server_1     | > [email protected] start /usr/src/app
node-server_1     | > node server.js
node-server_1     | 
node-server_1     | listening @ http://localhost:8080
node-server_1     | Input array size: 2304, example value: 0.9882352941176471.
node-server_1     | Tensor Dim: [{"size":1},{"size":48},{"size":48},{"size":3}].
node-server_1     | Input array size: 2304, example value: 0.9882352941176471.
node-server_1     | Tensor Dim: [{"size":1},{"size":48},{"size":48},{"size":1}].
node-server_1     | emotion prediction failed.
node-server_1     | { Error: input must be 4-dimensional[]
node-server_1     | 	 [[Node: image_array_2/convolution = Conv2D[T=DT_FLOAT, _output_shapes=[[-1,48,48,16]], data_format="NHWC", padding="SAME", strides=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_image_array_input_2_0_0, image_array_2/kernel/read)]]
node-server_1     |     at /usr/src/app/node_modules/grpc/src/node/src/client.js:442:17 code: 3, metadata: Metadata { _internal_repr: {} } }
node-server_1     | gender prediction failed.

And request:

MBPJ-2:client siarheiparashchanka$ npm start ./../images/cutouts/4.png

> [email protected] start /Users/siarheiparashchanka/sites/keras-serving/face-recog/client
> node request.js "./../images/cutouts/4.png"

Reading from path: /Users/siarheiparashchanka/sites/keras-serving/face-recog/images/cutouts/4.png
image flat size: 48.
Loaded image with size: 2304
Making request to CNN GRPC wrapper.
POST to http://localhost:8080/predict-emotion-and-gender with 43601 body length.
{
  "error": {
    "code": 3,
    "metadata": {
      "_internal_repr": {}
    }
  }
}

@krystianity
Copy link
Owner

Strange :/

On which OS are you running this? MacOSX?
I will try to set this up on a clean VM tomorrow, I cannot reproduce this error using Ubuntu 16.04

@krystianity
Copy link
Owner

Okay so I have spun up a clean Ubuntu 16.04 LTS VM and ran through the setup as defined in the READMEs. I have found 2 potential errors regarding the naming of the input vectors of the gender model as well as the output vectors, its possibly related to some strange docker-compose caching behavior that I wasnt able to see these locally.

The bugs are fixed in the current master, so please go ahead and run:

cd /keras-prod
git checkout .
git pull
python ./face-recog/export.py
./start-dual-servers.sh
#..
cd face-recog/client
npm start ./../images/cutouts/4.png

again :)

@jerminal
Copy link
Author

Cool. Now works!

@krystianity
Copy link
Owner

Awesome :)

Let me know if you want to take this in another direction.

I am currently trying to:

  • setup a Google Cloud BigQuery large dataset batch-training example
  • reduce the size of the tf-model-server image, 3,4 GB is way too large
  • *?

@jerminal
Copy link
Author

Maybe some preprocessing images. For example, I send big ".jpg" image and it reshapes it and responses with the output of models.

And also maybe more details how to add your own models and make it work.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants