Different sizes of the same image give different results

[ghnp5]

Hey!

As per #253,

NSFWJS will automatically resize the input image to be square for you.
No code should be needed on your side.

So, I'm confused why providing different sizes (keeping the aspect ratio) of a JPG brings me totally different outputs.

On this specific image I'm testing with (6794873 for my own reference), if I provide a 1365 × 2048 version of the image, it gives me:

porn: 56.75
neutral: 28.55
hentai: 9.34
sexy: 4.47
drawing: 0.89

But if I provide a resized version of it (480 × 720), it gives me:

porn: 96.25
hentai: 1.84
neutral: 1.03
sexy: 0.83
drawing: 0.05

To me, the output provided by the larger image is much more accurate, than the second output.
If anything, I'd give a higher sexy score, as there isn't any porn in the image (unless the lady in the image is an actress or so, I've no idea).

But still, I'm not understanding why different sizes of the same image are giving such different results.

I tested with several other images, and the results do vary in all of them. Sometimes very close, sometimes very far (like this one).

Is there any expectancy to give the largest image possible to this API, for more accurate results, or the smaller, the better?

Using InceptionV3.

Many thanks!

[GantMan]

Good question! There are a few factors to calculate into account.

All images are resized for you to the same square. For inception it is 299x299. Given that your images are roughly the same dimensions, this shouldn't affect it too much. The resizing should be the same using a Bilinear Resize.

I do know Inception is more attracted to details than other models, so it might be more sensitive. Can you try other models for me?

If you resize your input images to 299x299 with bilinearresize, we can look at see if there are any major visual differences between them.

[ghnp5]

Thank you for the reply.

I was now going to test the images in different models as you asked.

I'm not sure why, but when I test the same images in https://nsfwjs.com/ , with the same InceptionV3, I get different results...

1365 × 2048:

Porn - 91.62%
Neutral - 4.46%
Sexy - 2.86%
Hentai - 0.96%
Drawing - 0.10%

480 × 720:

Porn - 97.40%
Sexy - 1.24%
Hentai - 0.67%
Neutral - 0.67%
Drawing - 0.02%

I confirm the model files I'm using are exactly the same as the ones hosted in this repository.

In my NodeJS script, I'm basically getting the image in a bytes array, then:

import jpeg from 'jpeg-js'; // 0.4.4
import * as tf from '@tensorflow/tfjs-node'; // 4.22.0

const convertJPG = async (img) => {
	// Decoded image in UInt8 Byte array
	const image = jpeg.decode(img, { useTArray: true });

	const numChannels = 3;
	const numPixels = image.width * image.height;
	const values = new Int32Array(numPixels * numChannels);

	for (let i = 0; i < numPixels; i++) {
		for (let c = 0; c < numChannels; ++c) {
			values[i * numChannels + c] = image.data[i * 4 + c];
		}
	}

	return tf.tensor3d(values, [image.height, image.width, numChannels], 'int32');
};

const image = await convertJPG(dataBuffer);
const predictions = await model.classify(image);

Which seems to match what is in the README.

I sent the two images (big and small) to your gant@i*******.r** email, if you'd like to have a look!

Many thanks!!

[haZya]

I think this can be considered expected behaviour within the margin of error. When you resize an image, it loses some of the data as the number of pixels is now much less. This is how images work, hence why SVGs are a thing. In fact, you could try converting an image from PNG to JPEG or vice versa and you'll still see a slight difference. Anything that changes the file size of the image changes some of its details ever so slightly. In your case, you are resizing it by about 2.84 times which is quite a large size difference that would make the result slightly different. Although in your case the difference is about 6% typically it'll be 1% - 2% in my testing which I consider to be within the margin of error.

You could maybe try using sharp instead of jpeg-js to see if it would make any difference.

const convert = async (imgBuffer: Buffer) => {
  const { data, info } = await sharp(imgBuffer)
    .raw()
    .toBuffer({ resolveWithObject: true });
  const numChannels = 3;
  const values = new Int32Array(info.width * info.height * numChannels);
  for (let i = 0; i < values.length; i++) values[i] = data[i];
  return tensor3d(values, [info.height, info.width, numChannels], "int32");
};

FYI, on the website https://nsfwjs.com, it directly passes the image to the classfy function without converting it since tf.browser.fromPixels() accepts these types tf.PixelData | ImageData | HTMLImageElement | HTMLCanvasElement | HTMLVideoElement | ImageBitmap. Which is why the results there are different from what you see when you do the same on nodejs where you have to convert it first since there is no dom on the server.

[ghnp5]

@haZya - Thank you very much for providing me the code for using sharp. I tried, and the values returned in my unit tests are slightly different - no much change. I actually prefer to use sharp than jpeg-js, so I'll keep it this way :)

I understand your explanation as well, regarding why the values are different in the website, and why the different sizes can make a difference. 👍🏻