CrUX API request fails behind proxy #3874

diginote · 2023-06-08T14:58:41Z

Have you read the documentation?

Yes, I've read the how to make a reproducable bug guide
Yes, I've read the how to debug my script guide

URL

https://www.google.de

What are you trying to accomplish

I tried to run the crux plugin behind a proxy in a docker container with this cli command:

docker run \
        -e NODE_DEBUG=net,http,tls \
        -e http_proxy=http://$HTTP_PROXY \
        -e https_proxy=http://$HTTP_PROXY \
        -e $TIMEZONE \
        --shm-size 2g \
        --rm \
        --network=$DOCKER_NETWORK \
        -v $RUN_VOLUME_MAP3 \
        sitespeedio/sitespeed.io:27.8.0-plus1 \
        --graphite.host=$GRAPHITE_HOST_NAME \
        --graphite.namespace sitespeed_io.desktop \
        --proxy.http=$HTTP_PROXY \
        --proxy.https=$HTTP_PROXY \
        --crux.key $CRUX_API_KEY \
        --crux.formFactor PHONE \
        --crux.formFactor DESKTOP \
        --crux.formFactor TABLET \
        --crux.formFactor ALL \
        --plugins.remove /gpsi/lib/index.js \
        --plugins.remove /lighthouse/index.js \
        --plugins.remove assets \
        --plugins.remove browsertime \
        --plugins.remove budget \
        --plugins.remove coach \
        --plugins.remove domains \
        --plugins.remove graphite \
        --plugins.remove harstorer \
        --plugins.remove html \
        --plugins.remove lateststorer \
        --plugins.remove metrics \
        --plugins.remove pagexray \
        --plugins.remove remove \
        --plugins.remove text \
        --plugins.remove thirdparty \
        --plugins.remove tracestorer \
        --browsertime.proxy.https=$HTTP_PROXY \
        --browsertime.proxy.http=$HTTP_PROXY \
        --v \
        sites.txt.crux

this is the result:

[2023-06-08 10:52:15] INFO: [plugin.crux] Get CrUx data for url https://www....
HTTP 43: call onSocket 4 0
HTTP 43: createConnection chromeuxreport.googleapis.com:443::::::::::::::::::::: [Object: null prototype] {
  host: 'chromeuxreport.googleapis.com',
  port: 443,
  path: null,
  headers: { 'Content-Type': 'application/json', 'Content-Length': 106 },
  method: 'POST',
  _defaultAgent: Agent {
    _events: [Object: null prototype] {
      free: [Function (anonymous)],
      newListener: [Function: maybeEnableKeylog]
    },
    _eventsCount: 2,
    _maxListeners: undefined,
    defaultPort: 443,
    protocol: 'https:',
    options: [Object: null prototype] { noDelay: true, path: null },
    requests: [Object: null prototype] {},
    sockets: [Object: null prototype] {
      'chromeuxreport.googleapis.com:443:::::::::::::::::::::': [Array]
    },
    freeSockets: [Object: null prototype] {},
    keepAliveMsecs: 1000,
    keepAlive: false,
    maxSockets: Infinity,
    maxFreeSockets: 256,
    scheduling: 'lifo',
    maxTotalSockets: Infinity,
    totalSocketCount: 4,
    maxCachedSessions: 100,
    _sessionCache: { map: {}, list: [] },
    [Symbol(kCapture)]: false
  },
  noDelay: true,
  servername: 'chromeuxreport.googleapis.com',
  _agentKey: 'chromeuxreport.googleapis.com:443:::::::::::::::::::::'
}
TLS 43: client _init handle? true
NET 43: pipe false null
NET 43: connect: find host chromeuxreport.googleapis.com
NET 43: connect: dns options { family: undefined, hints: 32 }
HTTP 43: sockets chromeuxreport.googleapis.com:443::::::::::::::::::::: 5 5
HTTP 43: write ret = true
HTTP 43: outgoing message end.
[2023-06-08 10:52:15] INFO: [plugin.crux] Get CrUx data for domain ...
HTTP 43: call onSocket 5 0
HTTP 43: createConnection chromeuxreport.googleapis.com:443::::::::::::::::::::: [Object: null prototype] {
  host: 'chromeuxreport.googleapis.com',
  port: 443,
  path: null,
  headers: { 'Content-Type': 'application/json', 'Content-Length': 63 },
  method: 'POST',
  _defaultAgent: Agent {
    _events: [Object: null prototype] {
      free: [Function (anonymous)],
      newListener: [Function: maybeEnableKeylog]
    },
    _eventsCount: 2,
    _maxListeners: undefined,
    defaultPort: 443,
    protocol: 'https:',
    options: [Object: null prototype] { noDelay: true, path: null },
    requests: [Object: null prototype] {},
    sockets: [Object: null prototype] {
      'chromeuxreport.googleapis.com:443:::::::::::::::::::::': [Array]
    },
    freeSockets: [Object: null prototype] {},
    keepAliveMsecs: 1000,
    keepAlive: false,
    maxSockets: Infinity,
    maxFreeSockets: 256,
    scheduling: 'lifo',
    maxTotalSockets: Infinity,
    totalSocketCount: 5,
    maxCachedSessions: 100,
    _sessionCache: { map: {}, list: [] },
    [Symbol(kCapture)]: false
  },
  noDelay: true,
  servername: 'chromeuxreport.googleapis.com',
  _agentKey: 'chromeuxreport.googleapis.com:443:::::::::::::::::::::'
}
TLS 43: client _init handle? true
NET 43: pipe false null
NET 43: connect: find host chromeuxreport.googleapis.com
NET 43: connect: dns options { family: undefined, hints: 32 }
HTTP 43: sockets chromeuxreport.googleapis.com:443::::::::::::::::::::: 6 6
HTTP 43: write ret = true
HTTP 43: outgoing message end.
(node:43) Warning: Setting the NODE_DEBUG environment variable to 'http' can expose sensitive data (such as passwords, tokens and authentication headers) in the resulting log.
(Use `node --trace-warnings ...` to show where the warning was created)
TLS 43: client initRead handle? true buffered? false
NET 43: _read
NET 43: _read wait for connection
TLS 43: client initRead handle? true buffered? false
NET 43: _read
NET 43: _read wait for connection
TLS 43: client initRead handle? true buffered? false
NET 43: _read
NET 43: _read wait for connection
TLS 43: client initRead handle? true buffered? false
NET 43: _read
NET 43: _read wait for connection
TLS 43: client initRead handle? true buffered? false
NET 43: _read
NET 43: _read wait for connection
TLS 43: client initRead handle? true buffered? false
NET 43: _read
NET 43: _read wait for connection
NET 43: afterConnect
NET 43: destroy
NET 43: close
NET 43: close handle
HTTP 43: SOCKET ERROR: connect ECONNREFUSED 142.250.186.138:443 Error: connect ECONNREFUSED 142.250.186.138:443
    at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1494:16)
node:events:491
      throw er; // Unhandled 'error' event
      ^

Error: connect ECONNREFUSED 142.250.186.138:443
    at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1494:16)
Emitted 'error' event on ClientRequest instance at:
    at TLSSocket.socketErrorListener (node:_http_client:502:9)
    at TLSSocket.emit (node:events:513:28)
    at emitErrorNT (node:internal/streams/destroy:151:8)
    at emitErrorCloseNT (node:internal/streams/destroy:116:3)
    at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {
  errno: -111,
  code: 'ECONNREFUSED',
  syscall: 'connect',
  address: '142.250.186.138',
  port: 443
}

Node.js v18.16.0

this are two sections of the configuration:

"browsertime": {
      "proxy": {
        "https": "my_proxy:3128",
        "http": "my_proxy:3128"
      }
	}
	
"proxy": {
      "http": "my_proxy:3128",
      "https": "my_proxy:3128"
    }

Although the proxy was set correctly, it seems to be simply ignored during the request. I think the reason is that it is not respected here: https://github.com/sitespeedio/sitespeed.io/blob/main/lib/plugins/crux/send.js

When I try to curl the googleapis endpoint from within the container everythings works:

# start shell in container
docker run  -e NODE_DEBUG=net,http,tls -e HTTP_PROXY=http://my_proxy:3128 -e HTTPS_PROXY=http://my_proxy:3128 --rm -it --entrypoint bash sitespeedio/sitespeed.io:27.8.0-plus1



#curl endpoint
curl --location 'https://chromeuxreport.googleapis.com/v1/records:queryRecord?key=...' \
--header 'Content-Type: application/json' \
--data '{
  "origin": "https://www.../"
}'

Everything is fine when I run it on my localhost (without a proxy).
It may be possible to use axios here. It already respects the environment variables set for a proxy (see http_proxy or https_proxy on https://axios-http.com/docs/req_config)

What browser did you use?

Chrome

How to reproduce

run behind a proxy


docker run \
        -e NODE_DEBUG=net,http,tls \
        -e http_proxy=http://$HTTP_PROXY \
        -e https_proxy=http://$HTTP_PROXY \
        -e $TIMEZONE \
        --shm-size 2g \
        --rm \
        --network=$DOCKER_NETWORK \
        -v $RUN_VOLUME_MAP3 \
        sitespeedio/sitespeed.io:27.8.0-plus1 \
        --graphite.host=$GRAPHITE_HOST_NAME \
        --graphite.namespace sitespeed_io.desktop \
        --proxy.http=$HTTP_PROXY \
        --proxy.https=$HTTP_PROXY \
        --crux.key $CRUX_API_KEY \
        --crux.formFactor PHONE \
        --crux.formFactor DESKTOP \
        --crux.formFactor TABLET \
        --crux.formFactor ALL \
        --plugins.remove /gpsi/lib/index.js \
        --plugins.remove /lighthouse/index.js \
        --plugins.remove assets \
        --plugins.remove browsertime \
        --plugins.remove budget \
        --plugins.remove coach \
        --plugins.remove domains \
        --plugins.remove graphite \
        --plugins.remove harstorer \
        --plugins.remove html \
        --plugins.remove lateststorer \
        --plugins.remove metrics \
        --plugins.remove pagexray \
        --plugins.remove remove \
        --plugins.remove text \
        --plugins.remove thirdparty \
        --plugins.remove tracestorer \
        --browsertime.proxy.https=$HTTP_PROXY \
        --browsertime.proxy.http=$HTTP_PROXY \
        --v \
        sites.txt.crux

sites.txt.crux contains lines like

https://www.google.de 01-start-page google-com



### Relevant log output

```shell
[2023-06-08 10:52:15] INFO: [plugin.crux] Get CrUx data for url https://www....
HTTP 43: call onSocket 4 0
HTTP 43: createConnection chromeuxreport.googleapis.com:443::::::::::::::::::::: [Object: null prototype] {
  host: 'chromeuxreport.googleapis.com',
  port: 443,
  path: null,
  headers: { 'Content-Type': 'application/json', 'Content-Length': 106 },
  method: 'POST',
  _defaultAgent: Agent {
    _events: [Object: null prototype] {
      free: [Function (anonymous)],
      newListener: [Function: maybeEnableKeylog]
    },
    _eventsCount: 2,
    _maxListeners: undefined,
    defaultPort: 443,
    protocol: 'https:',
    options: [Object: null prototype] { noDelay: true, path: null },
    requests: [Object: null prototype] {},
    sockets: [Object: null prototype] {
      'chromeuxreport.googleapis.com:443:::::::::::::::::::::': [Array]
    },
    freeSockets: [Object: null prototype] {},
    keepAliveMsecs: 1000,
    keepAlive: false,
    maxSockets: Infinity,
    maxFreeSockets: 256,
    scheduling: 'lifo',
    maxTotalSockets: Infinity,
    totalSocketCount: 4,
    maxCachedSessions: 100,
    _sessionCache: { map: {}, list: [] },
    [Symbol(kCapture)]: false
  },
  noDelay: true,
  servername: 'chromeuxreport.googleapis.com',
  _agentKey: 'chromeuxreport.googleapis.com:443:::::::::::::::::::::'
}
TLS 43: client _init handle? true
NET 43: pipe false null
NET 43: connect: find host chromeuxreport.googleapis.com
NET 43: connect: dns options { family: undefined, hints: 32 }
HTTP 43: sockets chromeuxreport.googleapis.com:443::::::::::::::::::::: 5 5
HTTP 43: write ret = true
HTTP 43: outgoing message end.
[2023-06-08 10:52:15] INFO: [plugin.crux] Get CrUx data for domain ...
HTTP 43: call onSocket 5 0
HTTP 43: createConnection chromeuxreport.googleapis.com:443::::::::::::::::::::: [Object: null prototype] {
  host: 'chromeuxreport.googleapis.com',
  port: 443,
  path: null,
  headers: { 'Content-Type': 'application/json', 'Content-Length': 63 },
  method: 'POST',
  _defaultAgent: Agent {
    _events: [Object: null prototype] {
      free: [Function (anonymous)],
      newListener: [Function: maybeEnableKeylog]
    },
    _eventsCount: 2,
    _maxListeners: undefined,
    defaultPort: 443,
    protocol: 'https:',
    options: [Object: null prototype] { noDelay: true, path: null },
    requests: [Object: null prototype] {},
    sockets: [Object: null prototype] {
      'chromeuxreport.googleapis.com:443:::::::::::::::::::::': [Array]
    },
    freeSockets: [Object: null prototype] {},
    keepAliveMsecs: 1000,
    keepAlive: false,
    maxSockets: Infinity,
    maxFreeSockets: 256,
    scheduling: 'lifo',
    maxTotalSockets: Infinity,
    totalSocketCount: 5,
    maxCachedSessions: 100,
    _sessionCache: { map: {}, list: [] },
    [Symbol(kCapture)]: false
  },
  noDelay: true,
  servername: 'chromeuxreport.googleapis.com',
  _agentKey: 'chromeuxreport.googleapis.com:443:::::::::::::::::::::'
}
TLS 43: client _init handle? true
NET 43: pipe false null
NET 43: connect: find host chromeuxreport.googleapis.com
NET 43: connect: dns options { family: undefined, hints: 32 }
HTTP 43: sockets chromeuxreport.googleapis.com:443::::::::::::::::::::: 6 6
HTTP 43: write ret = true
HTTP 43: outgoing message end.
(node:43) Warning: Setting the NODE_DEBUG environment variable to 'http' can expose sensitive data (such as passwords, tokens and authentication headers) in the resulting log.
(Use `node --trace-warnings ...` to show where the warning was created)
TLS 43: client initRead handle? true buffered? false
NET 43: _read
NET 43: _read wait for connection
TLS 43: client initRead handle? true buffered? false
NET 43: _read
NET 43: _read wait for connection
TLS 43: client initRead handle? true buffered? false
NET 43: _read
NET 43: _read wait for connection
TLS 43: client initRead handle? true buffered? false
NET 43: _read
NET 43: _read wait for connection
TLS 43: client initRead handle? true buffered? false
NET 43: _read
NET 43: _read wait for connection
TLS 43: client initRead handle? true buffered? false
NET 43: _read
NET 43: _read wait for connection
NET 43: afterConnect
NET 43: destroy
NET 43: close
NET 43: close handle
HTTP 43: SOCKET ERROR: connect ECONNREFUSED 142.250.186.138:443 Error: connect ECONNREFUSED 142.250.186.138:443
    at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1494:16)
node:events:491
      throw er; // Unhandled 'error' event
      ^

Error: connect ECONNREFUSED 142.250.186.138:443
    at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1494:16)
Emitted 'error' event on ClientRequest instance at:
    at TLSSocket.socketErrorListener (node:_http_client:502:9)
    at TLSSocket.emit (node:events:513:28)
    at emitErrorNT (node:internal/streams/destroy:151:8)
    at emitErrorCloseNT (node:internal/streams/destroy:116:3)
    at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {
  errno: -111,
  code: 'ECONNREFUSED',
  syscall: 'connect',
  address: '142.250.186.138',
  port: 443
}

Node.js v18.16.0

The text was updated successfully, but these errors were encountered:

diginote · 2023-06-09T16:19:06Z

I solved it for me like this: #3875

diginote added the bug label Jun 8, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CrUX API request fails behind proxy #3874

CrUX API request fails behind proxy #3874

diginote commented Jun 8, 2023

diginote commented Jun 9, 2023

CrUX API request fails behind proxy #3874

CrUX API request fails behind proxy #3874

Comments

diginote commented Jun 8, 2023

Have you read the documentation?

URL

What are you trying to accomplish

What browser did you use?

How to reproduce

diginote commented Jun 9, 2023