Error raised when testing with an https outbound request

[emmamaypaul]

Hello! First of all I'd like to say I really like and appreciate Bypass!!! However, I'm running into an issue I'm having trouble resolving. I have a test that makes an https scheme outbound request using Bypass, which gives me this error:

11:31:03.775 [error] Cowboy returned 400 because it was unable to parse the request headers.

This may happen because there are no headers, or there are too many headers
or the header name or value are too large (such as a large cookie).

You can customize those values when configuring your http/https
server. The configuration option and default values are shown below:

    protocol_options: [
      max_header_name_length: 64,
      max_header_value_length: 4096,
      max_headers: 100,
      max_request_line_length: 8096
    ]

The error is a little confusing because the headers I have are:

[
  {"accept", "application/x-www-form-urlencoded"},
  {"content-type", "application/x-www-form-urlencoded"}
]

When I change the outbound request scheme to http, the tests pass and no error is raised

It appears that Bypass is not intended to make https outbound requests, and I'm wondering if that is the case, or if there is some sort of configuration I am missing?

Thank you in advance for your help!!!

[tanguilp]

Hello,

Same problem here for testing a library that requires HTTPS (because implementing RFC8414).

Looking at the code of Bypass, there are no options to enable HTTPS requests instead of HTTP request. It should be possible to implement it by switching to ranch_ssl when such an option would be set and generating a keypair {private key, certificate} on startup.
However, the client won't accept the connection (e.g. if you use HTTPoison) since the newly generated certificate will not be recognised. Two solutions here: temporarily disabling TLS cert verification during tests (HTTPoison: :insecure option), or whitelisting the newly generated certificate on the TLS connection (cacerts and cacertfile optoins of the SSL Erlang library). Sadly, these two solutions require changing your code since these are options to be passed when initiating the connection (Poison.get...). Is there a better way?

[voltone]

In tests you pass in a custom endpoint anyway, to direct the client to Bypass, so passing in an additional option to set the CA store shouldn't be such a big deal, no?

Using a self-signed server cert would require a custom verify_fun rather than just a cacerts option. Bypass could offer such a function, or else issue a server cert from a private CA (both could be bundled with Bypass, or generated on-the-fly) and expose the CA cert for use in cacerts.

Is it safe to assume Bypass will always be used with 'localhost' as the hostname, or are there use-cases where a different hostname would be required?

[tanguilp]

Wouldn't be a big deal technically speaking, but it would mean having that parameter (similar to debug: yes) on your function interface. Taking the main example of the library, it'd look like this:

    {:ok, client} = TwitterClient.start_link(url: endpoint_url(bypass.port))
    assert {:error, :rate_limited} == TwitterClient.post_tweet(client, "Elixir is awesome!", ssl: [cacerts: [<newly_generated_certificate>]])

Not very elegant in my opinion, and it allows the users of the library to actually modify SSL (and other) options of the HTTP request.

Regarding getting a real keypair from a recognized CA, do you think that could be accepted? There would be an expiration date anyway.

Using localhost is sufficient for my use-case, not sure however there aren't others.

[voltone]

I actually think it is good practice to expose the ssl options to users of your API: it allows users to apply their own hardening (e.g. limit protocol versions or cipher suites), and it may be necessary to customize the CA trust store in an enterprise environment with corporate PKI (own trusted root).

An actual trusted certificate is not practical: leaving aside the question of what hostname it should be issued to, CAs do not allow public distribution of certificates they issue, and they will revoke them when irregularities are detected.

I created a package called x509 that can be used to generate self-signed or private CA certificates. It can be used to generate certificates for use in testing, like the phx.gen.cert task in Phoenix 1.4 but with far fewer lines of code. If and when I have some time I want to add the possibility to issue a range of local test certificates that can replace badssl.com as an offline test endpoint for negative tests. Maybe Bypass could leverage that for HTTPS testing in the future...

[tanguilp]

Never thought about this but that makes sense to expose the ssl option indeed. Plus it can help with this test use-case.
Nice package, if I've got time I'll propose a patch to Bypass exposing an option for using ranch_ssl in addition to ranch_tcp, and using your package to automatically generate a keypair.

[gushonorato]

Hello everyone! First of all, thanks for making available this great library. I'm working on an Elixir version of the Mechanize library (https://github.com/gushonorato/mechanizex) and I'm really missing bypass support to HTTPS. Does anyone know if it's hard to make bypass work with HTTPS? Maybe I can submit a patch if anyone could give me some directions.

[zachdaniel]

Yeah, supporting HTTPS properly would be a pretty big deal for us!

[ream88]

I had a call with @MSch today about this, and we both think that supporting SSL in bypass would be an overkill for now. The idea imho behind bypass is to test THAT a request was made, not HOW the request was made.

Is there any lib that does NOT work with only HTTP?

[tanguilp]

Some security libs that require use of https, such as https://github.com/tanguilp/oauth2_metadata_updater for instance.

However, using Tesla and its mocking abilities is a good alternative.

[Nezteb]

I learned about this issue from a recent blog post by @danschultzer where he talks about an alternative tool he built: https://danschultzer.com/posts/testserver-mock-third-party-services

[dustinfarris]

Is there any lib that does NOT work with only HTTP?

oidcc gets upset

https://github.com/erlef/oidcc/blob/d353ea2e822b14b620eea1cb25590f1173961e7d/src/oidcc_decode_util.erl#L88-L94