Implement HTTP transport

[technomancy]

nREPL uses bencode by default but can also be made to work over other protocols like HTTP: https://github.com/cemerick/drawbridge/

[technomancy]

I'd be interested in implementing this at some point.

[bbatsov]

Don't think this is going to happen any time soon, so I'll close this ticket. Feel free to reopen it if you ever get to implementing it.

[martinraison]

I'm also very interested in this

[bbatsov]

@martinraison PRs welcome. :-) I'd love to see this supported in CIDER, but it's really low on our list of priorities.

[martinraison]

I doubt I can find time to do that at the moment, but if I ever get to it I'll let you know. An intermediary solution I was thinking of was to setup a nrepl server that proxies all my requests through http. Far less elegant but that could do the trick for my needs.

[jconti]

Still no hope on this one?

[jconti]

Actually cannot drawbridge be configured in cider?

[bbatsov]

The problem is not the server. The problem is that you have to add support to CIDER for http communication. Basically all code using bencode directly should be touched by such a change and that's why no one has been very eager to work on this.

[dpsutton]

@jconti if you want to hack around on it, there's basically a single function that is where it bencodes the info and sends it off:

(defun nrepl-send-request (request callback connection)
  "Send REQUEST and register response handler CALLBACK using CONNECTION.
REQUEST is a pair list of the form (\"op\" \"operation\" \"par1-name\"
\"par1\" ... ). See the code of `nrepl-request:clone',
`nrepl-request:stdin', etc.
Return the ID of the sent message."
  (with-current-buffer connection
    (when (and (not (lax-plist-get request "session"))
               nrepl-session)
      (setq request (append request (list "session" nrepl-session))))
    (let* ((id (nrepl-next-request-id connection))
           (request (cons 'dict (lax-plist-put request "id" id)))
           (message (nrepl-bencode request)))
      (nrepl-log-message request 'request)
      (puthash id callback nrepl-pending-requests)
      (process-send-string nil message)
      id)))

if you added http calls there you are half way there instead of the process send string. On the other side, you'd need a new filter that just watches for incoming info and then hooks that info up against the pending requests in a dictionary. I think this return side will be much more difficult. Also, since we no longer have a process in the buffer, you'll need to check if the http requests are still going, etc.

[bbatsov]

Also, since we no longer have a process in the buffer, you'll need to check if the http requests are still going, etc.

Well, that's actually going to be pretty simple given the fact he'll just have an http connection. At least for non-streaming responses. At least there are a ton of http libraries, so all of the communication plumbing will come for free.

[jconti]

I think I will give this a try. Recently I found this little project which looks like it approaches the problem from a different perspective: https://github.com/malyn/lein-catapult

[holtzermann17]

@jconti Did you have any luck with that?

I have tried today -- I created a fork of lein-catapult with contemporary dependencies swapped in, though I get the same results as described below with the original repo.

For simplicity here is my .lein/profiles.clj pointing to the original version:

{:user {:plugins [[lein-pprint "1.1.1"]
                  [lein-ancient "0.6.15"]
                  [lein-catapult "0.0.1"]]
        :dependencies [[slamhound "1.3.1"]]
        }}

Here is one pointing to my fork:

{:user {:plugins [[lein-pprint "1.1.1"]
                  [lein-ancient "0.6.15"]
                  [reifyhealth/lein-git-down "0.3.5"]
                  [com.github.holtzermann17/lein-catapult "4fa2f909f334d2094b2885567f0f036da3518820"]]
        :repositories [["jitpack" "https://jitpack.io"]]
        :dependencies [[slamhound "1.3.1"]]}}

In either case, I'm able to run lein catapult. When I try to connect from CIDER, I get some more information back:

$ lein catapult http://USER:[email protected]:8080/repl
:repositories detected in user-level profiles! [:user] 
See https://github.com/technomancy/leiningen/wiki/Repeatability
Retrieving com/github/holtzermann17/lein-catapult/4fa2f909f334d2094b2885567f0f036da3518820/lein-catapult-4fa2f909f334d2094b2885567f0f036da3518820.pom from jitpack
Listening on port  33853 ; connected to Drawbridge.
nREPL connection accepted.
>>> {:op clone, :id 388}
>>> {:op clone, :id 389}
>>> {:op describe, :session ba6f45c9-8bc1-4e9b-8bf6-85c51fa10aec, :id 390}
<<< {:id 390, :session ba6f45c9-8bc1-4e9b-8bf6-85c51fa10aec, :aux {:current-ns user}, :ops {:describe {}, :eval {}, :load-file {}, :stdin {}, :clone {}, :interrupt {}, :close {}, :ls-sessions {}}, :versions {:nrepl {:major 0, :minor 6, :incremental 0, :version-string 0.6.0}, :clojure {:major 1, :minor 10, :incremental 0, :version-string 1.10.0}, :java {:major 11, :minor 0, :incremental 6, :version-string 11.0.6}}, :status [done]}
>>> {:file *cider-repl system-test/exchange:localhost:33853(clj)*, :nrepl.middleware.print/quota 1048576, :nrepl.middleware.print/print cider.nrepl.pprint/pprint, :op eval, :column 1, :line 1002, :id 391, :code (clojure.core/apply clojure.core/require clojure.main/repl-requires), :content-type true, :inhibit-cider-middleware true, :nrepl.middleware.print/stream? 1, :nrepl.middleware.print/options {:right-margin 70}, :session ba6f45c9-8bc1-4e9b-8bf6-85c51fa10aec}

CIDER shows that it is still trying to connect. I get this in the cider-repl buffer:

;; Connected to nREPL server - nrepl://localhost:33853
;; CIDER 0.24.0 (package: 20200215.223) (India), nREPL 0.6.0

... but I don't get a prompt. Maybe it is close to working though, given that it got that far?

Maybe it's just very slow?

---

I noticed that I can type things into the cider-repl buffer, and when I hit RET they get sent across to the Catapult. E.g., sending (+ 1 1) I see this printed on the terminal:

>>> {:ns user, :file *cider-repl system-test/exchange:localhost:33853(clj)*, :nrepl.middleware.print/quota 1048576, :nrepl.middleware.print/print cider.nrepl.pprint/pprint, :op eval, :column 1, :line 43, :id 5, :code (+ 1 1), :content-type true, :nrepl.middleware.print/stream? 1, :nrepl.middleware.print/options {:right-margin 70}, :session 5afdde71-bf21-4d00-8c76-85731518897d}

My app prints out this error + stack trace:

Exception in thread "nRepl-session-c46f188e-0c31-4c8c-b7ba-c3c8a1fd1783" java.lang.ClassCastException: class java.lang.String cannot be cast to class java.lang.Character (java.lang.String and java.lang.Character are in module java.base of loader 'bootstrap')
	at clojure.lang.RT.intCast(RT.java:1216)
	at nrepl.middleware.interruptible_eval$source_logging_pushback_reader.invokeStatic(interruptible_eval.clj:42)
	at nrepl.middleware.interruptible_eval$source_logging_pushback_reader.invoke(interruptible_eval.clj:39)
	at nrepl.middleware.interruptible_eval$evaluate.invokeStatic(interruptible_eval.clj:98)
	at nrepl.middleware.interruptible_eval$evaluate.invoke(interruptible_eval.clj:55)
	at nrepl.middleware.interruptible_eval$interruptible_eval$fn__79676$fn__79680.invoke(interruptible_eval.clj:142)
	at clojure.lang.AFn.run(AFn.java:22)
	at nrepl.middleware.session$session_exec$main_loop__79777$fn__79781.invoke(session.clj:171)
	at nrepl.middleware.session$session_exec$main_loop__79777.invoke(session.clj:170)
	at clojure.lang.AFn.run(AFn.java:22)
	at java.base/java.lang.Thread.run(Thread.java:834)

After waiting a while longer, I see this printed out on the command line:

>>> {:op eval, :code (seq (.split (System/getProperty "java.class.path") ":")), :session a7a5cfcd-d460-4924-9b8d-b4aa6eef7607, :id 6}

Followed by a big list with all of the dependencies in the app! (Abbreviated here.)

{:id 6, :session a7a5cfcd-d460-4924-9b8d-b4aa6eef7607, :ns user,
:value ("/home/joe/system-test/exchange/test" "/home/joe/system-test/exchange/test/clj" "/home/joe/system-test/exchange/src/clj" 
...
 "/usr/local/lib/m2/viebel/codox-klipse-theme/0.0.1/codox-klipse-theme-0.0.1.jar")}

But after that I waited a couple hours and saw nothing further.

The Lein Catapult README says:

Drawbridge sends a new GET request every time lein-catapult checks the nREPL transport. This causes lein-catapult to send an HTTP request to your web server every 100ms, which is not only wasteful, but also slow (since you have to wait for the next poll to get results back from your REPL). This is Drawbridge issue #10.