QuartoNotebookWorker precompile error when trying quarto render #210
Comments
|
I assume Julia 1.10 does not exhibit this same issue for you?
Does it appear to matter what is the contents of the notebook, or does this happen with any contents? Does that notebook use a specific If |
|
Additionally, is there any difference for you when running this with 1.5.57, https://github.com/quarto-dev/quarto-cli/releases/tag/v1.5.57? |
I upgraded from 1.5.57 because I ran into the error that was fixed by this PR quarto-dev/quarto-cli#10244 I missed, that I have QuartoNotebookWorker added to my local environment. But even after removing it from the environment it does not work. Rendering a single quarto file without a Project.toml works. |
|
Same issue for me on macos with julia-1.11.1 and quarto 1.5.57 ( |
Found a Workaround by activating a local environment and adding only QuartoNotebookRunner.jl and packages used inside my qmd document. Quarto uses this local environment and everything is fine. So I guess the precompilation of QuartoNotebookWorker failed because of incompatibility with some packages installed inside environment |
|
I think I fixed it for me by upgrading the local environment that I use for the packages in the Quarto Document. |
Just to clarify: the error was happening when you had a |
|
I think I also had to do an update of the local environment. I can't reproduce the exact steps anymore. |
|
Thanks, that's a few more clues at least from these two reports so we might be able to track down and resolve the underlying issue. |
|
How do I find which julia environment is used from Quarto to run QuartoNotebookRunner? This is the error I get from just running quarto preview with the attached qmd. This is the content of the qmd file that is run: |
That might be unrelated. Possibly a different bug here https://github.com/PumasAI/QuartoNotebookRunner.jl/blame/407921d34a671873280728f0dd8948c0c0bc03ce/src/QuartoNotebookWorker/src/render.jl#L452 Looks like |
|
The thing is that the qmd is correctly rendered into an ipynb when I use the Dev docs and run the QNR from a dedicated environment with QNR 0.11.6 |
Could you provide that |
|
Yes, the ipynb is attached: |
|
Looks like that plot only gets a |
|
I ran into this issue with precompilation too, and as above the failure condition involves (at least) a 1.10 Manifest and Quarto being run with 1.11. Interestingly, precompilation works fine when Project.toml contains only Random, but it fails when Project.toml contains only Distributions. So I bisected my way down Distributions's deps / transitive deps, and I think the culprit (or at least one of them) is REPL – in that this [deps]
REPL = "3fa0cd96-eef1-5676-8a61-b3b8758bbffb"Remainder of the setupI'm running Quarto off the main branch (quarto-dev/quarto-cli@e29433f). At first I was running
---
title: Test
engine: julia
execute:
daemon: false
---
```{julia}
versioninfo()
```
# Restart Quarto processes
kill -9 $(ps -ax | grep quarto | grep -v grep | cut -d" " -f1)
# Set up Manifest with Julia 1.10
rm Manifest.toml
julia +1.10 --project=. -e "using Pkg; Pkg.instantiate()"
# Render with Julia 1.11
QUARTO_JULIA=$(julia +1.11 -e "println(Sys.BINDIR)")/julia quarto render index.qmd |
|
Nice digging. I can reproduce this with your script. I also tried to test it on nightly instead of julia 1.11 but then the versioninfo that printed in the resulting html is 1.11 when I run it without the REPL dependency so I am wondering whether the QUARTO_JULIA environment variable is even doing what it is supposed to do. |
|
The environment variable does work, afaik, but you have to make sure that any existing background Quarto processes are killed before rerunning Quarto. That's what the line with If you see 'starting Julia control server' when rendering, I think that means it has indeed started a new Julia process, which should use the executable specified in |
|
Yes, there was some quarto process still hanging around. I run it now with nightly and ran into some new problems that I might open as a new issue. When I am switching from 1.12 to 1.11 I also get some problems from a Manifest file that is still hanging around and resolved with julia 1.12. This is the error I get from the switch from 1.12 to 1.11: Precompiling Pkg...
Info Given Pkg was explicitly requested, output will be shown live
ERROR: LoadError: ArgumentError: Package LibGit2 does not have Base64 in its dependencies:
- Note that the following manifests in the load path were resolved with a different
julia version, which may be the cause of the error. Try to re-resolve them in the
current version, or consider deleting them if that fails:
/run/user/1001/julia/Manifest.toml (v1.12.0-DEV)
- You may have a partially installed environment. Try `Pkg.instantiate()`
to ensure all packages in the environment are installed.
- Or, if you have LibGit2 checked out for development and have
added Base64 as a dependency but haven't updated your primary
environment's manifest file, try `Pkg.resolve()`.
- Otherwise you may need to report an issue with LibGit2
Stacktrace:
[1] macro expansion
@ ./loading.jl:2226 [inlined]
[2] macro expansion
@ ./lock.jl:273 [inlined]
[3] __require(into::Module, mod::Symbol)
@ Base ./loading.jl:2198
[4] #invoke_in_world#3
@ ./essentials.jl:1089 [inlined]
[5] invoke_in_world
@ ./essentials.jl:1086 [inlined]
[6] require(into::Module, mod::Symbol)
@ Base ./loading.jl:2191
[7] include
@ ./Base.jl:557 [inlined]
[8] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt128}}, source::String)
@ Base ./loading.jl:2790
[9] top-level scope
@ stdin:5
in expression starting at /home/fcremer/.julia/juliaup/julia-1.11.1+0.x64.linux.gnu/share/julia/stdlib/v1.11/LibGit2/src/LibGit2.jl:3
in expression starting at stdin:5
ERROR: LoadError: Failed to precompile LibGit2 [76f85450-5226-5b5a-8eaa-529ad045b433] to "/home/fcremer/.julia/compiled/v1.11/LibGit2/jl_LNfzcY".
Stacktrace:
[1] error(s::String)
@ Base ./error.jl:35
[2] compilecache(pkg::Base.PkgId, path::String, internal_stderr::IO, internal_stdout::IO, keep_loaded_modules::Bool; flags::Cmd, cacheflags::Base.CacheFlags, reasons::Dict{String, Int64}, isext::Bool)
@ Base ./loading.jl:3085
[3] (::Base.var"#1082#1083"{Base.PkgId})()
@ Base ./loading.jl:2492
[4] mkpidlock(f::Base.var"#1082#1083"{Base.PkgId}, at::String, pid::Int32; kwopts::@Kwargs{stale_age::Int64, wait::Bool})
@ FileWatching.Pidfile ~/.julia/juliaup/julia-1.11.1+0.x64.linux.gnu/share/julia/stdlib/v1.11/FileWatching/src/pidfile.jl:95
[5] #mkpidlock#6
@ ~/.julia/juliaup/julia-1.11.1+0.x64.linux.gnu/share/julia/stdlib/v1.11/FileWatching/src/pidfile.jl:90 [inlined]
[6] trymkpidlock(::Function, ::Vararg{Any}; kwargs::@Kwargs{stale_age::Int64})
@ FileWatching.Pidfile ~/.julia/juliaup/julia-1.11.1+0.x64.linux.gnu/share/julia/stdlib/v1.11/FileWatching/src/pidfile.jl:116
[7] #invokelatest#2
@ ./essentials.jl:1057 [inlined]
[8] invokelatest
@ ./essentials.jl:1052 [inlined]
[9] maybe_cachefile_lock(f::Base.var"#1082#1083"{Base.PkgId}, pkg::Base.PkgId, srcpath::String; stale_age::Int64)
@ Base ./loading.jl:3609
[10] maybe_cachefile_lock
@ ./loading.jl:3606 [inlined]
[11] _require(pkg::Base.PkgId, env::String)
@ Base ./loading.jl:2488
[12] __require_prelocked(uuidkey::Base.PkgId, env::String)
@ Base ./loading.jl:2315
[13] #invoke_in_world#3
@ ./essentials.jl:1089 [inlined]
[14] invoke_in_world
@ ./essentials.jl:1086 [inlined]
[15] _require_prelocked(uuidkey::Base.PkgId, env::String)
@ Base ./loading.jl:2302
[16] macro expansion
@ ./loading.jl:2241 [inlined]
[17] macro expansion
@ ./lock.jl:273 [inlined]
[18] __require(into::Module, mod::Symbol)
@ Base ./loading.jl:2198
[19] #invoke_in_world#3
@ ./essentials.jl:1089 [inlined]
[20] invoke_in_world
@ ./essentials.jl:1086 [inlined]
[21] require(into::Module, mod::Symbol)
@ Base ./loading.jl:2191
[22] include(mod::Module, _path::String)
@ Base ./Base.jl:557
[23] include(x::String)
@ Pkg ~/.julia/juliaup/julia-1.11.1+0.x64.linux.gnu/share/julia/stdlib/v1.11/Pkg/src/Pkg.jl:3
[24] top-level scope
@ ~/.julia/juliaup/julia-1.11.1+0.x64.linux.gnu/share/julia/stdlib/v1.11/Pkg/src/Pkg.jl:65
[25] include
@ ./Base.jl:557 [inlined]
[26] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt128}}, source::Nothing)
@ Base ./loading.jl:2790
[27] top-level scope
@ stdin:5
in expression starting at /home/fcremer/.julia/juliaup/julia-1.11.1+0.x64.linux.gnu/share/julia/stdlib/v1.11/Pkg/src/GitTools.jl:3
in expression starting at /home/fcremer/.julia/juliaup/julia-1.11.1+0.x64.linux.gnu/share/julia/stdlib/v1.11/Pkg/src/Pkg.jl:3
in expression starting at stdin:5
✗ LibGit2
✗ Pkg
0 dependencies successfully precompiled in 7 seconds. 22 already precompiled.
ERROR: LoadError: ArgumentError: Package LibGit2 does not have Base64 in its dependencies:
- Note that the following manifests in the load path were resolved with a different
julia version, which may be the cause of the error. Try to re-resolve them in the
current version, or consider deleting them if that fails:
/run/user/1001/julia/Manifest.toml (v1.12.0-DEV)
- You may have a partially installed environment. Try `Pkg.instantiate()`
to ensure all packages in the environment are installed.
- Or, if you have LibGit2 checked out for development and have
added Base64 as a dependency but haven't updated your primary
environment's manifest file, try `Pkg.resolve()`.
- Otherwise you may need to report an issue with LibGit2
Stacktrace:
[1] macro expansion
@ ./loading.jl:2226 [inlined]
[2] macro expansion
@ ./lock.jl:273 [inlined]
[3] __require(into::Module, mod::Symbol)
@ Base ./loading.jl:2198
[4] #invoke_in_world#3
@ ./essentials.jl:1089 [inlined]
[5] invoke_in_world
@ ./essentials.jl:1086 [inlined]
[6] require(into::Module, mod::Symbol)
@ Base ./loading.jl:2191
[7] include
@ ./Base.jl:557 [inlined]
[8] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt128}}, source::String)
@ Base ./loading.jl:2790
[9] top-level scope
@ stdin:5
in expression starting at /home/fcremer/.julia/juliaup/julia-1.11.1+0.x64.linux.gnu/share/julia/stdlib/v1.11/LibGit2/src/LibGit2.jl:3
in expression starting at stdin:5
ERROR: LoadError: Failed to precompile LibGit2 [76f85450-5226-5b5a-8eaa-529ad045b433] to "/home/fcremer/.julia/compiled/v1.11/LibGit2/jl_UTnVZD".
Stacktrace:
[1] error(s::String)
@ Base ./error.jl:35
[2] compilecache(pkg::Base.PkgId, path::String, internal_stderr::IO, internal_stdout::IO, keep_loaded_modules::Bool; flags::Cmd, cacheflags::Base.CacheFlags, reasons::Dict{String, Int64}, isext::Bool)
@ Base ./loading.jl:3085
[3] (::Base.var"#1082#1083"{Base.PkgId})()
@ Base ./loading.jl:2492
[4] mkpidlock(f::Base.var"#1082#1083"{Base.PkgId}, at::String, pid::Int32; kwopts::@Kwargs{stale_age::Int64, wait::Bool})
@ FileWatching.Pidfile ~/.julia/juliaup/julia-1.11.1+0.x64.linux.gnu/share/julia/stdlib/v1.11/FileWatching/src/pidfile.jl:95
[5] #mkpidlock#6
@ ~/.julia/juliaup/julia-1.11.1+0.x64.linux.gnu/share/julia/stdlib/v1.11/FileWatching/src/pidfile.jl:90 [inlined]
[6] trymkpidlock(::Function, ::Vararg{Any}; kwargs::@Kwargs{stale_age::Int64})
@ FileWatching.Pidfile ~/.julia/juliaup/julia-1.11.1+0.x64.linux.gnu/share/julia/stdlib/v1.11/FileWatching/src/pidfile.jl:116
[7] #invokelatest#2
@ ./essentials.jl:1057 [inlined]
[8] invokelatest
@ ./essentials.jl:1052 [inlined]
[9] maybe_cachefile_lock(f::Base.var"#1082#1083"{Base.PkgId}, pkg::Base.PkgId, srcpath::String; stale_age::Int64)
@ Base ./loading.jl:3609
[10] maybe_cachefile_lock
@ ./loading.jl:3606 [inlined]
[11] _require(pkg::Base.PkgId, env::String)
@ Base ./loading.jl:2488
[12] __require_prelocked(uuidkey::Base.PkgId, env::String)
@ Base ./loading.jl:2315
[13] #invoke_in_world#3
@ ./essentials.jl:1089 [inlined]
[14] invoke_in_world
@ ./essentials.jl:1086 [inlined]
[15] _require_prelocked(uuidkey::Base.PkgId, env::String)
@ Base ./loading.jl:2302
[16] macro expansion
@ ./loading.jl:2241 [inlined]
[17] macro expansion
@ ./lock.jl:273 [inlined]
[18] __require(into::Module, mod::Symbol)
@ Base ./loading.jl:2198
[19] #invoke_in_world#3
@ ./essentials.jl:1089 [inlined]
[20] invoke_in_world
@ ./essentials.jl:1086 [inlined]
[21] require(into::Module, mod::Symbol)
@ Base ./loading.jl:2191
[22] include(mod::Module, _path::String)
@ Base ./Base.jl:557
[23] include(x::String)
@ Pkg ~/.julia/juliaup/julia-1.11.1+0.x64.linux.gnu/share/julia/stdlib/v1.11/Pkg/src/Pkg.jl:3
[24] top-level scope
@ ~/.julia/juliaup/julia-1.11.1+0.x64.linux.gnu/share/julia/stdlib/v1.11/Pkg/src/Pkg.jl:65
[25] include
@ ./Base.jl:557 [inlined]
[26] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt128}}, source::String)
@ Base ./loading.jl:2790
[27] top-level scope
@ stdin:5
in expression starting at /home/fcremer/.julia/juliaup/julia-1.11.1+0.x64.linux.gnu/share/julia/stdlib/v1.11/Pkg/src/GitTools.jl:3
in expression starting at /home/fcremer/.julia/juliaup/julia-1.11.1+0.x64.linux.gnu/share/julia/stdlib/v1.11/Pkg/src/Pkg.jl:3
in expression starting at stdin:5
ERROR: LoadError: Failed to precompile Pkg [44cfe95a-1eb2-52ea-b672-e2afdf69b78f] to "/home/fcremer/.julia/compiled/v1.11/Pkg/jl_uEYXiy".
Stacktrace:
[1] error(s::String)
@ Base ./error.jl:35
[2] compilecache(pkg::Base.PkgId, path::String, internal_stderr::IO, internal_stdout::IO, keep_loaded_modules::Bool; flags::Cmd, cacheflags::Base.CacheFlags, reasons::Dict{String, Int64}, isext::Bool)
@ Base ./loading.jl:3085
[3] (::Base.var"#1082#1083"{Base.PkgId})()
@ Base ./loading.jl:2492
[4] mkpidlock(f::Base.var"#1082#1083"{Base.PkgId}, at::String, pid::Int32; kwopts::@Kwargs{stale_age::Int64, wait::Bool})
@ FileWatching.Pidfile ~/.julia/juliaup/julia-1.11.1+0.x64.linux.gnu/share/julia/stdlib/v1.11/FileWatching/src/pidfile.jl:95
[5] #mkpidlock#6
@ ~/.julia/juliaup/julia-1.11.1+0.x64.linux.gnu/share/julia/stdlib/v1.11/FileWatching/src/pidfile.jl:90 [inlined]
[6] trymkpidlock(::Function, ::Vararg{Any}; kwargs::@Kwargs{stale_age::Int64})
@ FileWatching.Pidfile ~/.julia/juliaup/julia-1.11.1+0.x64.linux.gnu/share/julia/stdlib/v1.11/FileWatching/src/pidfile.jl:116
[7] #invokelatest#2
@ ./essentials.jl:1057 [inlined]
[8] invokelatest
@ ./essentials.jl:1052 [inlined]
[9] maybe_cachefile_lock(f::Base.var"#1082#1083"{Base.PkgId}, pkg::Base.PkgId, srcpath::String; stale_age::Int64)
@ Base ./loading.jl:3609
[10] maybe_cachefile_lock
@ ./loading.jl:3606 [inlined]
[11] _require(pkg::Base.PkgId, env::String)
@ Base ./loading.jl:2488
[12] __require_prelocked(uuidkey::Base.PkgId, env::String)
@ Base ./loading.jl:2315
[13] #invoke_in_world#3
@ ./essentials.jl:1089 [inlined]
[14] invoke_in_world
@ ./essentials.jl:1086 [inlined]
[15] _require_prelocked(uuidkey::Base.PkgId, env::String)
@ Base ./loading.jl:2302
[16] macro expansion
@ ./loading.jl:2241 [inlined]
[17] macro expansion
@ ./lock.jl:273 [inlined]
[18] __require(into::Module, mod::Symbol)
@ Base ./loading.jl:2198
[19] #invoke_in_world#3
@ ./essentials.jl:1089 [inlined]
[20] invoke_in_world
@ ./essentials.jl:1086 [inlined]
[21] require(into::Module, mod::Symbol)
@ Base ./loading.jl:2191
in expression starting at /opt/quarto/share/julia/ensure_environment.jl:1
ERROR: Ensuring an updated julia server environment failed
Stack trace:
at ensureQuartoNotebookRunnerEnvironment (file:///opt/quarto/bin/quarto.js:41203:15)
at eventLoopTick (ext:core/01_core.js:175:7)
at async startOrReuseJuliaServer (file:///opt/quarto/bin/quarto.js:41111:13)
at async getJuliaServerConnection (file:///opt/quarto/bin/quarto.js:41224:24)
at async executeJulia (file:///opt/quarto/bin/quarto.js:41283:18)
at async Object.execute (file:///opt/quarto/bin/quarto.js:41028:20)
at async renderExecute (file:///opt/quarto/bin/quarto.js:85764:27)
at async renderFileInternal (file:///opt/quarto/bin/quarto.js:85932:43)
at async renderFiles (file:///opt/quarto/bin/quarto.js:85800:17)
at async render (file:///opt/quarto/bin/quarto.js:90702:21) |
|
Maybe we could use version specific Manifest files so that we won't run into this reresolving issue. See https://pkgdocs.julialang.org/v1/toml-files/#Different-Manifests-for-Different-Julia-versions |
Thanks for the additional investigation @penelopeysm.
Yes, that probably is part of the solution.
Making it clearer to the user that they need to update their manifest first is likely what the behaviour should be here as far as I'm concerned. Another option is to detect if the manifest is a different Julia version and then automatically resolve it, but I'm inclined to avoid modifying user environments and instead just inform them that they need to re-resolve and try again. |
I am trying to render a quarto document after updateing the quarto version to 1.6.33 and Julia 1.11 and I get the following error when running quarto render
The text was updated successfully, but these errors were encountered: