Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Migrate the PDF build to Buildkite #26

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

DilumAluthge
Copy link
Member

@DilumAluthge DilumAluthge commented Sep 11, 2022

@StefanKarpinski
Copy link
Member

Thanks for doing that. Very much look forward to not having someone complain about the broken PDF link for the first time in the past seven years.

@DilumAluthge DilumAluthge marked this pull request as ready for review October 8, 2022 09:10
@mortenpi
Copy link
Collaborator

@DilumAluthge Buildkite is not yet deploying the PDF though, right? Or did I miss something?

@DilumAluthge
Copy link
Member Author

Ah you're right, I forgot we hadn't implemented the upload step yet.

What are the commands that need to be run for the upload? (Assuming that we have the PDF file.)

@mortenpi
Copy link
Collaborator

This is the function that handle the actual push to the assets branch:

function commit()
if get(ENV, "GITHUB_EVENT_NAME", nothing) == "pull_request"
@info "skipping commit from pull requests."
return
end
if !isdir(JULIA_DOCS_TMP)
@info "No new PDFs created, skipping commit."
return
end
@info "committing built PDF files."
# Make sure the repo is up to date
run(`git fetch origin`)
run(`git reset --hard origin/assets`)
# Copy file from JULIA_DOCS_TMP to JULIA_DOCS
for file in readdir(JULIA_DOCS_TMP)
endswith(file, ".pdf") || continue
from = joinpath(JULIA_DOCS_TMP, file)
@debug "Copying a PDF" file from pwd()
cp(from, file; force = true)
end
mktemp() do keyfile, iokey; mktemp() do sshconfig, iossh
# Set up keyfile
write(iokey, base64decode(get(ENV, "DOCUMENTER_KEY_PDF", "")))
close(iokey)
chmod(keyfile, 0o600)
# Set up ssh config file
print(iossh,
"""
Host github.com
StrictHostKeyChecking no
HostName github.com
IdentityFile $keyfile
BatchMode yes
""")
close(iossh)
chmod(sshconfig, 0o600)
# Configure git
run(`git config user.name "docs.julialang.org"`)
run(`git config user.email "[email protected]"`)
run(`git remote set-url origin [email protected]:JuliaLang/docs.julialang.org.git`)
run(`git config core.sshCommand "ssh -F $(sshconfig)"`)
# Committing all .pdf files
run(`git add '*.pdf'`)
run(`git commit --amend --date=now -m "PDF versions of Julia's manual."`)
# Push
run(`git push -f origin assets`)
end end
end

Basically, add/update the PDFs to the branch, and then commit --amend and force push. We squash the history on every commit so that the assets branch wouldn't blow up in size due to the PDFs from master.

@DilumAluthge
Copy link
Member Author

We also need to rename the PDF file, right? Currently, the PDF file being uploaded as a Buildkite artifact is just named TheJuliaLanguage.pdf. Where is the code/logic for determining the new name of the PDF file?

@mortenpi
Copy link
Collaborator

Sorry for the delay. But if I'm not mistaken, Documenter already sets the name, and we just copy any PDF there is: https://github.com/JuliaDocs/Documenter.jl/blob/998d26e568a20e3b1764d0c2cef30ebe4d2563ae/src/Writers/LaTeXWriter.jl#L167-L174

@DilumAluthge
Copy link
Member Author

Hmmm. I'm looking at a recent run on master (https://buildkite.com/julialang/julia-master/builds/17280#0183fada-b0ed-4d00-ab04-1d752f70514d), and in the artifacts tab, the PDF file is just named TheJuliaLanguage.pdf. So it doesn't look like any renaming is happening currently on Buildkite.

@mortenpi
Copy link
Collaborator

Oh, sorry, you are right. The renaming logic works by passing this

file = "julia-$(v).pdf"

which has the version number, to this

function copydocs(file)
isdir(JULIA_DOCS_TMP) || mkpath(JULIA_DOCS_TMP)
output = "$(JULIA_SOURCE)/doc/_build/pdf/en"
destination = "$(JULIA_DOCS_TMP)/$(file)"
for f in readdir(output)
if startswith(f, "TheJuliaLanguage") && endswith(f, ".pdf")
cp("$(output)/$(f)", destination; force=true)
@info "finished, output file copied to $(destination)."
break
end
end
end

where we look for any PDF and rename it to file. The simplest solution might be to run a small Julia script on Buildkite that checks if it's a tagged Julia, like we do in doc/make.jl to determine if it is a tagged version or not.

Although I guess a more robust solution would be to handle it in doc/make.jl? The Julia docs build will indeed always produce TheJuliaLanguage.pdf right now. We use the .version field for the version, which we could determine from TRAVIS_TAG since we do not explicitly set in the Julia make.jl, but TRAVIS_TAG is not set anywhere. So we will never attach the version number to the filename in Documenter. We could also set the version option to LaTeX there.

@DilumAluthge
Copy link
Member Author

but TRAVIS_TAG is not set anywhere

The Buildkite equivalent is BUILDKITE_TAG.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants