Use Artifact system to ship test .root files

[Moelf]

https://github.com/simeonschaub/ArtifactUtils.jl

[Yuan-Ru-Lin]

Where can I download the file (as a .tar.gz)?

[Yuan-Ru-Lin]

Or maybe the question should be "where should the file be uploaded to?"

[Moelf]

yeah right now they are https://github.com/JuliaHEP/UnROOT.jl/tree/master/test/samples

we can upload to a repo in JuliaHEP, and then you can point to URLs like: https://pkgdocs.julialang.org/v1/artifacts/#Basic-Usage

[Yuan-Ru-Lin]

Makes sense. Can I be a member of JuliaHEP and do that?

[tamasgal]

Yep sure, I invited you 🙂 We can figure out the details of that repository later.

[Yuan-Ru-Lin]

According to https://pkgdocs.julialang.org/v1/artifacts/, it seems that we need to compress those sample files before we can list them as Artifacts. Should they be compressed separately, or packed into a single .tar.gz? What do you think?

The latter would be easier, since if we have a samples.tar.gz, the only adjustment to runtests.jl would be just replacing joinpath(@__DIR__, samples) with artifacts"samples" (and the working directory wouldn't have to be test/ anymore, yeah!)

[Moelf]

I believe they have to be tarball (.tar), but doesn't have to be compressed (.tar.gz). But yeah I think pack everything into a single tarball is fine.

[Yuan-Ru-Lin]

I created a .tar file by tar -xf samples.tar samples/ and there was this error message

- file:///Users/yuanrulin/Projects/turn-sample-files-into-artifacts/samples.tar
    Error: This does not appear to be a TAR file/stream — malformed chksum field: ".root". Note: Tar.jl does not handle decompression; if the tarball is compressed you must use an external command like `gzcat` or package like CodecZlib.jl to decompress it. See the README file for examples.

If I use .tar.gz instead, things are good. Maybe we should go with .tar.gz?

I created another .tar file by tar -xf samples2.tar * inside samples/ and it now works.

I stole a call related to those samples and adopt the change. Now it looks like

julia> using UnROOT
julia> ROOTFile(joinpath(artifact"flat_sample_tar", "km3net_online.root"))
ROOTFile with 10 entries and 54 streamers.
/Users/yuanrulin/.julia/artifacts/75217ef39ca402609e402e1143faf3bac86f218b/km3net_online.root
├─ E (TTree)
│  └─ "Evt"
├─ KM3NET_TIMESLICE (TTree)
│  └─ "KM3NET_TIMESLICE"
├─ KM3NET_TIMESLICE_L0 (TTree)
│  └─ "km3net_timeslice_L0"
├─ KM3NET_TIMESLICE_L1 (TTree)
│  └─ "km3net_timeslice_L1"
├─ KM3NET_TIMESLICE_L2 (TTree)
│  └─ "km3net_timeslice_L2"
├─ KM3NET_TIMESLICE_SN (TTree)
│  └─ "km3net_timeslice_SN"
├─ KM3NET_EVENT (TTree)
│  └─ "KM3NET_EVENT"
├─ KM3NET_SUMMARYSLICE (TTree)
│  └─ "KM3NET_SUMMARYSLICE"
├─ JTRIGGER::JTriggerParameters (JTRIGGER::JTriggerParameters)
└─ META (TDirectory)
   ├─ JMeta (TNamed)
   └─ JDataWriter (TNamed)

It wouldn't be hard to replace all relevant calls with the adopted ones.

[Yuan-Ru-Lin]

But I do realize that the artifact will have to change if there are new files in the future. Maybe that's something of concern, or maybe not?

[Moelf]

good point... maybe we should keep these small files as how they are for now, and if we have larger (>5MB) test files in the future, use artifact system for those one by one to avoid changing artifact every time?

[Yuan-Ru-Lin]

Yeah. That sounds reasonable. I'd say maybe those test samples should be grouped according to some criteria. Otherwise, we will have to write something like joinpath(artifact"km3net_online", "km3net_online.root") which looks cumbersome. In short, artifacts are more like directories, instead of files.

[Yuan-Ru-Lin]

Another possibility is to make a .tar.gz for every sample file, and then define a macro such as

macro samples_str(file)
    :( joinpath(@artifact_str(replace($file, ".root"=>"")), $file) )
end

so that sample"km3_online.root" will result in joinpath(artifact"km3net_online", "kme3net_online.root").

What do you think?