NF(WiP): SpecObject .find and .__iadd__ #418
Draft
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…NDistribution, and VenvDistribution
…s, not instance (which may be None)
changed diff to a_and_b
collecting diff objects in a_only and b_only
Should make it possible to "join" multiple specifications, and also avoid duplicates while retracing. Seems to run without crashing on a sample but I do not think it works correctly yet
* origin/master: TST: resource/singularity: Adjust test for previous commit RF(PEP8): make resource/singularity.py pep8 compliant (besides TODO) BF: singularity 3.x has changed to "instance CMD" instead of "instance.CMD" BF: fix for singularity 3.x having new --version output format RF: use /bin/bash instead of /bin/ed or /sbin/iptables, define centrally BF: exception cannot be logged, preceding debug() already logs its exc_str TST: travis: Remove 'route' commands for NONETWORK run RF: external_versions: Drop cmd:datalad BF: skip: Use datalad module to check DataLad availability TST: orchestrator: Fix a test's prepare_remote() call TST: travis: Install datalad-container in the DataLad run DOC: run: Mention container job parameter ENH: orchestrators: Support containers in subdatasets BF: orchestrator: Fix target commit checkout for shell resource TST: orchestrators: Rename test and add description BF: orchestrators: Fix plain copy for inputs in subdirectory TST: orchestrators: Move repeated setup to dataset fixture TST: orchestrators: Check that branch is restored
…nly with files This is a very preliminary crude implementation, needed to bring sanity to test tracing conda and venv. Even without tests ATM :-/ Ideally we should rewamp configuration management so all those options could be specified at command line and have some centralized definition/management and easy ways to mock/patch for testing
…to use
TODOs:
- possibly RF for better API (changed to return dict in get_tracer_classes)
- we should know which tracers are available without importing the classes
which should be delayed until use
- might as well take care about
# TODO: automate discovery of available tracers
which should be possible if we unify naming for submodules/tracer classes
- that list of tracers should appear in the --help for -t, --tracer
- add basic tests for selection
Before RFing into dict it was a list to which we were appending. With switching to dict we started to override the same tracer. Now they all will exist and in the order they were added
* origin/master: (114 commits) 0.2.0 release changes BF: orchestrators: Make dirty check work better with subdatasets ENH: orchestrators: Add directory parameter to _assert_clean_repo() BF: orchestrators: Guard dirty check against Git configuration STY: orchestrators: Convert string command to list ENH: run: Abort if dataset is dirty TST: orchestrators: Switch more tests over to SSH resource TST: orchestrators: Move repeated instance creation to a helper ENH: datalad-pair-run orchestrator: Reset to original commit BF: orchestrators: Fix parsing status from ls-tree output TST: orchestrators: Drop module-scoped dataset fixture TST: orchestrators: Add check_orc_plain for SSH resource TST: orchestrators: Add ssh fixture TST: travis: Install DataLad system-wide CLN: Remove stray unicode_literals imports TST: Adjust ExceptionInfo use for compatiblity with pytest v5.0.0 DOC: orchestrators: Add missing word to comment BF: ssh: Don't pass None values for Fabric's connect_kwargs TST: travis: Set up SSH for localhost BF: conda: Temporary pin version in miniconda URL ...
Codecov Report
@@ Coverage Diff @@
## master #418 +/- ##
==========================================
- Coverage 89.48% 84.68% -4.81%
==========================================
Files 148 148
Lines 12112 12282 +170
==========================================
- Hits 10839 10401 -438
- Misses 1273 1881 +608
Continue to review full report at Codecov.
|
Codecov Report
@@ Coverage Diff @@
## master #418 +/- ##
==========================================
- Coverage 89.54% 84.59% -4.95%
==========================================
Files 148 148
Lines 12132 12296 +164
==========================================
- Hits 10863 10402 -461
- Misses 1269 1894 +625
Continue to review full report at Codecov.
|
returning actual attribute values, not strings, in SpecObject._diff_vals
|
On the call, @chaselgrove asked if anyone had a snippet for triggering the duplicate distributions in order to test this PR. I wasn't able to find a snippet posted anywhere (e.g., to generate linked traces in gh-367). If my understanding of the issue is correct, the necessary condition is for a tracer to add a new file to the list of unknown files that will be picked up by a tracer on the next iteration. Because of this, the order of the tracers in retrace.py matters. The current order in retrace.py is On a Debian system, here's a snippet that triggers two Debian distribution entries: cd $(mktemp -d --tmpdir reproman-retrace-XXXXXXX)
virtualenv --python=python3 venv
. ./venv/bin/activate
reproman retrace /bin/sh $PWD/venv/lib/python3.7/abc.pySo for the first pass, the On a non-Debian system I think this is harder to trigger. The easiest thing I came up with so far is to put diff --git a/reproman/interface/retrace.py b/reproman/interface/retrace.py
index 62f1be60d..597b327aa 100644
--- a/reproman/interface/retrace.py
+++ b/reproman/interface/retrace.py
@@ -244,6 +244,6 @@ def get_tracer_classes():
from reproman.distributions.vcs import VCSTracer
from reproman.distributions.docker import DockerTracer
from reproman.distributions.singularity import SingularityTracer
- Tracers = [DebTracer, RPMTracer, CondaTracer, VenvTracer, VCSTracer,
+ Tracers = [DebTracer, RPMTracer, CondaTracer, VCSTracer, VenvTracer,
DockerTracer, SingularityTracer]
return TracersThen pass cd $(mktemp -d --tmpdir reproman-retrace-XXXXXXX)
git clone https://github.com/docker/docker-py
virtualenv --python=python3 venv
. ./venv/bin/activate
pip install -e docker-py
reproman retrace $PWD/venv/lib/python3.7/site-packages/six.py $PWD/docker-py/setup.pyI see the distribution for the virtual environment sandwiched between two git distributions. output |
|
@chaselgrove will you have some time in the near future to address @kyleam 's comment? |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Sits on top of #273. Only the last commit (6e9178f to be rewritten during WiP) is pertinent to this PR ATM
Should make it possible to "join" multiple specifications, and also
avoid duplicates while retracing.
Seems to run without crashing on a sample but I do not think it works correctly yet