DM-51000: Prepare for eups less installations

[mwittgen]

DM-51000: Prepare for eups less installations

[leeskelvin]

I'm not an expert here, so entry-level question time. Should a redefinition like this work without issue on other butler installs outside the USDF? E.g., here at Princeton for example?

[timj]

What do you think is the issue? It's pointing at its own package so if this ticket merges it should be self-consistent. We are trying to stop relying on environment variables to find code that should be findable.

[leeskelvin]

I'm not objecting to the exercise one iota. I'm just not familiar with specifying resource paths in pipeline YAMLs this way, so wanted to raise the prospect. If it works without any extra config needed here at Princeton or elsewhere, that's a 👍 from me.

[timj]

The entire purpose of this work is to move items that python needs to load into the python tree so that python package resources can find them. The end state is that you do not need any env vars for the code to work.

[leeskelvin]

Right. I understand the end goal is for us to not need any env vars for the code to work.

Will this YAML file "just work", out of the box, without any extra config required, after this ticket gets merged, here at Princeton?

[timj]

Yes. The code doesn't care about where it is running (it's more portable than before).

It does close an interesting wrinkle in that previously you could set PYTHONPATH to one location and DRP_PIPE_DIR to a completely different location and the code would come from one place and the pipelines from another. That now can't happen by design.

[leeskelvin]

So on that topic then, if I'm developing drp_pipe and I want to use a locally-checked-out repo, I'm guessing setup -j -r ... will no longer be functional post-EUPS. What's the post-EUPS equivalent? If this is/will be documented somewhere, please do point me to that instead.

[timj]

setup -j -r . will be fine since it doesn't try to setup dependencies. We are working on making sure that when you are working from the EUPS-less build that sconsUtils still knows how to find things like afw.cfg for the C++ linking. For pure python packages they will Just Work. Granted we haven't done a deep investigation into how shadowing works if there is an installed drp_pipe and PYTHONPATH has an additional drp_pipe. I think there is an issue with entry points (you might get both) but I think that PYTHONPATH should win.

Note that for normally shared stack or lsstsw usage there won't be any change at all. This change is focused on CernVMFS and Docker situations where halving code import time is a real win. We are treating docker as the priority since that is what the bulk of our users are going to be using.

[leeskelvin]

It looks like this needs a final newline.

How robust is this kind of relative pathing across different installs?

[timj]

sconsUtils installs through the symlink so it's fine.

[leeskelvin]

It looks like this needs a final newline too.

[timj]

This is not what you think it is. GitHub does a terrible job of representing symlinks.

[leeskelvin]

If it doesn't need a trailing newline, that's fine. The symlink issue is unrelated to my comment here.

[timj]

GitHub is confused. It's a symlink but has no way to represent it and tries to say it needs a new line. Seems like it's not very common for GitHub. It's trying to tell you that this "file" is a symlink to the directory in ../../../../../pipelines

[leeskelvin]

Got it. I can't say I've seen that many symlinks merged to GH before, so that's showing my ignorance here.

[leeskelvin]

The renaming exercise seems sensible to me. A couple of clarification comment questions, mostly for my own understanding.