[BUG] [salt-ssh] `config.option` behavior is non-obvious (and possibly wrong?)

[tjlevel12]

Description
I'm seeing inconsistent behavior in config.option. I first came across this while trying to get the various postgres_ states and modules up and running (which utilize several config options for host, port, user, and password), but I'm able to reproduce with a small example.

• When calling config.option directly as a module, I'm able to retrieve pillar values but not grains values.
• When calling config.option from within a module (via __salt__["config.option"]), I'm able to retrieve grains values but not pillar values.

I've provided an example below.

Setup
salt-ssh was installed into and run from a virtualenv:

python3 -m venv ./venv
source ./venv/bin/activate
pip install salt-ssh

---

# ./Saltfile
salt-ssh:
  roster_file: ./roster
  config_dir: .

# ./master
root_dir: .
cachedir: ./cachedir
top_file_merging_strategy: same

state_output_diff: True

fileserver_backend:
  - roots

file_roots:
  base:
    - ./states

module_dirs:
  - ./states

pillar_roots:
  base:
    - ./pillar

# ./roster
localhost:
  host: localhost
  user: vagrant
  minion_opts:
    grains:
      foo.grain: this is grains value foo.grain

# ./pillar/top.sls
base:
  '*':
    - all

# ./pillar/all.sls
foo.pillar: this is the pillar value foo.pillar

# ./states/_modules/foo.py
def pillar():
    return __salt__["config.option"]("foo.pillar", default="foo.pillar not found")

def grain():
    return __salt__["config.option"]("foo.grain", default="foo.grain not found")

• on-prem machine
• VM (Virtualbox, KVM, etc. please specify): vagrant / VMWare Workstation Pro 16
• VM running on a cloud service, please be explicit and add details
• container (Kubernetes, Docker, containerd, etc. please specify)
• or a combination, please be explicit
• jails if it is FreeBSD

Steps to Reproduce the behavior

# verify the pillar value
$ salt-ssh localhost pillar.get foo.pillar
localhost:
    this is the pillar value foo.pillar

# verify the grains value
$ salt-ssh localhost grains.get foo.grain
localhost:
    this is grains value foo.grain

# run config.option directly, successfully fetch the pillar value
$ salt-ssh localhost config.option foo.pillar
localhost:
    this is the pillar value foo.pillar

# run config.option directly, fail to fetch the grains value
$ salt-ssh localhost config.option foo.grain
localhost:

# call __salt__["config.option"] via module, fail to fetch the pillar value
$ salt-ssh localhost foo.pillar
localhost:
    foo.pillar not found

# call __salt__["config.option"] via module, successfully fetch the grains value
$ salt-ssh localhost foo.grain
localhost:
    this is grains value foo.grain

Expected behavior
Whether called directly or via the __salt__ dictionary, config.option should correctly provide both the pillar and the grains values.

Versions Report

salt --versions-report

(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)

Salt Version:
          Salt: 3004
 
Dependency Versions:
          cffi: Not Installed
      cherrypy: Not Installed
      dateutil: Not Installed
     docker-py: Not Installed
         gitdb: 4.0.9
     gitpython: 3.1.24
        Jinja2: 3.0.3
       libgit2: Not Installed
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.3
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     pycparser: Not Installed
      pycrypto: Not Installed
  pycryptodome: 3.11.0
        pygit2: Not Installed
        Python: 3.8.10 (default, Sep 28 2021, 16:10:42)
  python-gnupg: Not Installed
        PyYAML: 6.0
         PyZMQ: 21.0.2
         smmap: 5.0.0
       timelib: Not Installed
       Tornado: 4.5.3
           ZMQ: 4.3.3
 
System Versions:
          dist: ubuntu 20.04 focal
        locale: utf-8
       machine: x86_64
       release: 5.4.0-89-generic
        system: Linux
       version: Ubuntu 20.04 focal
 

[welcome]

Hi there! Welcome to the Salt Community! Thank you for making your first contribution. We have a lengthy process for issues and PRs. Someone from the Core Team will follow up as soon as possible. In the meantime, here’s some information that may help as you continue your Salt journey.
Please be sure to review our Code of Conduct. Also, check out some of our community resources including:

• Community Wiki
• Salt’s Contributor Guide
• Join our Community Slack
• IRC on LiberaChat
• Salt Project YouTube channel
• Salt Project Twitch channel

There are lots of ways to get involved in our community. Every month, there are around a dozen opportunities to meet with other contributors and the Salt Core team and collaborate in real time. The best way to keep track is by subscribing to the Salt Community Events Calendar.
If you have additional questions, email us at saltproject@vmware.com. We’re glad you’ve joined our community and look forward to doing awesome things with you!

[tjlevel12]

Has there been any motion on this? Is there anything I can do to help move this along?

[tjlevel12]

@twangboy Just checking to see if there's been any motion on this, or if there's anything I can do to help move this along.

[tjlevel12]

@twangboy Once again checking to see if there's been any motion on this, or if there's anything I can do to help move this along. Have you been able to reproduce? Am I misunderstanding how config.option is supposed to work?

[Ch3LL]

Does this also occur without salt-ssh?

[garethgreenaway]

@tjlevel12 Checking in and seeing if you saw the comment above from @Ch3LL?

[tjlevel12]

@garethgreenaway I had not, thanks for tagging me.

@Ch3LL I'm not seeing the behavior when running on a masterless minion:

$ sudo salt-call --local pillar.get foo.pillar
local:
    this is the pillar value foo.pillar

$ sudo salt-call --local grains.get foo.grain
local:
    this is grains value foo.grain

$ sudo salt-call --local config.option foo.pillar
local:
    this is the pillar value foo.pillar

$ sudo salt-call --local config.option foo.grain
local:
    this is grains value foo.grain

$ sudo salt-call --local foo.pillar
local:
    this is the pillar value foo.pillar

$ sudo salt-call --local foo.grain
local:
    this is grains value foo.grain

Is it expected that salt-ssh behaves differently in this situation?

[tjlevel12]

And just to round things out, the behavior also seems to work correctly in a master/minion topology, so it definitely seems like a bug with salt-ssh.