Merging pillars when one is empty gives empty result

[creideiki]

Description of Issue/Question

I'm trying to merge pillar data, as suggested in #28394 (comment) . However, if the data I try to merge in happens to be empty (as a result of template rendering), the end result is completely empty, ignoring any previous data.

Setup

pillar/top.sls:

base:
  '*':
    - common
    - modules

pillar/common.sls:

program:
  modules:
    00-definitions.conf: ~
    01-cleanup.conf: ~

pillar/modules.sls:

program:
  modules:
    {% if 'test' in grains['roles'] %}
    10-dostuff.conf: ~
    {% endif %}

From this, I expect a minion with the grain roles: [test] to get the pillar {'00-definitions.conf': None, '01-cleanup.conf': None, '10-dostuff.conf': None}, while other minions should get {'00-definitions.conf': None, '01-cleanup.conf': None}.

However, what actually happens is that minions without the role test get nothing at all:

# salt --no-colour 'salt*' grains.get roles   
saltminion:
    - test
saltmaster:
    - saltmaster

# salt --no-colour 'salt*' pillar.get program
saltminion:
    ----------
    modules:
        ----------
        00-definitions.conf:
            None
        01-cleanup.conf:
            None
        10-dostuff.conf:
            None
saltmaster:
    ----------
    modules:
        None

Changing pillar_source_merging_strategy from smart to recurse gives the same result.

Versions Report

Salt Version:
           Salt: 2015.8.8

Dependency Versions:
         Jinja2: 2.8
       M2Crypto: Not Installed
           Mako: Not Installed
         PyYAML: 3.11
          PyZMQ: 15.2.0
         Python: 2.7.11 (default, May  7 2016, 22:38:59)
           RAET: Not Installed
        Tornado: 4.3
            ZMQ: 4.1.4
           cffi: 1.6.0
       cherrypy: Not Installed
       dateutil: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
        libgit2: Not Installed
        libnacl: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.7
   mysql-python: Not Installed
      pycparser: 2.14
       pycrypto: 2.6.1
         pygit2: Not Installed
   python-gnupg: Not Installed
          smmap: Not Installed
        timelib: Not Installed

System Versions:
           dist: gentoo 2.2 
        machine: x86_64
        release: 4.5.5-hardened-r1
         system: Gentoo Base System 2.2 

[Ch3LL]

@creideiki I am not able to replicate this. Here is my test case:

pillar top file:

➜  salt cat /srv/salt/pillar/top.sls 
base:
  '*':
    - common
    - modules

➜  salt cat /srv/salt/pillar/common.sls 
program:
  modules:
    00-definitions.conf: ~
    01-cleanup.conf: ~

➜  salt cat /srv/salt/pillar/modules.sls 
program:
  modules:
    {% if 'test' in grains['roles'] %}
    10-dostuff.conf: ~
    {% endif %}

➜  salt sudo salt 'ch3ll-mas*' grains.get roles       
ch3ll-master:

➜  salt sudo salt 'ch3ll-mas*' saltutil.refresh_pillar
ch3ll-master:
    True

➜  salt sudo salt 'ch3ll-mas*' pillar.get program
ch3ll-master:
    ----------
    modules:
        ----------
        00-definitions.conf:
            None
        01-cleanup.conf:
            None

As you can see above a minion that does not have the roles grain indeed gets the two pillar values as expected. What does your master config look like? Also is there anything useful in the debug logs?

[creideiki]

I checked out 2016.3.0-589-gdd10fdc from Git and ran it on saltmaster, both as server and client. I still end up with an empty result.

I can't seem to find any debug logs relating to template rendering. Running commands with salt -l debug gives lots of details about network communication but nothing at all about what actually goes on internally.

My master configuration does not contain anything particularly exotic:

# grep -v ^# /etc/salt/master | grep .
interface: '::'
ipv6: True
file_recv: True
file_roots:
  base:
    - /etc/salt/repository/
hash_type: sha256
fileserver_backend:
  - roots
  - minion
pillar_roots:
  base:
    - /etc/salt/pillar
logstash_udp_handler:
  host: elk
  port: 5045
  version: 1

But I noticed that you have grains['roles'] unset, whereas I have it set to a list containing elements which don't match the template. Sure enough, if I empty /etc/salt/grains on saltmaster I get the expected result for both clients:

# salt --no-colour 'salt*' grains.get roles
saltminion:
    - test
saltmaster:

# salt --no-colour 'salt*' saltutil.refresh_pillar
saltmaster:
    True
saltminion:
    True

# salt --no-colour 'salt*' pillar.get program
saltminion:
    ----------
    modules:
        ----------
        00-definitions.conf:
            None
        01-cleanup.conf:
            None
        10-dostuff.conf:
            None
saltmaster:
    ----------
    modules:
        ----------
        00-definitions.conf:
            None
        01-cleanup.conf:
            None

And if I set it back so that saltmaster has a role that isn't test, the result goes back to being empty. Could you please try something like:

$ sudo salt --no-colour 'ch3ll-mas*' grains.set roles "['somethingelse']"

and see what results you get?

[Ch3LL]

@creideiki ahhh nice catch. I never ran pillar.items and now that I do its telling me I have a render error on the module sls. I can now replicate this:

➜  git sudo salt --no-colour 'ch3ll-mas*' grains.set roles "['somethingelse']" force=True
ch3ll-master:
    ----------
    changes:
        ----------
        roles:
            - somethingelse
    comment:
    result:
        True

➜  git sudo salt 'ch3ll-mas*' saltutil.refresh_pillar                                    
ch3ll-master:
    True

➜  git sudo salt 'ch3ll-mas*' pillar.item program    
ch3ll-master:
    ----------
    program:
        ----------
        modules:
            None

It seems its caused by the pillar program:modules being none when that pillar does not exist, but I think when merging pillars it should not cause this sort of behavior even if the one of the pillars is empty on a merge. I'll go ahead and label this as a bug.