[BUG] 3002.5 How to use an S3 external pillar?

[dcarriger]

Description
I want to use an S3 external pillar in conjunction with regular pillars on the Salt master. I am unable to identify why this is not working.

Setup
I am testing on the Salt master itself. I have environments set up in my pillar configuration:

pillar_roots:
...
  ops:
    - /srv/salt_roots/pillar/ops
    - /srv/salt_roots/pillar/base

I also have the external S3 pillar defined:

ext_pillar:
  - s3:
      bucket: my-ext-pillar-bucket
      multiple_env: False
      environment: ops
      prefix: ops
      verify_ssl: True
      service_url: s3.amazonaws.com
      s3_cache_expire: 30
      s3_sync_on_update: True
      path_style: False
      https_enable: True

My top file (in /srv/salt_roots/pillar/base/top.sls, we don't use per-environment top file) has some pillars defined:

ops:
  'ops-saltmaster-*':
    - aws <--------- Local
    - ntp <--------- Local
    - saltmaster <-- Local
    - test <--------- I want to get this from the S3 external pillar

Steps to Reproduce the behavior
If I try to run highstate, I get the following:

local:
    Data failed to compile:
----------
    Pillar failed to render with the following messages:
----------
    Specified SLS 'test' in environment 'ops' is not available on the salt master

If I remove the 'test' pillar from my top file (the only one coming from the S3 external pillar), then highstate gives me this instead:

local:
    Data failed to compile:
----------
    Pillar failed to render with the following messages:
----------
    Specified SLS 'aws' in environment 'ops' is not available on the salt master
----------
    Specified SLS 'ntp' in environment 'ops' is not available on the salt master
----------
    Specified SLS 'saltmaster' in environment 'ops' is not available on the salt master

I know the S3 bucket sync is working because I can see the data on my local FS:

/var/cache/salt/master/pillar_s3fs/ops/my-ext-pillar-bucket/ops

It has the expected data in it as well.

Expected behavior
I expect to be able to use both internal and external pillars in conjunction with one another and for the pillar data to be merged together.

Versions Report

salt --versions-report

(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)

Salt Version:
          Salt: 3002.5

Dependency Versions:
          cffi: Not Installed
      cherrypy: Not Installed
      dateutil: 2.7.3
     docker-py: Not Installed
         gitdb: 2.0.6
     gitpython: 3.0.7
        Jinja2: 2.10.1
       libgit2: Not Installed
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 0.6.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     pycparser: Not Installed
      pycrypto: Not Installed
  pycryptodome: 3.6.1
        pygit2: Not Installed
        Python: 3.8.5 (default, Jan 27 2021, 15:41:15)
  python-gnupg: 0.4.5
        PyYAML: 5.3.1
         PyZMQ: 18.1.1
         smmap: 2.0.5
       timelib: Not Installed
       Tornado: 4.5.3
           ZMQ: 4.3.2

System Versions:
          dist: ubuntu 20.04 focal
        locale: utf-8
       machine: x86_64
       release: 5.4.0-1038-aws
        system: Linux
       version: Ubuntu 20.04 focal