Not seeing certificate attributes when not running locally - hyperledger

Question

The Problem
When I deploy a 4 peer nodes with PBFT or NOOPS in the cloud, any user certificate attributes are not seen. The values are blank.
Observations
Everything works locally. This suggests that I am calling the API correctly, and the chaincode is accessing attributes correctly.
When I attach to the membership container, I see the correct membersrvc.yaml with aca.enabled set to true. This is the same yaml that works locally. For good measure, I'm also passing the ENV variable MEMBERSRVC_CA_ACA_ENABLED=true.
I can see the attributes for the users in the membership service's ACA database. (suggesting that the users were created with attributes)
When I look at the actual certificate from the log (Bytes to Hex then Base64 decode) I see the attributes. (Appending certificate [30 82 02 dd 30 8....)
All attributes are blank when deployed. No errors.
Membership Service Logs
I enabled debug logging, and see that Membership services thinks it's enabled ACA:
19:57:46.421 [server] main -> DEBU 049 ACA was enabled [aca.enabled == true]
19:57:46.421 [aca] Start -> INFO 04a Staring ACA services...
19:57:46.421 [aca] startACAP -> INFO 04b ACA PUBLIC gRPC API server started
19:57:46.421 [aca] Start -> INFO 04c ACA services started
This looks good. What am I missing?
Guess
Could it be that the underlying docker container the chaincode deploys into doesn't have security enabled? Does it use the ENV passed to the parent peer? One difference is that locally I'm using "dev mode" without the base-image shenanigans.
Membership Service
membersrvc:
container_name: membersrvc
image: hyperledger/fabric-membersrvc
volumes:
- /home/ec2-user/membership:/user/membership
- /var/hyperledger:/var/hyperledger
command: sh -c "cp /user/membership/membersrvc.yaml /opt/gopath/src/github.com/hyperledger/fabric/membersrvc && membersrvc"
restart: unless-stopped
environment:
- MEMBERSRVC_CA_ACA_ENABLED=true
ports:
- 7054:7054
Root Peer Service
rootpeer:
container_name: root-peer
image: hyperledger/fabric-peer
restart: unless-stopped
environment:
- CORE_VM_ENDPOINT=unix:///var/run/docker.sock
- CORE_LOGGING_LEVEL=DEBUG
- CORE_PEER_ID=vp1
- CORE_SECURITY_ENROLLID=vp1
- CORE_SECURITY_ENROLLSECRET=xxxxxxxx
- CORE_SECURITY_ENABLED=true
- CORE_SECURITY_ATTRIBUTES_ENABLED=true
- CORE_PEER_PKI_ECA_PADDR=members.x.net:7054
- CORE_PEER_PKI_TCA_PADDR=members.x.net:7054
- CORE_PEER_PKI_TLSCA_PADDR=members.x.net:7054
- CORE_PEER_VALIDATOR_CONSENSUS_PLUGIN=NOOPS
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- /var/hyperledger:/var/hyperledger
command: sh -c "peer node start"
ports:
- 7051:7051
- 7050:7050
Here's the request:
{
"jsonrpc": "2.0",
"method":"query",
"params": {
"chaincodeID": {
"name" :"659cb5dcc3063054e4c90908050eebf68eb2bd193cc1520f1f2d198f0ff42268"
},
"ctorMsg": {
"args":["get_results", "{\"Id\":\"abc123\"}"]
},
"secureContext": "user123",
"attributes":["account_id","role"]
},
"id": 2
}
Edited*: I previously thought this was just PBFT...but it's also happening on NOOPS on the cloud. I reduced the example to NOOPS.

Answer 1

My problem is that the fabric version inside the fabric-baseimage docker container is a good bit newer. This is my fault - because I populated that image with the fabric version manually.
Background
If one is using non-vagrant with 0.6 and not in DEV mode, deploying chaincode will have a "cannot find :latest tag" error. To solve this, I pulled a fabric-baseimage version, and populated it with what I needed, including a git-clone of fabric. I should have pulled the 0.6 branch, but instead it was pulling master.
So essentially, my fabric-peer, node-sdk deployer, and baseimage were using slightly different hyperledger versions.
After about 48 hours of configuration hell, I think I have it straightened out by sending everything back to 0.6. I have terraform spinning everything up successfully now.
I do wish the documentation included something about deploying in a non-dev multi-node environment.