Rails 7.0.4 ActiveSupport::MessageEncryptor::InvalidMessage - ruby-on-rails

Question

I've started a new ruby 3.1.2 + Rails 7.0.4 app and I'm deploying from Ubuntu 22.04 to Debian 11. I'm using capistrano, nginx, passenger, rbenv as part of my stack.
The error I'm getting when deploying is;
00:04 deploy:assets:precompile
01 $HOME/.rbenv/bin/rbenv exec bundle exec rake assets:precompile
01 rake aborted!
01 ActiveSupport::MessageEncryptor::InvalidMessage: ActiveSupport::MessageEncryptor::InvalidMessage
01 /var/www/profile/shared/bundle/ruby/3.1.0/gems/activesupport-7.0.4/lib/active_support/message_encryptor.rb:209:in `rescue in _decrypt'
01 /var/www/profile/shared/bundle/ruby/3.1.0/gems/activesupport-7.0.4/lib/active_support/message_encryptor.rb:186:in `_decrypt'
01 /var/www/profile/shared/bundle/ruby/3.1.0/gems/activesupport-7.0.4/lib/active_support/message_encryptor.rb:160:in `decrypt_and_verify'
01 /var/www/profile/shared/bundle/ruby/3.1.0/gems/activesupport-7.0.4/lib/active_support/messages/rotator.rb:22:in `decrypt_and_verify'
01 /var/www/profile/shared/bundle/ruby/3.1.0/gems/activesupport-7.0.4/lib/active_support/encrypted_file.rb:104:in `decrypt'
01 /var/www/profile/shared/bundle/ruby/3.1.0/gems/activesupport-7.0.4/lib/active_support/encrypted_file.rb:66:in `read'
01 /var/www/profile/shared/bundle/ruby/3.1.0/gems/activesupport-7.0.4/lib/active_support/encrypted_configuration.rb:21:in `read'
01 /var/www/profile/shared/bundle/ruby/3.1.0/gems/activesupport-7.0.4/lib/active_support/encrypted_configuration.rb:33:in `config'
01 /var/www/profile/shared/bundle/ruby/3.1.0/gems/activesupport-7.0.4/lib/active_support/encrypted_configuration.rb:48:in `options'
I've tried the following to fix it so far;
Created credentials using bin/rails credentials:edit
(commit, and deploy)
Created a production key and copied it over to production and configured capistrano to link that in
Configure production to use master key instead of production.key. I first removed the production.key I created in the last step, copied over my master.key to production shared area and added to linked_files to copy that in. This also failed.
Tried changing key generator hash from SHA256 to SHA1 (also didn't work) with the following
config.active_support.key_generator_hash_digest_class = OpenSSL::Digest::SHA1
Recreating credentials.yml.enc and master.key again from scratch and copying over the master.key to production and redeploying, also fails.
Update #1: Tried using production.key instead
I've tried creating a production key and I still get the same error on deploying.
Steps taken on my development box were;
$ rails secret
$ EDITOR=mg rails credentials:edit
$ rails credentials:edit --environment production
$ RAILS_ENV=production rails c
$ scp config/credentials/production.key 10.1.1.6:/var/www/profile/shared/config/credentials/
$ git add .
$ git ca -m 'Add production encrypted file'
Added the following to my deploy.rb
append :linked_files, 'config/credentials/production.key'
Then deployed
$ cap production deploy
I checked the release after deploy on the server and the production key file was properly linked from my shared config credentials area.
Update #2: Create production.yml.enc on production server
Just to test I logged into the production server and did the following commands;
Created production.yml.enc
$ cd /var/www/profile/releases/20221221021047
$ rm config/credentials/production.*
$ EDITOR=mg rails credentials:edit -e production
Test with rails console which wasn't working before, this now worked
$ RAILS_ENV=production rails c
Back on development box, I copied the new key from production, commited and deployed.
$ scp 10.1.1.6:/var/www/profile/releases/20221221021047/config/credentials/production.yml.enc ~/code/profile/config/credentials
$ git commit -am 'Updated production encrypted yaml with server generated one'
$ cap production deploy
Now I don't get the error on deploy (I still get it through nginx even after restarting nginx). It doesn't feel like this is the correct way to do this.
Update #3: remove .rbenv-vars file
I noticed that the documentation I was following got me to create a .rbenv-vars file on the production system. This was creating issues on my deploy. I remove the file and then I could create the production file and copy the production.key over and it now works, even in nginx.
- [X] DEV: Create production.yml.enc on dev and commit
: $ rm config/credentials/production.yml.enc
: $ scp config/credentials/production.key 10.1.1.6:/var/www/profile/shared/config/credentials/
: $ rake secret
: $ rails credentials:edit --environment production
Enter in 'secret_key_base: <secret numbers from rake secret command>'
: $ g ca -m 'updated production encrypted file with secret again'
: $ cap production deploy