GitLab: Receive 500 Error when trying to Create Merge Requests - ruby-on-rails

Question

I run my own Private GitLab CE VM on my Server. I'm updated to the latest version of 14.1.1.
As the title says: when I (or anyone else) clicks the Create merge request, we receive the 500 - Whoops, something went wrong on our end. This isn't even a large Branch: about 6 files with a few lines of code each. This has only started happening a couple days ago.
Now I should mention that I have been having on-going issues with the 500 Error; this includes Merging and Committing as well. As soon as I believe I've fixed the issue, it rears its ugly head back.
Logs
Here are some logs when I try creating a Merge Request:
root#gitlab:~# sudo gitlab-ctl tail | grep error
==> /var/log/gitlab/nginx/error.log <==
==> /var/log/gitlab/nginx/gitlab_error.log <==
2021-08-02_01:32:01.06995 {"#level":"debug","#message":"datasource: registering query type handler","#timestamp":"2021-08-02T01:32:01.069922Z","queryType":"random_walk_with_error"}
2021-08-02_01:32:01.07003 {"#level":"debug","#message":"datasource: registering query type handler","#timestamp":"2021-08-02T01:32:01.069948Z","queryType":"server_error_500"}
{"method":"POST","path":"/team/project/-/merge_requests","format":"html","controller":"Projects::MergeRequests::CreationsController","action":"create","status":500,"time":"2021-08-02T18:09:19.919Z","params":[{"key":"authenticity_token","value":"[FILTERED]"},{"key":"merge_request","value":{"title":"[FILTERED]","description":"[FILTERED]","assignee_ids":["0"],"reviewer_ids":["0"],"label_ids":[""],"force_remove_source_branch":"0","squash":"0","lock_version":"0","source_project_id":"41","source_branch":"AscentCharacterAttributes","target_project_id":"41","target_branch":"master"}},{"key":"merge_request_diff_head_sha","value":"f9d4a01e30d74b8b843e7b10b2810795e5c8966b"},{"key":"namespace_id","value":"team"},{"key":"project_id","value":"project"}],"remote_ip":"23.82.194.104","user_id":4,"username":"names_are_useless","ua":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:90.0) Gecko/20100101 Firefox/90.0","correlation_id":"01FC40SVDY9T1DQ0YTY1FMN7NM","meta.user":"names_are_useless","meta.project":"team/project","meta.root_namespace":"team","meta.caller_id":"Projects::MergeRequests::CreationsController#create","meta.remote_ip":"23.82.194.104","meta.feature_category":"code_review","meta.client_id":"user/4","gitaly_calls":8,"gitaly_duration_s":87.627672,"redis_calls":15,"redis_duration_s":0.00976,"redis_read_bytes":2886,"redis_write_bytes":2254,"redis_cache_calls":14,"redis_cache_duration_s":0.007988,"redis_cache_read_bytes":2705,"redis_cache_write_bytes":859,"redis_shared_state_calls":1,"redis_shared_state_duration_s":0.001772,"redis_shared_state_read_bytes":181,"redis_shared_state_write_bytes":1395,"db_count":31,"db_write_count":9,"db_cached_count":4,"cpu_s":2.259331,"mem_objects":246072,"mem_bytes":63989931,"mem_mallocs":58706,"mem_total_bytes":73832811,"exception.class":"ActiveRecord::StatementInvalid","exception.message":"PG::UnableToSend: no connection to the server\n","exception.backtrace":["app/services/issuable_base_service.rb:209:in `create'","app/services/merge_requests/base_service.rb:105:in `create'","app/services/merge_requests/create_service.rb:29:in `create'","app/services/merge_requests/create_service.rb:13:in `execute'","app/controllers/projects/merge_requests/creations_controller.rb:22:in `create'","app/controllers/application_controller.rb:483:in `set_current_admin'","lib/gitlab/session.rb:11:in `with_session'","app/controllers/application_controller.rb:474:in `set_session_storage'","lib/gitlab/i18n.rb:99:in `with_locale'","lib/gitlab/i18n.rb:105:in `with_user_locale'","app/controllers/application_controller.rb:468:in `set_locale'","app/controllers/application_controller.rb:462:in `set_current_context'","lib/gitlab/middleware/speedscope.rb:13:in `call'","lib/gitlab/request_profiler/middleware.rb:17:in `call'","lib/gitlab/jira/middleware.rb:19:in `call'","lib/gitlab/middleware/go.rb:20:in `call'","lib/gitlab/etag_caching/middleware.rb:21:in `call'","lib/gitlab/middleware/multipart.rb:172:in `call'","lib/gitlab/middleware/read_only/controller.rb:50:in `call'","lib/gitlab/middleware/read_only.rb:18:in `call'","lib/gitlab/middleware/same_site_cookies.rb:27:in `call'","lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'","lib/gitlab/middleware/basic_health_check.rb:25:in `call'","lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'","lib/gitlab/middleware/request_context.rb:21:in `call'","config/initializers/fix_local_cache_middleware.rb:11:in `call'","lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:19:in `call'","lib/gitlab/metrics/requests_rack_middleware.rb:74:in `call'","lib/gitlab/middleware/release_env.rb:12:in `call'"],"db_duration_s":0.20229,"view_duration_s":0.0,"duration_s":89.59493}
lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'
{"args":["/opt/gitlab/embedded/bin/git","--git-dir","/var/opt/gitlab/git-data/repositories/#hashed/3d/91/3d914f9348c9cc0ff8a79716700b9fcd4d2f3e711608004eb8f138bcba7f14d9.git","-c","core.fsyncObjectFiles=true","-c","gc.auto=0","-c","core.autocrlf=input","log","--pretty=%H","--reverse","--max-count","2147483647","master..0000000000000000000000000000000000000000","--end-of-options"],"command.exitCode":128,"command.inblock":0,"command.maxrss":63964,"command.oublock":0,"command.real_time_ms":6.851257,"command.system_time_ms":6.352,"command.user_time_ms":0,"correlation_id":"01FC40WYVZ9BBNJHH5PYVE4NWM","grpc.meta.auth_version":"v2","grpc.meta.client_name":"gitlab-sidekiq","grpc.meta.deadline_type":"unknown","grpc.method":"CommitsBetween","grpc.request.deadline":"2021-08-02T18:13:20.661Z","grpc.request.fullMethod":"/gitaly.CommitService/CommitsBetween","grpc.request.glProjectPath":"team/project","grpc.request.glRepository":"project-41","grpc.request.repoPath":"#hashed/3d/91/3d914f9348c9cc0ff8a79716700b9fcd4d2f3e711608004eb8f138bcba7f14d9.git","grpc.request.repoStorage":"default","grpc.service":"gitaly.CommitService","grpc.start_time":"2021-08-02T18:09:32.661Z","level":"error","msg":"fatal: Invalid revision range master..0000000000000000000000000000000000000000\n","path":"/opt/gitlab/embedded/bin/git","peer.address":"#","pid":900,"remote_ip":"127.0.0.1","span.kind":"server","system":"grpc","time":"2021-08-02T18:09:32.669Z","username":"NicholuasP"}
{"correlation_id":"01FC40WYVZ9BBNJHH5PYVE4NWM","error":"exit status 128","grpc.meta.auth_version":"v2","grpc.meta.client_name":"gitlab-sidekiq","grpc.meta.deadline_type":"unknown","grpc.method":"CommitsBetween","grpc.request.deadline":"2021-08-02T18:13:20.661Z","grpc.request.fullMethod":"/gitaly.CommitService/CommitsBetween","grpc.request.glProjectPath":"team/project","grpc.request.glRepository":"project-41","grpc.request.repoPath":"#hashed/3d/91/3d914f9348c9cc0ff8a79716700b9fcd4d2f3e711608004eb8f138bcba7f14d9.git","grpc.request.repoStorage":"default","grpc.service":"gitaly.CommitService","grpc.start_time":"2021-08-02T18:09:32.661Z","level":"info","msg":"ignoring git-log error","peer.address":"#","pid":1430,"remote_ip":"127.0.0.1","span.kind":"server","system":"grpc","time":"2021-08-02T18:09:32.669Z","username":"NicholuasP"}
{"severity":"WARN","time":"2021-08-02T18:09:34.054Z","class":"Git::BranchHooksService","correlation_id":"01FC40WYVZ9BBNJHH5PYVE4NWM","project_id":41,"project_path":"team/project","message":"Error creating pipeline","errors":"Reference not found","pipeline_params":{"before":"f9d4a01e30d74b8b843e7b10b2810795e5c8966b","after":"0000000000000000000000000000000000000000","ref":"refs/heads/AscentCharacterAttributes","variables_attributes":[],"checkout_sha":null},"retry":0}
root#gitlab:~# sudo gitlab-ctl tail nginx/gitlab_error.log
(Nothing)
root#gitlab:~# tail -f /var/log/gitlab/gitlab-rails/production.log | grep error
lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'
root#gitlab:~# tail -f /var/log/gitlab/nginx/error.log
(Nothing)
root#gitlab:~# tail -f /var/log/gitlab/nginx/gitlab_error.log
(Nothing)
And because the following logs are too long, I have pasted the outputs to PasteBin links:
tail -f /var/log/gitlab/gitlab-rails/production.log
tail -f /var/log/gitlab/gitlab-rails/production_json.log
tail -f /var/log/gitlab/gitlab-rails/api_json.log
And in case that isn't enough, I've tried to log as much of gitlab-ctl tail right when I click the Create merge request button: sudo gitlab-ctl tail
Obviously I gave generic names for the Team, Project, IPs and our Website.
Potential Root of the Problem
The most important piece, I think, looks to be in production.log:
==> /var/log/gitlab/gitlab-rails/production_json.log <==
{"method":"POST","path":"/team/project/-/merge_requests","format":"html","controller":"Projects::MergeRequests::CreationsController","action":"create","status":500,"time":"2021-08-03T16:56:23.192Z","params":[{"key":"authenticity_token","value":"[FILTERED]"},{"key":"merge_request","value":{"title":"[FILTERED]","description":"[FILTERED]","assignee_ids":["0"],"reviewer_ids":["0"],"label_ids":[""],"force_remove_source_branch":"0","squash":"0","lock_version":"0","source_project_id":"41","source_branch":"AscentCharacterAttributes","target_project_id":"41","target_branch":"master"}},{"key":"merge_request_diff_head_sha","value":"f9d4a01e30d74b8b843e7b10b2810795e5c8966b"},{"key":"namespace_id","value":"team"},{"key":"project_id","value":"project"}],"remote_ip":"1.2.3.4","user_id":4,"username":"names_are_useless","ua":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:88.0) Gecko/20100101 Firefox/88.0","correlation_id":"01FC6F15DTJ7VQD0WE820Z2ZW0","meta.user":"names_are_useless","meta.project":"team/project","meta.root_namespace":"team","meta.caller_id":"Projects::MergeRequests::CreationsController#create","meta.remote_ip":"1.2.3.4","meta.feature_category":"code_review","meta.client_id":"user/4","gitaly_calls":8,"gitaly_duration_s":82.237132,"redis_calls":13,"redis_duration_s":0.00787,"redis_read_bytes":2516,"redis_write_bytes":1470,"redis_cache_calls":12,"redis_cache_duration_s":0.007299,"redis_cache_read_bytes":2335,"redis_cache_write_bytes":722,"redis_shared_state_calls":1,"redis_shared_state_duration_s":0.000571,"redis_shared_state_read_bytes":181,"redis_shared_state_write_bytes":748,"db_count":29,"db_write_count":9,"db_cached_count":4,"cpu_s":2.260525,"mem_objects":250013,"mem_bytes":67239344,"mem_mallocs":75906,"mem_total_bytes":77239864,"exception.class":"ActiveRecord::StatementInvalid","exception.message":"PG::UnableToSend: no connection to the server\n","exception.backtrace":["app/services/issuable_base_service.rb:209:in `create'","app/services/merge_requests/base_service.rb:105:in `create'","app/services/merge_requests/create_service.rb:29:in `create'","app/services/merge_requests/create_service.rb:13:in `execute'","app/controllers/projects/merge_requests/creations_controller.rb:22:in `create'","app/controllers/application_controller.rb:483:in `set_current_admin'","lib/gitlab/session.rb:11:in `with_session'","app/controllers/application_controller.rb:474:in `set_session_storage'","lib/gitlab/i18n.rb:99:in `with_locale'","lib/gitlab/i18n.rb:105:in `with_user_locale'","app/controllers/application_controller.rb:468:in `set_locale'","app/controllers/application_controller.rb:462:in `set_current_context'","lib/gitlab/middleware/speedscope.rb:13:in `call'","lib/gitlab/request_profiler/middleware.rb:17:in `call'","lib/gitlab/jira/middleware.rb:19:in `call'","lib/gitlab/middleware/go.rb:20:in `call'","lib/gitlab/etag_caching/middleware.rb:21:in `call'","lib/gitlab/middleware/multipart.rb:172:in `call'","lib/gitlab/middleware/read_only/controller.rb:50:in `call'","lib/gitlab/middleware/read_only.rb:18:in `call'","lib/gitlab/middleware/same_site_cookies.rb:27:in `call'","lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'","lib/gitlab/middleware/basic_health_check.rb:25:in `call'","lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'","lib/gitlab/middleware/request_context.rb:21:in `call'","config/initializers/fix_local_cache_middleware.rb:11:in `call'","lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:19:in `call'","lib/gitlab/metrics/requests_rack_middleware.rb:74:in `call'","lib/gitlab/middleware/release_env.rb:12:in `call'"],"db_duration_s":0.28854,"view_duration_s":0.0,"duration_s":84.33137}
==> /var/log/gitlab/gitlab-rails/production.log <==
Completed 500 Internal Server Error in 85422ms (ActiveRecord: 164.6ms | Elasticsearch: 0.0ms | Allocations: 232093)
ActiveRecord::StatementInvalid (PG::UnableToSend: no connection to the server):
What causes the ActiveRecord::StatementInvalid (PG::UnableToSend: no connection to the server ): I have no earthly idea, but I current suspect is the root of the problem.
Any ideas what is going on? And if so, how to fix GitLab so I can create Merge Requests without receiving this error?