I run my own Private GitLab CE VM on my Server. I'm updated to the latest version of 14.1.1.
As the title says: when I (or anyone else) clicks the Create merge request, we receive the 500 - Whoops, something went wrong on our end. This isn't even a large Branch: about 6 files with a few lines of code each. This has only started happening a couple days ago.
Now I should mention that I have been having on-going issues with the 500 Error; this includes Merging and Committing as well. As soon as I believe I've fixed the issue, it rears its ugly head back.
Logs
Here are some logs when I try creating a Merge Request:
root#gitlab:~# sudo gitlab-ctl tail | grep error
==> /var/log/gitlab/nginx/error.log <==
==> /var/log/gitlab/nginx/gitlab_error.log <==
2021-08-02_01:32:01.06995 {"#level":"debug","#message":"datasource: registering query type handler","#timestamp":"2021-08-02T01:32:01.069922Z","queryType":"random_walk_with_error"}
2021-08-02_01:32:01.07003 {"#level":"debug","#message":"datasource: registering query type handler","#timestamp":"2021-08-02T01:32:01.069948Z","queryType":"server_error_500"}
{"method":"POST","path":"/team/project/-/merge_requests","format":"html","controller":"Projects::MergeRequests::CreationsController","action":"create","status":500,"time":"2021-08-02T18:09:19.919Z","params":[{"key":"authenticity_token","value":"[FILTERED]"},{"key":"merge_request","value":{"title":"[FILTERED]","description":"[FILTERED]","assignee_ids":["0"],"reviewer_ids":["0"],"label_ids":[""],"force_remove_source_branch":"0","squash":"0","lock_version":"0","source_project_id":"41","source_branch":"AscentCharacterAttributes","target_project_id":"41","target_branch":"master"}},{"key":"merge_request_diff_head_sha","value":"f9d4a01e30d74b8b843e7b10b2810795e5c8966b"},{"key":"namespace_id","value":"team"},{"key":"project_id","value":"project"}],"remote_ip":"23.82.194.104","user_id":4,"username":"names_are_useless","ua":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:90.0) Gecko/20100101 Firefox/90.0","correlation_id":"01FC40SVDY9T1DQ0YTY1FMN7NM","meta.user":"names_are_useless","meta.project":"team/project","meta.root_namespace":"team","meta.caller_id":"Projects::MergeRequests::CreationsController#create","meta.remote_ip":"23.82.194.104","meta.feature_category":"code_review","meta.client_id":"user/4","gitaly_calls":8,"gitaly_duration_s":87.627672,"redis_calls":15,"redis_duration_s":0.00976,"redis_read_bytes":2886,"redis_write_bytes":2254,"redis_cache_calls":14,"redis_cache_duration_s":0.007988,"redis_cache_read_bytes":2705,"redis_cache_write_bytes":859,"redis_shared_state_calls":1,"redis_shared_state_duration_s":0.001772,"redis_shared_state_read_bytes":181,"redis_shared_state_write_bytes":1395,"db_count":31,"db_write_count":9,"db_cached_count":4,"cpu_s":2.259331,"mem_objects":246072,"mem_bytes":63989931,"mem_mallocs":58706,"mem_total_bytes":73832811,"exception.class":"ActiveRecord::StatementInvalid","exception.message":"PG::UnableToSend: no connection to the server\n","exception.backtrace":["app/services/issuable_base_service.rb:209:in `create'","app/services/merge_requests/base_service.rb:105:in `create'","app/services/merge_requests/create_service.rb:29:in `create'","app/services/merge_requests/create_service.rb:13:in `execute'","app/controllers/projects/merge_requests/creations_controller.rb:22:in `create'","app/controllers/application_controller.rb:483:in `set_current_admin'","lib/gitlab/session.rb:11:in `with_session'","app/controllers/application_controller.rb:474:in `set_session_storage'","lib/gitlab/i18n.rb:99:in `with_locale'","lib/gitlab/i18n.rb:105:in `with_user_locale'","app/controllers/application_controller.rb:468:in `set_locale'","app/controllers/application_controller.rb:462:in `set_current_context'","lib/gitlab/middleware/speedscope.rb:13:in `call'","lib/gitlab/request_profiler/middleware.rb:17:in `call'","lib/gitlab/jira/middleware.rb:19:in `call'","lib/gitlab/middleware/go.rb:20:in `call'","lib/gitlab/etag_caching/middleware.rb:21:in `call'","lib/gitlab/middleware/multipart.rb:172:in `call'","lib/gitlab/middleware/read_only/controller.rb:50:in `call'","lib/gitlab/middleware/read_only.rb:18:in `call'","lib/gitlab/middleware/same_site_cookies.rb:27:in `call'","lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'","lib/gitlab/middleware/basic_health_check.rb:25:in `call'","lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'","lib/gitlab/middleware/request_context.rb:21:in `call'","config/initializers/fix_local_cache_middleware.rb:11:in `call'","lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:19:in `call'","lib/gitlab/metrics/requests_rack_middleware.rb:74:in `call'","lib/gitlab/middleware/release_env.rb:12:in `call'"],"db_duration_s":0.20229,"view_duration_s":0.0,"duration_s":89.59493}
lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'
{"args":["/opt/gitlab/embedded/bin/git","--git-dir","/var/opt/gitlab/git-data/repositories/#hashed/3d/91/3d914f9348c9cc0ff8a79716700b9fcd4d2f3e711608004eb8f138bcba7f14d9.git","-c","core.fsyncObjectFiles=true","-c","gc.auto=0","-c","core.autocrlf=input","log","--pretty=%H","--reverse","--max-count","2147483647","master..0000000000000000000000000000000000000000","--end-of-options"],"command.exitCode":128,"command.inblock":0,"command.maxrss":63964,"command.oublock":0,"command.real_time_ms":6.851257,"command.system_time_ms":6.352,"command.user_time_ms":0,"correlation_id":"01FC40WYVZ9BBNJHH5PYVE4NWM","grpc.meta.auth_version":"v2","grpc.meta.client_name":"gitlab-sidekiq","grpc.meta.deadline_type":"unknown","grpc.method":"CommitsBetween","grpc.request.deadline":"2021-08-02T18:13:20.661Z","grpc.request.fullMethod":"/gitaly.CommitService/CommitsBetween","grpc.request.glProjectPath":"team/project","grpc.request.glRepository":"project-41","grpc.request.repoPath":"#hashed/3d/91/3d914f9348c9cc0ff8a79716700b9fcd4d2f3e711608004eb8f138bcba7f14d9.git","grpc.request.repoStorage":"default","grpc.service":"gitaly.CommitService","grpc.start_time":"2021-08-02T18:09:32.661Z","level":"error","msg":"fatal: Invalid revision range master..0000000000000000000000000000000000000000\n","path":"/opt/gitlab/embedded/bin/git","peer.address":"#","pid":900,"remote_ip":"127.0.0.1","span.kind":"server","system":"grpc","time":"2021-08-02T18:09:32.669Z","username":"NicholuasP"}
{"correlation_id":"01FC40WYVZ9BBNJHH5PYVE4NWM","error":"exit status 128","grpc.meta.auth_version":"v2","grpc.meta.client_name":"gitlab-sidekiq","grpc.meta.deadline_type":"unknown","grpc.method":"CommitsBetween","grpc.request.deadline":"2021-08-02T18:13:20.661Z","grpc.request.fullMethod":"/gitaly.CommitService/CommitsBetween","grpc.request.glProjectPath":"team/project","grpc.request.glRepository":"project-41","grpc.request.repoPath":"#hashed/3d/91/3d914f9348c9cc0ff8a79716700b9fcd4d2f3e711608004eb8f138bcba7f14d9.git","grpc.request.repoStorage":"default","grpc.service":"gitaly.CommitService","grpc.start_time":"2021-08-02T18:09:32.661Z","level":"info","msg":"ignoring git-log error","peer.address":"#","pid":1430,"remote_ip":"127.0.0.1","span.kind":"server","system":"grpc","time":"2021-08-02T18:09:32.669Z","username":"NicholuasP"}
{"severity":"WARN","time":"2021-08-02T18:09:34.054Z","class":"Git::BranchHooksService","correlation_id":"01FC40WYVZ9BBNJHH5PYVE4NWM","project_id":41,"project_path":"team/project","message":"Error creating pipeline","errors":"Reference not found","pipeline_params":{"before":"f9d4a01e30d74b8b843e7b10b2810795e5c8966b","after":"0000000000000000000000000000000000000000","ref":"refs/heads/AscentCharacterAttributes","variables_attributes":[],"checkout_sha":null},"retry":0}
root#gitlab:~# sudo gitlab-ctl tail nginx/gitlab_error.log
(Nothing)
root#gitlab:~# tail -f /var/log/gitlab/gitlab-rails/production.log | grep error
lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'
root#gitlab:~# tail -f /var/log/gitlab/nginx/error.log
(Nothing)
root#gitlab:~# tail -f /var/log/gitlab/nginx/gitlab_error.log
(Nothing)
And because the following logs are too long, I have pasted the outputs to PasteBin links:
tail -f /var/log/gitlab/gitlab-rails/production.log
tail -f /var/log/gitlab/gitlab-rails/production_json.log
tail -f /var/log/gitlab/gitlab-rails/api_json.log
And in case that isn't enough, I've tried to log as much of gitlab-ctl tail right when I click the Create merge request button: sudo gitlab-ctl tail
Obviously I gave generic names for the Team, Project, IPs and our Website.
Potential Root of the Problem
The most important piece, I think, looks to be in production.log:
==> /var/log/gitlab/gitlab-rails/production_json.log <==
{"method":"POST","path":"/team/project/-/merge_requests","format":"html","controller":"Projects::MergeRequests::CreationsController","action":"create","status":500,"time":"2021-08-03T16:56:23.192Z","params":[{"key":"authenticity_token","value":"[FILTERED]"},{"key":"merge_request","value":{"title":"[FILTERED]","description":"[FILTERED]","assignee_ids":["0"],"reviewer_ids":["0"],"label_ids":[""],"force_remove_source_branch":"0","squash":"0","lock_version":"0","source_project_id":"41","source_branch":"AscentCharacterAttributes","target_project_id":"41","target_branch":"master"}},{"key":"merge_request_diff_head_sha","value":"f9d4a01e30d74b8b843e7b10b2810795e5c8966b"},{"key":"namespace_id","value":"team"},{"key":"project_id","value":"project"}],"remote_ip":"1.2.3.4","user_id":4,"username":"names_are_useless","ua":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:88.0) Gecko/20100101 Firefox/88.0","correlation_id":"01FC6F15DTJ7VQD0WE820Z2ZW0","meta.user":"names_are_useless","meta.project":"team/project","meta.root_namespace":"team","meta.caller_id":"Projects::MergeRequests::CreationsController#create","meta.remote_ip":"1.2.3.4","meta.feature_category":"code_review","meta.client_id":"user/4","gitaly_calls":8,"gitaly_duration_s":82.237132,"redis_calls":13,"redis_duration_s":0.00787,"redis_read_bytes":2516,"redis_write_bytes":1470,"redis_cache_calls":12,"redis_cache_duration_s":0.007299,"redis_cache_read_bytes":2335,"redis_cache_write_bytes":722,"redis_shared_state_calls":1,"redis_shared_state_duration_s":0.000571,"redis_shared_state_read_bytes":181,"redis_shared_state_write_bytes":748,"db_count":29,"db_write_count":9,"db_cached_count":4,"cpu_s":2.260525,"mem_objects":250013,"mem_bytes":67239344,"mem_mallocs":75906,"mem_total_bytes":77239864,"exception.class":"ActiveRecord::StatementInvalid","exception.message":"PG::UnableToSend: no connection to the server\n","exception.backtrace":["app/services/issuable_base_service.rb:209:in `create'","app/services/merge_requests/base_service.rb:105:in `create'","app/services/merge_requests/create_service.rb:29:in `create'","app/services/merge_requests/create_service.rb:13:in `execute'","app/controllers/projects/merge_requests/creations_controller.rb:22:in `create'","app/controllers/application_controller.rb:483:in `set_current_admin'","lib/gitlab/session.rb:11:in `with_session'","app/controllers/application_controller.rb:474:in `set_session_storage'","lib/gitlab/i18n.rb:99:in `with_locale'","lib/gitlab/i18n.rb:105:in `with_user_locale'","app/controllers/application_controller.rb:468:in `set_locale'","app/controllers/application_controller.rb:462:in `set_current_context'","lib/gitlab/middleware/speedscope.rb:13:in `call'","lib/gitlab/request_profiler/middleware.rb:17:in `call'","lib/gitlab/jira/middleware.rb:19:in `call'","lib/gitlab/middleware/go.rb:20:in `call'","lib/gitlab/etag_caching/middleware.rb:21:in `call'","lib/gitlab/middleware/multipart.rb:172:in `call'","lib/gitlab/middleware/read_only/controller.rb:50:in `call'","lib/gitlab/middleware/read_only.rb:18:in `call'","lib/gitlab/middleware/same_site_cookies.rb:27:in `call'","lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'","lib/gitlab/middleware/basic_health_check.rb:25:in `call'","lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'","lib/gitlab/middleware/request_context.rb:21:in `call'","config/initializers/fix_local_cache_middleware.rb:11:in `call'","lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:19:in `call'","lib/gitlab/metrics/requests_rack_middleware.rb:74:in `call'","lib/gitlab/middleware/release_env.rb:12:in `call'"],"db_duration_s":0.28854,"view_duration_s":0.0,"duration_s":84.33137}
==> /var/log/gitlab/gitlab-rails/production.log <==
Completed 500 Internal Server Error in 85422ms (ActiveRecord: 164.6ms | Elasticsearch: 0.0ms | Allocations: 232093)
ActiveRecord::StatementInvalid (PG::UnableToSend: no connection to the server):
What causes the ActiveRecord::StatementInvalid (PG::UnableToSend: no connection to the server ): I have no earthly idea, but I current suspect is the root of the problem.
Any ideas what is going on? And if so, how to fix GitLab so I can create Merge Requests without receiving this error?
I have hundreds of files in a directory that I would like to count the occurrence of a string in each file.
I would like the output to be a summary file that contains the original file name plus the count (ideally on the same line)
for example
file1 6
file2 3
file3 4
etc
Thanks for your consideration
CAUTION: I am pretty much an enthusiastic amateur, so take everything with a grain of salt.
Several questions for you - depending on your answers, the solution below may need some adjustments.
Are all your files in the same directory, or do you also need to look through subdirectories and sub-subdirectories, etc.? Below I make the simplest assumption - that all your files are in a single directory.
Are all your files text files? In the example below, the directory will contain text files, executable files, symbolic links, and directories; the count will only be given for text files. (What linux believe to be text files, anyway.)
There may be files that do not contain the searched-for string at all. Those are not included in the output below. Do you need to show them too, with a count of 0?
I assume by "count occurrences" you mean all of them - even if the string appears more than once on the same line. (Which is why a simple grep -c won't cut it, as that only counts lines that contain the substring, no matter how many times each.)
Do you need to include hidden files (whose name begins with a period)? In my code below I assumed you don't.
Do you care that the count appears first, and then the file name?
OK, so here goes.
[oracle#localhost test]$ ls -al
total 20
drwxr-xr-x. 3 oracle oinstall 81 Apr 3 18:42 .
drwx------. 39 oracle oinstall 4096 Apr 3 18:42 ..
-rw-r--r--. 1 oracle oinstall 40 Apr 3 17:44 aa
lrwxrwxrwx. 1 oracle oinstall 2 Apr 3 18:04 bb -> aa
drwxr-xr-x. 2 oracle oinstall 6 Apr 3 17:40 d1
-rw-r--r--. 1 oracle oinstall 38 Apr 3 17:56 f1
-rw-r--r--. 1 oracle oinstall 0 Apr 3 17:56 f2
-rwxr-xr-x. 1 oracle oinstall 123 Apr 3 18:15 zfgrep
-rw-r--r--. 1 oracle oinstall 15 Apr 3 18:42 .zz
Here's the command to count 'waca' in the text files in this directory (not recursive). I define a variable substr to hold the desired string. (Note that it could also be a regular expression, more generally - but I didn't test that so you will have to, if that's your use case.)
[oracle#localhost test]$ substr=waca
[oracle#localhost test]$ find . -maxdepth 1 -type f \
> -exec grep -osHI "$substr" {} \; | sed "s/^\.\/\(.*\):$substr$/\1/" | uniq -c
8 aa
2 f1
1 .zz
Explanation: I use find to find just the files in the current directory (excluding directories, links, and whatever other trash I may have in the directory). This will include the hidden files, and it will include binary files, not just text. In this example I find in the current directory, but you can use any path instead of . I limit the depth to 1, so the command only applies to files in the current directory - the search is not recursive. Then I pass the results to grep. -o means find all matches (even if multiple matches per line of text) and show each match on a separate line. -s is for silent mode (just in case grep thinks of printing messages), -H is to include file names (even when there is only one file matching the substring), and -I is to ignore binary files.
Then I pass this to sed so that from each row output by grep I keep just the file name, without the leading ./ and without the trailing :waca. This step may not be necessary - if you don't mind the output like this:
8 ./aa:waca
2 ./f1:waca
1 ./.zz:waca
Then I pass the output to uniq -c to get the counts.
You can then redirect the output to a file, if that's what you need. (Left as a trivial exercise - since I forgot that was part of the requirement, sorry.)
Thanks for the detailed answer it provides me with ideas for future projects.
In my case the files were all the same format (output from another script) and the only files in the directory.
I found the answer in another thread
grep -c -R 'xxx'
I am trying to run the jmeter test from Jenkins. I've already installed performance plugin and restarted the jenkins. I don't want to use any maven/ant.
Execute shell command
cd /Users/Shared/Jenkins/Home/jobs/meineTui-QA-Test-Jmeter/workspace
java -jar /Users/Shared/Jenkins/apache-jmeter/bin/ApacheJMeter.jar -n -t Login_Logout.jmx -l result.jtl
In the post build actions of jenkin-> publish performance test result report -> jmeter -> report files -> **/*.jtl
While I am running from jenkin the console says
Performance: Failed to parse /Users/Shared/Jenkins/Home/jobs/meineTui-QA-Test-Jmeter/builds/2013-10-03_17-14-53/performance-reports/JMeter/result.jtl: Content is not allowed in prolog.
So I am not able view the result/report in the performance Report section. Any suggestion how to fix.
==================================console output=============
+ cd /Users/Shared/Jenkins/Home/jobs/meineTui-QA-Test-Jmeter/workspace
+ java -jar /Users/Shared/Jenkins/apache-jmeter/bin/ApacheJMeter.jar -n -t Login_Logout.jmx -l result.jtl
Creating summariser <summary>
Created the tree successfully using Login_Logout.jmx
Starting the test # Thu Oct 03 17:14:55 BST 2013 (1380816895721)
Waiting for possible shutdown message on port 4445
summary + 2 in 4.1s = 0.5/s Avg: 2013 Min: 766 Max: 3260 Err: 0 (0.00%) Active: 1 Started: 1 Finished: 0
summary + 10 in 4s = 2.5/s Avg: 392 Min: 286 Max: 573 Err: 0 (0.00%) Active: 0 Started: 1 Finished: 1
summary = 12 in 8s = 1.5/s Avg: 662 Min: 286 Max: 3260 Err: 0 (0.00%)
Tidying up ... # Thu Oct 03 17:15:04 BST 2013 (1380816904307)
... end of run
Performance: Percentage of errors greater or equal than 0% sets the build as unstable
Performance: Percentage of errors greater or equal than 0% sets the build as failure
Performance: Recording JMeter reports '**/*.jtl'
Performance: Parsing JMeter report file result.jtl
Performance: Failed to parse /Users/Shared/Jenkins/Home/jobs/meineTui-QA-Test-Jmeter/builds /2013-10-03_17-14-53/performance-reports/JMeter/result.jtl: Content is not allowed in prolog.
Finished: SUCCESS
result.jtl
1380816896268,766,Login,200,OK,Group1 1-1,text,true,230,766
1380816897071,3260,Reservations,200,OK,Group1 1-1,text,true,3295,3260
1380816900339,335,ReservationID,200,OK,Group1 1-1,text,true,8683,335
1380816900681,353,Weather,200,OK,Group1 1-1,text,true,2022,353
1380816901039,563,Summary,200,OK,Group1 1-1,text,true,6528,563
1380816901607,573,Home,200,OK,Group1 1-1,text,true,11955,573
1380816902187,329,HolidayCountdown,200,OK,Group1 1-1,text,true,344,329
1380816902520,375,Contacts,200,OK,Group1 1-1,text,true,2835,375
1380816902899,286,Excursions,200,OK,Group1 1-1,text,true,237,286
1380816903189,361,TravelAgent,200,OK,Group1 1-1,text,true,570,361
1380816903554,319,Profile,200,OK,Group1 1-1,text,true,395,319
make the following changes in the jmeter.properties file:
remove the comment from the following line and change csv by xml
#jmeter.save.saveservice.output_format=csv
like this:
jmeter.save.saveservice.output_format=xml
remove the (#) comment from the following lines:
jmeter.save.saveservice.data_type=true
jmeter.save.saveservice.label=true
jmeter.save.saveservice.response_code=true
jmeter.save.saveservice.successful=true
jmeter.save.saveservice.thread_name=true
And change the extension of the file to be generate jtl to xml.
With latest versions of Jenkins Performance plugin (e.g. v1.14) you can parse both CSV and XML formats.
Depending on the format of your result files, you need to select the appropriate report type in the "Publish performance tests result report" section:
chose "JMeter" report type if your result files are XML
chose "JMeterCSV" report type if your result files are CSV.