Extract IP address from a file - grep

I have a log file where between the lines we have strings like " ,true-client-ip=[1.1.1.1]". I want to write a grep/sed command to extract the IPs from the file. What is the best way to extract this output and write it to another file?
cache-control=[no-cache, max-age=0],origin=[https://www.example.com],pragma=[no-cache],te=[chunked;q=1.0],true-client-ip=[108.81.148.189],user-agent=[Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36 Edge/16.16299]

You can use expression:
([0-9]{1,3}\.){3}[0-9]{1,3}
You can write the output of the grep command to a new file as follows:
grep -Eo '([0-9]{1,3}\.){3}[0-9]{1,3}' yourlogfile.txt >> ipfile.txt
A second option would be to use the Perl dialect, and to use a lookbehind to extract the IP after the true-client-ip field.
grep -Po '(?<=true-client-ip=\[)[^\]]+' yourlogfile.txt >> ipfile.txt
ipfile.txt now contains:
108.81.148.189

Related

GitLab: Receive 500 Error when trying to Create Merge Requests

I run my own Private GitLab CE VM on my Server. I'm updated to the latest version of 14.1.1.
As the title says: when I (or anyone else) clicks the Create merge request, we receive the 500 - Whoops, something went wrong on our end. This isn't even a large Branch: about 6 files with a few lines of code each. This has only started happening a couple days ago.
Now I should mention that I have been having on-going issues with the 500 Error; this includes Merging and Committing as well. As soon as I believe I've fixed the issue, it rears its ugly head back.
Logs
Here are some logs when I try creating a Merge Request:
root#gitlab:~# sudo gitlab-ctl tail | grep error
==> /var/log/gitlab/nginx/error.log <==
==> /var/log/gitlab/nginx/gitlab_error.log <==
2021-08-02_01:32:01.06995 {"#level":"debug","#message":"datasource: registering query type handler","#timestamp":"2021-08-02T01:32:01.069922Z","queryType":"random_walk_with_error"}
2021-08-02_01:32:01.07003 {"#level":"debug","#message":"datasource: registering query type handler","#timestamp":"2021-08-02T01:32:01.069948Z","queryType":"server_error_500"}
{"method":"POST","path":"/team/project/-/merge_requests","format":"html","controller":"Projects::MergeRequests::CreationsController","action":"create","status":500,"time":"2021-08-02T18:09:19.919Z","params":[{"key":"authenticity_token","value":"[FILTERED]"},{"key":"merge_request","value":{"title":"[FILTERED]","description":"[FILTERED]","assignee_ids":["0"],"reviewer_ids":["0"],"label_ids":[""],"force_remove_source_branch":"0","squash":"0","lock_version":"0","source_project_id":"41","source_branch":"AscentCharacterAttributes","target_project_id":"41","target_branch":"master"}},{"key":"merge_request_diff_head_sha","value":"f9d4a01e30d74b8b843e7b10b2810795e5c8966b"},{"key":"namespace_id","value":"team"},{"key":"project_id","value":"project"}],"remote_ip":"23.82.194.104","user_id":4,"username":"names_are_useless","ua":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:90.0) Gecko/20100101 Firefox/90.0","correlation_id":"01FC40SVDY9T1DQ0YTY1FMN7NM","meta.user":"names_are_useless","meta.project":"team/project","meta.root_namespace":"team","meta.caller_id":"Projects::MergeRequests::CreationsController#create","meta.remote_ip":"23.82.194.104","meta.feature_category":"code_review","meta.client_id":"user/4","gitaly_calls":8,"gitaly_duration_s":87.627672,"redis_calls":15,"redis_duration_s":0.00976,"redis_read_bytes":2886,"redis_write_bytes":2254,"redis_cache_calls":14,"redis_cache_duration_s":0.007988,"redis_cache_read_bytes":2705,"redis_cache_write_bytes":859,"redis_shared_state_calls":1,"redis_shared_state_duration_s":0.001772,"redis_shared_state_read_bytes":181,"redis_shared_state_write_bytes":1395,"db_count":31,"db_write_count":9,"db_cached_count":4,"cpu_s":2.259331,"mem_objects":246072,"mem_bytes":63989931,"mem_mallocs":58706,"mem_total_bytes":73832811,"exception.class":"ActiveRecord::StatementInvalid","exception.message":"PG::UnableToSend: no connection to the server\n","exception.backtrace":["app/services/issuable_base_service.rb:209:in `create'","app/services/merge_requests/base_service.rb:105:in `create'","app/services/merge_requests/create_service.rb:29:in `create'","app/services/merge_requests/create_service.rb:13:in `execute'","app/controllers/projects/merge_requests/creations_controller.rb:22:in `create'","app/controllers/application_controller.rb:483:in `set_current_admin'","lib/gitlab/session.rb:11:in `with_session'","app/controllers/application_controller.rb:474:in `set_session_storage'","lib/gitlab/i18n.rb:99:in `with_locale'","lib/gitlab/i18n.rb:105:in `with_user_locale'","app/controllers/application_controller.rb:468:in `set_locale'","app/controllers/application_controller.rb:462:in `set_current_context'","lib/gitlab/middleware/speedscope.rb:13:in `call'","lib/gitlab/request_profiler/middleware.rb:17:in `call'","lib/gitlab/jira/middleware.rb:19:in `call'","lib/gitlab/middleware/go.rb:20:in `call'","lib/gitlab/etag_caching/middleware.rb:21:in `call'","lib/gitlab/middleware/multipart.rb:172:in `call'","lib/gitlab/middleware/read_only/controller.rb:50:in `call'","lib/gitlab/middleware/read_only.rb:18:in `call'","lib/gitlab/middleware/same_site_cookies.rb:27:in `call'","lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'","lib/gitlab/middleware/basic_health_check.rb:25:in `call'","lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'","lib/gitlab/middleware/request_context.rb:21:in `call'","config/initializers/fix_local_cache_middleware.rb:11:in `call'","lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:19:in `call'","lib/gitlab/metrics/requests_rack_middleware.rb:74:in `call'","lib/gitlab/middleware/release_env.rb:12:in `call'"],"db_duration_s":0.20229,"view_duration_s":0.0,"duration_s":89.59493}
lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'
{"args":["/opt/gitlab/embedded/bin/git","--git-dir","/var/opt/gitlab/git-data/repositories/#hashed/3d/91/3d914f9348c9cc0ff8a79716700b9fcd4d2f3e711608004eb8f138bcba7f14d9.git","-c","core.fsyncObjectFiles=true","-c","gc.auto=0","-c","core.autocrlf=input","log","--pretty=%H","--reverse","--max-count","2147483647","master..0000000000000000000000000000000000000000","--end-of-options"],"command.exitCode":128,"command.inblock":0,"command.maxrss":63964,"command.oublock":0,"command.real_time_ms":6.851257,"command.system_time_ms":6.352,"command.user_time_ms":0,"correlation_id":"01FC40WYVZ9BBNJHH5PYVE4NWM","grpc.meta.auth_version":"v2","grpc.meta.client_name":"gitlab-sidekiq","grpc.meta.deadline_type":"unknown","grpc.method":"CommitsBetween","grpc.request.deadline":"2021-08-02T18:13:20.661Z","grpc.request.fullMethod":"/gitaly.CommitService/CommitsBetween","grpc.request.glProjectPath":"team/project","grpc.request.glRepository":"project-41","grpc.request.repoPath":"#hashed/3d/91/3d914f9348c9cc0ff8a79716700b9fcd4d2f3e711608004eb8f138bcba7f14d9.git","grpc.request.repoStorage":"default","grpc.service":"gitaly.CommitService","grpc.start_time":"2021-08-02T18:09:32.661Z","level":"error","msg":"fatal: Invalid revision range master..0000000000000000000000000000000000000000\n","path":"/opt/gitlab/embedded/bin/git","peer.address":"#","pid":900,"remote_ip":"127.0.0.1","span.kind":"server","system":"grpc","time":"2021-08-02T18:09:32.669Z","username":"NicholuasP"}
{"correlation_id":"01FC40WYVZ9BBNJHH5PYVE4NWM","error":"exit status 128","grpc.meta.auth_version":"v2","grpc.meta.client_name":"gitlab-sidekiq","grpc.meta.deadline_type":"unknown","grpc.method":"CommitsBetween","grpc.request.deadline":"2021-08-02T18:13:20.661Z","grpc.request.fullMethod":"/gitaly.CommitService/CommitsBetween","grpc.request.glProjectPath":"team/project","grpc.request.glRepository":"project-41","grpc.request.repoPath":"#hashed/3d/91/3d914f9348c9cc0ff8a79716700b9fcd4d2f3e711608004eb8f138bcba7f14d9.git","grpc.request.repoStorage":"default","grpc.service":"gitaly.CommitService","grpc.start_time":"2021-08-02T18:09:32.661Z","level":"info","msg":"ignoring git-log error","peer.address":"#","pid":1430,"remote_ip":"127.0.0.1","span.kind":"server","system":"grpc","time":"2021-08-02T18:09:32.669Z","username":"NicholuasP"}
{"severity":"WARN","time":"2021-08-02T18:09:34.054Z","class":"Git::BranchHooksService","correlation_id":"01FC40WYVZ9BBNJHH5PYVE4NWM","project_id":41,"project_path":"team/project","message":"Error creating pipeline","errors":"Reference not found","pipeline_params":{"before":"f9d4a01e30d74b8b843e7b10b2810795e5c8966b","after":"0000000000000000000000000000000000000000","ref":"refs/heads/AscentCharacterAttributes","variables_attributes":[],"checkout_sha":null},"retry":0}
root#gitlab:~# sudo gitlab-ctl tail nginx/gitlab_error.log
(Nothing)
root#gitlab:~# tail -f /var/log/gitlab/gitlab-rails/production.log | grep error
lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'
root#gitlab:~# tail -f /var/log/gitlab/nginx/error.log
(Nothing)
root#gitlab:~# tail -f /var/log/gitlab/nginx/gitlab_error.log
(Nothing)
And because the following logs are too long, I have pasted the outputs to PasteBin links:
tail -f /var/log/gitlab/gitlab-rails/production.log
tail -f /var/log/gitlab/gitlab-rails/production_json.log
tail -f /var/log/gitlab/gitlab-rails/api_json.log
And in case that isn't enough, I've tried to log as much of gitlab-ctl tail right when I click the Create merge request button: sudo gitlab-ctl tail
Obviously I gave generic names for the Team, Project, IPs and our Website.
Potential Root of the Problem
The most important piece, I think, looks to be in production.log:
==> /var/log/gitlab/gitlab-rails/production_json.log <==
{"method":"POST","path":"/team/project/-/merge_requests","format":"html","controller":"Projects::MergeRequests::CreationsController","action":"create","status":500,"time":"2021-08-03T16:56:23.192Z","params":[{"key":"authenticity_token","value":"[FILTERED]"},{"key":"merge_request","value":{"title":"[FILTERED]","description":"[FILTERED]","assignee_ids":["0"],"reviewer_ids":["0"],"label_ids":[""],"force_remove_source_branch":"0","squash":"0","lock_version":"0","source_project_id":"41","source_branch":"AscentCharacterAttributes","target_project_id":"41","target_branch":"master"}},{"key":"merge_request_diff_head_sha","value":"f9d4a01e30d74b8b843e7b10b2810795e5c8966b"},{"key":"namespace_id","value":"team"},{"key":"project_id","value":"project"}],"remote_ip":"1.2.3.4","user_id":4,"username":"names_are_useless","ua":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:88.0) Gecko/20100101 Firefox/88.0","correlation_id":"01FC6F15DTJ7VQD0WE820Z2ZW0","meta.user":"names_are_useless","meta.project":"team/project","meta.root_namespace":"team","meta.caller_id":"Projects::MergeRequests::CreationsController#create","meta.remote_ip":"1.2.3.4","meta.feature_category":"code_review","meta.client_id":"user/4","gitaly_calls":8,"gitaly_duration_s":82.237132,"redis_calls":13,"redis_duration_s":0.00787,"redis_read_bytes":2516,"redis_write_bytes":1470,"redis_cache_calls":12,"redis_cache_duration_s":0.007299,"redis_cache_read_bytes":2335,"redis_cache_write_bytes":722,"redis_shared_state_calls":1,"redis_shared_state_duration_s":0.000571,"redis_shared_state_read_bytes":181,"redis_shared_state_write_bytes":748,"db_count":29,"db_write_count":9,"db_cached_count":4,"cpu_s":2.260525,"mem_objects":250013,"mem_bytes":67239344,"mem_mallocs":75906,"mem_total_bytes":77239864,"exception.class":"ActiveRecord::StatementInvalid","exception.message":"PG::UnableToSend: no connection to the server\n","exception.backtrace":["app/services/issuable_base_service.rb:209:in `create'","app/services/merge_requests/base_service.rb:105:in `create'","app/services/merge_requests/create_service.rb:29:in `create'","app/services/merge_requests/create_service.rb:13:in `execute'","app/controllers/projects/merge_requests/creations_controller.rb:22:in `create'","app/controllers/application_controller.rb:483:in `set_current_admin'","lib/gitlab/session.rb:11:in `with_session'","app/controllers/application_controller.rb:474:in `set_session_storage'","lib/gitlab/i18n.rb:99:in `with_locale'","lib/gitlab/i18n.rb:105:in `with_user_locale'","app/controllers/application_controller.rb:468:in `set_locale'","app/controllers/application_controller.rb:462:in `set_current_context'","lib/gitlab/middleware/speedscope.rb:13:in `call'","lib/gitlab/request_profiler/middleware.rb:17:in `call'","lib/gitlab/jira/middleware.rb:19:in `call'","lib/gitlab/middleware/go.rb:20:in `call'","lib/gitlab/etag_caching/middleware.rb:21:in `call'","lib/gitlab/middleware/multipart.rb:172:in `call'","lib/gitlab/middleware/read_only/controller.rb:50:in `call'","lib/gitlab/middleware/read_only.rb:18:in `call'","lib/gitlab/middleware/same_site_cookies.rb:27:in `call'","lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'","lib/gitlab/middleware/basic_health_check.rb:25:in `call'","lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'","lib/gitlab/middleware/request_context.rb:21:in `call'","config/initializers/fix_local_cache_middleware.rb:11:in `call'","lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:19:in `call'","lib/gitlab/metrics/requests_rack_middleware.rb:74:in `call'","lib/gitlab/middleware/release_env.rb:12:in `call'"],"db_duration_s":0.28854,"view_duration_s":0.0,"duration_s":84.33137}
==> /var/log/gitlab/gitlab-rails/production.log <==
Completed 500 Internal Server Error in 85422ms (ActiveRecord: 164.6ms | Elasticsearch: 0.0ms | Allocations: 232093)
ActiveRecord::StatementInvalid (PG::UnableToSend: no connection to the server):
What causes the ActiveRecord::StatementInvalid (PG::UnableToSend: no connection to the server ): I have no earthly idea, but I current suspect is the root of the problem.
Any ideas what is going on? And if so, how to fix GitLab so I can create Merge Requests without receiving this error?

Sidekiq + Rails 6.0.0 in API mode : WARN -- : attack prevented by Rack::Protection::AuthenticityToken

I'm currently trying to setup a standalone rails 6.0.0 API fresh install (rails new xxx --api) + sidekiq (that includes Rack::Protection)
After setting-up Rack::Cors, i get the following 403 error on any non-GET request :
WARN -- : attack prevented by Rack::Protection::AuthenticityToken
To my understanding this is related to CSRF tokens, which don't make sense for a json API.
Here is a copy of the request I'm doing :
curl 'http://127.0.0.1:5100/searches' -H 'Accept: application/json,
text/plain, /' -H 'Referer: http://localhost:8080/' -H 'Origin:
http://localhost:8080' -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel
Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko)
Chrome/76.0.3809.132 Safari/537.36' -H 'Sec-Fetch-Mode: cors' -H
'Content-Type: application/x-www-form-urlencoded' --data '{query:
'test'}' --compressed
The 2 possible solutions that I cannot solve :
disable rack-protection
get a token in a controller (ActionController::API) to pass to the requests
In ApplicationController try protect_from_forgery with: :null_session, or
Rails.configuration.middleware.insert_before 0, Rack::Cors do
allow do
origins "*"
end
end
in config/initializers/rack-cors.rb

Getting/setting session with HTTParty

Using HTTParty, I'm able to read some pages fine, other on the same site strangely give a 404 unless I set the proper session headers. So I'm trying to get and set them via HTTParty.
This works:
HTTParty.get 'https://www.instagram.com/explore/locations/24993086/pfriem-family-brewers/'
This gives a 404:
HTTParty.get 'https://www.instagram.com/explore/locations/295648950/trio-salon-spa/'
Curl also gives a 404 for that:
url -I https://www.instagram.com/explore/locations/295648950/trio-salon-spa/
...unless I set all the headers that Chrome sends:
curl -I 'https://www.instagram.com/explore/locations/295648950/trio-salon-spa/' -H 'pragma: no-cache' -H 'accept-encoding: gzip, deflate, br' -H 'accept-language: en-US,en;q=0.9,da;q=0.8,fr;q=0.7' -H 'upgrade-insecure-requests: 1' -H 'user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Safari/537.36' -H 'accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8' -H 'cache-control: no-cache' -H 'authority: www.instagram.com' -H 'cookie: shbid=15682; csrftoken=7FQ5Z0SZikfkYiS02bcTRotjAYEvdooD; mid=Wk7bTgAEAAHaOKTW39cyGMXo8vLj; ds_user_id=2055054912; mcd=3; sessionid=IGSC07d4796576bd227dda2a5353ee0365bbd6a6f6b2da7567d57a83cde58c0ae870%3AOvd5knFMpZUQaZpRyr0QkeMitWBFnGDP%3A%7B%22_auth_user_id%22%3A2055054912%2C%22_auth_user_backend%22%3A%22accounts.backends.CaseInsensitiveModelBackend%22%2C%22_auth_user_hash%22%3A%22%22%2C%22_platform%22%3A4%2C%22_token_ver%22%3A2%2C%22_token%22%3A%222055054912%3AhdEI59s33u2BM3M2f8p2ZkSkZ9HeZR5Y%3A7359e774fd121f9726db15a24e660d43a0f464b4733bbbaa31f619aec3f433ba%22%2C%22last_refreshed%22%3A1524728608.4418663979%7D; rur=FTW; fbm_124024574287414=base_domain=.instagram.com; fbsr_124024574287414=9pBDQeojfCbPmhlXZHwx_OGhduHFlQusvBdewwiZDY4.eyJhbGdvcml0aG0iOiJITUFDLVNIQTI1NiIsImNvZGUiOiJBUUJVTEZmWHBfYS1mdVYyMmhKZEhnV3Bvb3dKVXppV3oxVVhxcDBGelFnTmpnNmpRZjNDY1ZZV2xybWZYSE5JV0dxeVUtNE9UaDBFUUdMbFJFWVVwZzZPYngxYmdxbUxHLV9pVGl5U3hGa1JxbGRJRExITHV5V09WVEVsbjlHUFhsTmRCQUNheUdYMmh4ZWcxajJEcERwczZ2X282aWRUcWd6UmYxaHE2VjVObEFGX0w5bFM1RXI5bHg3b2c1bWk0ak9OdmVCcVpLTG5nY1llM0NnWHVtOWdWQWJTVi1SdWpQU2J1UmhHNFdaS0xweWtPVEYzdmhsUlVaT2FLZ3FXcFo2TXFXY2xqSTM0T3JWZjR3dzFyY3J0S0RkdVh0Qk5zellKRG9weU1IdG1kVERUVGVrZmwyUHhpRzZsSmRkUVpSbWc0MTNMWERHQ0ZORDRVRS13OUhvMyIsImlzc3VlZF9hdCI6MTUyNDc1ODM3NCwidXNlcl9pZCI6IjExNTU2MzM2MjkifQ; urlgen="{\"time\": 1524728608\054 \"65.157.26.82\": 209}:1fBjJI:P27GLkp5R5uijWSi-SEkHW-Mo0c"' --compressed
So, I'm trying to do a basic get with HTTP, read the session, and make another request with it, but it's not working properly:
require 'httparty'
url = 'https://www.instagram.com/explore/locations/295648950/trio-salon-spa/'
get_response = HTTParty.get(url)
cookie_hash = CookieHash.new
get_response.get_fields('Set-Cookie').each { |c| cookie_hash.add_cookies(c) }
get_response_cookie = parse_cookie(get_response.headers['Set-Cookie'])
post_response = HTTParty.get( url, headers: {'Cookie' => cookie_hash.to_cookie_string } )
How can I get this to work?

curl returns blank from valid url

I am trying to lookup a zipcode from usps using curl like so:
curl "https://tools.usps.com/go/ZipLookupResultsAction!input.action?resultMode=2&companyName=&address1=&address2=&city=&state=Select&urbanCode=&postalCode=11111&zip="
This url does return normal output from browser.
One needs to specify the User-Agent header (using the -A option) so that the request resembles a request from a browser. Also, due to the presence of ! in the URL, one has to use ' instead of ":
curl -A "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:48.0) Gecko/20100101 Firefox/48.0" \
'https://tools.usps.com/go/ZipLookupResultsAction!input.action?resultMode=2&companyName=&address1=&address2=&city=&state=Select&urbanCode=&postalCode=10001&zip='

How to Post Payload to URL using cURL

Apologies for my ignorance here.
I have the following payload that I need to send to a URL via cURL (Mac Bash).
{“requestid":"2323423432",
"partnermatchid":"56d576ee-2d74-4dda-b8ff-d71b34311dd2",
"usercontext":{"ipaddressmasked":"209.252.7.186",
"useragent":"mozilla/5.0 (windows nt 6.1; wow64) applewebkit/537.36 (khtml, like
gecko) chrome/28.0.1500.95 safari/537.36",
"country":"us"},
"pagecontext":{"pagetypeid":"3","numslots":"6"},
"istest":false}
Can you help with the command syntax to do that? I tried this, but it's generating weird HTML in Bash and I don't understand what is going on.
echo ‘{“requestid":"2323423432",
"partnermatchid":"56d576ee-2d74-4dda-b8ff-d71b34311dd2",
"usercontext":{"ipaddressmasked":"209.252.7.186",
"useragent":"mozilla/5.0 (windows nt 6.1; wow64) applewebkit/537.36 (khtml, like
gecko) chrome/28.0.1500.95 safari/537.36",
"country":"us"},
"pagecontext":{"pagetypeid":"3","numslots":"6"},
"istest":false}’ | curl —data-binary #- http://www.thisistheurl.com
Thanks so much!!
Since you're POSTing JSON data, you need to specify the content type of the request as well.
Try:
curl -d '{ "requestid": "2323423432", "partnermatchid": "56d576ee-2d74-4dda-b8ff-d71b34311dd2", "usercontext": {"ipaddressmasked": "209.252.7.186","useragent": "mozilla/5.0 (windows nt 6.1; wow64) applewebkit/537.36 (khtml, like gecko) chrome / 28.0 .1500 .95 safari / 537.36", "country": "us" }, "pagecontext": { "pagetypeid": "3", "numslots": "6" }, "istest": false }' \
-H "Content-Type: application/json" http://url
Also in your original post, there are some "fancy quotes" which should be replaced with normal quotes, in case that's what you're trying to use on the command line.

Resources