I'm using Fedora 14 and httpd timeout a lot.
Is there a log or something that tell me how many connnections to httpd and mysqld every second/minutes...etc
I'm very new to linux, please help me :)
You can view the Access Log in httpd to see requests:
LogFormat "%h %l %u %t \"%r\" %>s %b" common
CustomLog logs/access_log common
May be different for your version:
Access Log
Or if the established session is timing out too early, set KeepAlive and KeepAliveTimeout:
KeepAlive
KeepAliveTimeout
Related
I need to run several docker containers running apache.
To centralize the log files in single location, I want to use the hostname of the docker container where apache is running (not the virtual server name) as a part of the log file name, i.e. I need something like /var/log/apache2/${APP_NAME}.access.${HOSTNAME}.log
I prepared a virtual host config like this:
<VirtualHost *:80>
...
...
...
ErrorLog /var/log/apache2/${APP_NAME}.error.log
LogLevel warn
LogFormat "%V - %{CLIENTIP}e %l %u [%{%d/%b/%Y %T}t.%{msec_frac}t %{%z}t] \"%r\" %>s %b %D \"%{Referer}i\" \"%{User-agent}i\"" extended
CustomLog /var/log/apache2/${APP_NAME}.access.${HOSTNAME}.log extended
</VirtualHost>
where $HOSTNAME and $APP_NAME are environment variables passed by the shell where apache is running.
This configuration seam to work, in the sense that when apache is running in docker container named ad331fa1 it creates a file named like /var/log/apache2/myappname.access.ad331fa1.log
But.... the timetaken to handle the request increased by 4-5 seconds per request!!
This happens only for requests that are handled by the php engine, while no delay is added when serving static files (.img, .css, etc...)
The problem disappears when using a config like this:
<VirtualHost *:80>
...
...
...
ErrorLog /var/log/apache2/${APP_NAME}.error.${HOSTNAME}.log
LogLevel warn
LogFormat "%V - %{CLIENTIP}e %l %u [%{%d/%b/%Y %T}t.%{msec_frac}t %{%z}t] \"%r\" %>s %b %D \"%{Referer}i\" \"%{User-agent}i\"" extended
CustomLog /var/log/apache2/${APP_NAME}.access.log extended
</VirtualHost
i.e. removing the $HOSTNAME solve the issue.
Note that both $APP_NAME and $HOSTNAME are environment variables, but the issue happens only when $HOSTNAME is used in the log file name.
Any suggestion?
I have here a RoR app, what I am using with the thin appserver.
Its configuration is in an .yml file, so:
---
pid: /srv/cica/tmp/pids/thin.pid
group: cica
wait: 30
timeout: 30
log: /srv/cica/log/thin.log
max_conns: 1024
require: []
environment: production
max_persistent_conns: 512
servers: 4
daemonize: true
user: cica
socket: /srv/cica/tmp/thin.sock
chdir: /srv/cica
How could I use a TCP socket instead of a unix socket for listening?
The documentation I've found somehow never mentions even the possibility, although indirect references say it is possible.
The cause of the problem is that the frontend web (apache2) isn't very strong to proxying http requests to a unix path. It wouldn't be a problem with nginx.
In theory, you can use simply an IP:ADDR instead of the socket path:
socket: 127.0.0.1:3000
will work. But, if you use multiple thin processes, you will have a problem.
(Which is very likely, because the whole ruby is a singlethreaded thing. Considering the IO waiting times, maybe even a significantly higher process number is also possible as the number of your CPU cores).
Somehow the socket address decoder of the thin configuration interpreter is enough smart to use the ordinary IP address, but it increases the IP and not the port for the additional sockets. Thus, you will have multiple thin instances listening on
# thin will listen on these addresses
127.0.0.1:3000
127.0.0.2:3000
127.0.0.3:3000
127.0.0.4:3000
rather they would be listening on
# it would be okay, but not this happens
127.0.0.1:3000
127.0.0.1:3001
127.0.0.1:3002
127.0.0.1:3003
This surreal behavior is likely not what you want. (Although if you have active interfaces on all of the IPs, it could work.)
However, this ruby thing has the nice feature that there is a direct assignment between its command line options and configuration file options. And a thin --help command will show them to you. You can enforce a TCP listening using the address and port options:
#socket: /srv/cica/tmp/thin.sock
address: 127.0.0.1
port: 3000
So you will get already the correct result.
The default values are 0.0.0.0 and 3000.
As apache wants to proxy only to a single tcp port with its most common settings (ProxyPass, ProxyPassReverse directives), also there you need some little trickery, a load balancing proxy cluster. The relevant config snippet:
<Proxy balancer://cicas>
BalancerMember http://localhost:3000 disablereuse=On route=cica1
BalancerMember http://localhost:3001 disablereuse=On route=cica2
BalancerMember http://localhost:3002 disablereuse=On route=cica3
BalancerMember http://localhost:3003 disablereuse=On route=cica4
ProxySet lbmethod=byrequests
</Proxy>
ProxyPass / balancer://cicas/
I have a Rails application deployed on Apache-Passenger which runs fine when access from localhost, but doesn't run via remote access.
Let's say the server name is server.name.com. The server info is -
[kbc#server KBC]$ uname -a
Linux server.name.com 2.6.32-431.el6.x86_64 #1 SMP Fri Nov 22 03:15:09 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
[kbc#server KBC]$ cat /etc/issue
CentOS release 6.5 (Final)
Kernel \r on an \m
When I do
[kbc#server ]$ curl http://localhost:3000/, it returns the home page for the application.
But when I try to access the Rails app from my laptop, I get the following error -
→ curl http://server.name.com:3000/
curl: (7) Failed to connect to server.name.com port 3000: Connection refused
To check if I can access the server, I tried -
→ ping server.name.com:3000
ping: cannot resolve server.name.com:3000: Unknown host
But, I can ping the server by -
→ ping server.name.com
PING server.name.com (#.#.#.#): 56 data bytes
64 bytes from #.#.#.#: icmp_seq=0 ttl=61 time=1.526 ms
64 bytes from #.#.#.#: icmp_seq=1 ttl=61 time=6.624 ms
Here is the Passenger configuration -
<VirtualHost *:3000>
ServerName server.name.com
ServerAlias server.name.com
DocumentRoot /home/kbc/KBC/public
<Directory /home/kbc/KBC/public>
AllowOverride all
Options -MultiViews
</Directory>
ErrorLog /var/log/httpd/kbc_error.log
CustomLog /var/log/httpd/kbc_access.log common
</VirtualHost>
NameVirtualHost *:3000
PassengerPreStart https://server.name.com:3000/
and
LoadModule passenger_module /home/kbc/.rvm/gems/ruby-2.3.0#kbc/gems/passenger-5.0.30/buildout/apache2/mod_passenger.so
<IfModule mod_passenger.c>
PassengerRoot /home/kbc/.rvm/gems/ruby-2.3.0#kbc/gems/passenger-5.0.30
PassengerDefaultRuby /home/kbc/.rvm/wrappers/ruby-2.3.0/ruby
PassengerRuby /home/kbc/.rvm/wrappers/ruby-2.3.0/ruby
PassengerMaxPoolSize 5
PassengerPoolIdleTime 90
PassengerMaxRequests 10000
</IfModule>
Passenger-status info -
[kbc#server ]$ passenger-status
Version : 5.0.30
Date : 2016-10-17 11:30:08 -0400
Instance: bKUJ0ptp (Apache/2.2.15 (Unix) DAV/2 Phusion_Passenger/5.0.30)
----------- General information -----------
Max pool size : 5
App groups : 1
Processes : 1
Requests in top-level queue : 0
----------- Application groups -----------
/home/kbc/KBC:
App root: /home/kbc/KBC
Requests in queue: 0
* PID: 5696 Sessions: 0 Processed: 1 Uptime: 1m 45s
CPU: 0% Memory : 38M Last used: 1m 45s ago
What am I doing wrong? Please let me know if you need more information.
This sounds like a connectivity problem, not a Passenger/Apache problem. The host you're running the server on may not accept inbound connections on port 3000 (due to iptables, firewall, or security group access control rules).
Take a look at apache not accepting incoming connections from outside of localhost and Apache VirtualHost and localhost, for instance.
#Jatin, could you please post the apache main configuration ? (/etc/apache2/apache2.conf or similar)
Also, please provide the output of the following :
sudo netstat -nl
sudo iptables -L
Just for the record, the ping utility can only test connectivity at the IP layer, meaning that it can tell you whether the host at a given IP is responding. It cannot, however, tell you if a specific TCP port is open on the remote system.
Testing TCP connectivity can be achieved easily with telnet or netcat :
telnet server.name.com 3000
If you get something like :
Trying #.#.#.#...
Connected to server.name.com.
Escape character is '^]'.
then this means you can correctly access the TCP endpoint, eliminating any possibility of network-related issues. In other words, if this works, you probably have a configuration problem with Apache.
When I restart my gitlab server with sudo gitlab-ctl restart (Debian Jessie), while the server is loading, I get the 502 error, instead of the "Deploying" page. Then everything works fine. I don't understand why this happens.
Please be aware that this appears only when loading, then everything works fine. I'm not satisfied because I used to get the better "Deploying" page.
This started happening after I configured my gitlab to work through gitlab-workhorse to go through my apache server proxy (prior to that I had huge problems with web interface communication. Snapshots downloads didn't work, but now everything is fine). Before fixing that, I used to proxy gitlab's nginx.
The configuration I currently use for apache is the following, which complies to the standard configuration of Gitlab:
<VirtualHost *:443>
ServerName git.example.com
ServerSignature Off
ProxyPreserveHost On
# Ensure that encoded slashes are not decoded but left in their encoded state.
# http://doc.gitlab.com/ce/api/projects.html#get-single-project
AllowEncodedSlashes NoDecode
<Location />
# New authorization commands for apache 2.4 and up
# http://httpd.apache.org/docs/2.4/upgrading.html#access
Require all granted
#Allow forwarding to gitlab-workhorse
ProxyPassReverse http://127.0.0.1:8283
ProxyPassReverse https://git.example.com/
</Location>
RewriteEngine on
#Don't escape encoded characters in api requests
RewriteCond %{REQUEST_URI} ^/api/v3/.*
RewriteRule .* http://127.0.0.1:8283%{REQUEST_URI} [P,QSA,NE]
#Forward all requests to gitlab-workhorse except existing files like error documents
RewriteCond %{DOCUMENT_ROOT}/%{REQUEST_FILENAME} !-f [OR]
RewriteCond %{REQUEST_URI} ^/uploads/.*
RewriteRule .* http://127.0.0.1:8283%{REQUEST_URI} [P,QSA]
# needed for downloading attachments
DocumentRoot /opt/gitlab/embedded/service/gitlab-rails/public
#Set up apache error documents, if back end goes down (i.e. 503 error) then a maintenance/deploy page is thrown up.
ErrorDocument 404 /404.html
ErrorDocument 422 /422.html
ErrorDocument 500 /500.html
ErrorDocument 503 /deploy.html
#... ssl config
RequestHeader set X_FORWARDED_PROTO 'https'
RequestHeader set X-Forwarded-Ssl on
LogFormat "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b" common_forwarded
ErrorLog /home/myuser/logs/gitlab_error.log
CustomLog /home/myuser/logs/gitlab_forwarded.log common_forwarded
CustomLog /home/myuser/logs/gitlab_access.log combined env=!dontlog
CustomLog /home/myuser/logs/gitlab.log combined
</VirtualHost>
The only configuration I changed, is my /etc/gitlab/gitlab.rb, which I did for the proxy to work.
external_url 'https://git.example.com'
nginx['enable'] = false
web_server['external_users'] = ['www-data']
gitlab_workhorse['listen_network'] = "tcp"
gitlab_workhorse['listen_addr'] = "127.0.0.1:8283"
What is it that I'm doing wrong to get that 502 error instead of the nice "Deploying" page that I used to get before doing these changes?
If you require any additional information, please ask.
Usually I see this page after gitlab-ctl start.
My Solution is
sudo gitlab-ctl start
sudo gitlab-ctl reconfigure
refresh browser :)
Well, technically this is the way it should work, it's a 502 error after all. If you want to show the deploy page there is a manual way to set it up before each restart:
gitlab-ctl deploy-page up
Wait a minute and then again:
gitlab-ctl deploy-page down
Otherwise you can add to your Apache config:
ErrorDocument 502 /deploy.html
which will show the deploy page every time a 502 error occurs.
config server firewall to allow http and https:
sudo ufw enable
sudo ufw allow http
sudo ufw allow https
sudo ufw allow OpenSSH
sudo ufw status
Output
Status: active
To Action From
-- ------ ----
OpenSSH ALLOW Anywhere
80/tcp ALLOW Anywhere
443/tcp ALLOW Anywhere
OpenSSH (v6) ALLOW Anywhere (v6)
80/tcp (v6) ALLOW Anywhere (v6)
443/tcp (v6) ALLOW Anywhere (v6)
Background
Ubuntu 15.10
Apache/2.4.10 (Ubuntu)
Module dependencies are all enabled :
mod_rewrite
mod_proxy
mod_proxy_http
GitLab Community Edition 8.0.3
GitLab docroot : /opt/gitlab/embedded/service/gitlab-rails/public
I install GitLab this way : https://about.gitlab.com/downloads/#ubuntu1404
Problem
On my server I have a website running with apache2 (exemple-site.com).
I have GitLab running with the bundled nginx (exemple-gitlab.com)
I have 1 server, 1 IP, and multiple FQDN.
Like this, all my domain names are pointing to GitLab.
So exemple-gitlab.com point to GitLab as wanted but exemple-site.com point to GitLab too, and all others FQDN too.
Ways of solution
I think I have to (and I tried) to :
Disable the bundled nginx and configure gitlab with apache2 (hard to do for me)
Configure the bundled nginx as a reverse proxy of apache2 (hard to do for me)
MAJ : in fact the problem is Apache and bundled-nginx run on the same IP with the same port (80). And I don't want to run a website on port 81 or whatever, only port 80.
I prefer to use apache2 for all my PHP websites and I don't mind if gitlab use apache2 or bundled nginx, all I want is using all my FQDN for each of my websites and not all my FQDN redirectinf to gitlab.
Understanding
I don't understand how omnibus or rails or reverse proxy work.
I tried disabling bundled nginx in /etc/gitlab/gitlab.rb
nginx['enable'] = false
# For GitLab CI, use the following:
ci_nginx['enable'] = false
add www-data to gitlab-www group and modify :
web_server['external_users'] = ['www-data']
and adding a modified vhost.conf to apache2
I take it from https://gitlab.com/gitlab-org/gitlab-recipes/blob/master/web-server/apache/gitlab-apache24.conf
<VirtualHost *:80>
ServerName exemple-gitlab.com
ServerSignature Off
ProxyPreserveHost On
AllowEncodedSlashes NoDecode
<Location />
Require all granted
#Allow forwarding to gitlab-git-http-server
ProxyPassReverse http://127.0.0.1:8181
#Allow forwarding to GitLab Rails app (Unicorn)
ProxyPassReverse http://127.0.0.1:8080
ProxyPassReverse http://exemple-gitlab.com/
</Location>
#apache equivalent of nginx try files
RewriteEngine on
#Forward these requests to gitlab-git-http-server
RewriteCond %{REQUEST_URI} ^/[\w\.-]+/[\w\.-]+/repository/archive.* [OR]
RewriteCond %{REQUEST_URI} ^/api/v3/projects/.*/repository/archive.* [OR]
RewriteCond %{REQUEST_URI} ^/[\w\.-]+/[\w\.-]+/(info/refs|git-upload-pack|git-receive-pack)$
RewriteRule .* http://127.0.0.1:8181%{REQUEST_URI} [P,QSA]
#Forward any other requests to GitLab Rails app (Unicorn)
RewriteCond %{DOCUMENT_ROOT}/%{REQUEST_FILENAME} !-f [OR]
RewriteCond %{REQUEST_URI} ^/uploads
RewriteRule .* http://127.0.0.1:8080%{REQUEST_URI} [P,QSA,NE]
# needed for downloading attachments
/opt/gitlab/embedded/service/gitlab-rails/public
#Set up apache error documents, if back end goes down (i.e. 503 error) then a maintenance/deploy page is thrown up.
ErrorDocument 404 /404.html
ErrorDocument 422 /422.html
ErrorDocument 500 /500.html
ErrorDocument 503 /deploy.html
# /var/log/apache2.
LogFormat "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b" common_forwarded
ErrorLog /var/log/apache2/logs/gitlab.example.com_error.log
CustomLog /var/log/apache2/logs/gitlab.example.com_forwarded.log common_forwarded
CustomLog /var/log/apache2/logs/gitlab.example.com_access.log combined env=!dontlog
CustomLog /var/log/apache2/logs/gitlab.example.com.log combined
</VirtualHost>
But this conf bug my apache2 :
~# systemctl status apache2.service
● apache2.service - LSB: Apache2 web server
Loaded: loaded (/etc/init.d/apache2)
Active: failed (Result: exit-code) since mar. 2015-11-10 15:41:08 CET; 1min 9s ago
Docs: man:systemd-sysv-generator(8)
Process: 18315 ExecStop=/etc/init.d/apache2 stop (code=exited, status=0/SUCCESS)
Process: 18342 ExecStart=/etc/init.d/apache2 start (code=exited, status=1/FAILURE)
nov. 10 15:41:08 vpsxxx.ovh.net apache2[18342]: * The apache2 configtest failed.
nov. 10 15:41:08 vpsxxx.ovh.net apache2[18342]: Output of config test was:
nov. 10 15:41:08 vpsxxx.ovh.net apache2[18342]: (2)No such file or directory: AH02291: Cannot access di...f:10
nov. 10 15:41:08 vpsxxx.ovh.net apache2[18342]: AH00014: Configuration check failed
nov. 10 15:41:08 vpsxxx.ovh.net apache2[18342]: Action 'configtest' failed.
nov. 10 15:41:08 vpsxxx.ovh.net apache2[18342]: The Apache error log may have more information.
nov. 10 15:41:08 vpsxxx.ovh.net systemd[1]: apache2.service: Control process exited, code=exited status=1
nov. 10 15:41:08 vpsxxx.ovh.net systemd[1]: Failed to start LSB: Apache2 web server.
nov. 10 15:41:08 vpsxxx.ovh.net systemd[1]: apache2.service: Unit entered failed state.
nov. 10 15:41:08 vpsxxx.ovh.net systemd[1]: apache2.service: Failed with result 'exit-code'.
Hint: Some lines were ellipsized, use -l to show in full.
The simplest way would be to have omnibus install and configure nginx on a loopback address, but then use apache in front as a reverse proxy.
Simply re-enable nginx and add the following to your /etc/gitlab/gitlab.rb:
nginx['listen_addresses'] = ['127.0.1.1']
Then reconfigure your gitlab installation.
Then, in your apache site configuration, use something similar to this:
<VirtualHost *:80>
ServerName exemple-gitlab.com
ProxyPreserveHost On
ProxyPass / http://127.0.1.1/
ProxyPassReverse / http://127.0.1.1/
</VirtualHost>
By allowing omnibus to manage the nginx configuration, you don't have to deal with modifying the web server configuration every time they move a service from the unicorn workers to the gitlab-git-http-server Go server. Apache will seamlessly proxy requests to nginx (on the loopback) which will be automatically keep up to date by omnibus during updates.
Sorry I posted it on ServerFault and found the solution :
https://serverfault.com/questions/735270/gitlab-8-0-3-with-apache2-nginx/735273#735273
I found why apache2 crashed by commenting each line (damm logs were
useless).
I just had to create /var/log/apache2/logs
ErrorLog /var/log/apache2/logs/gitlab.example.com_error.log
CustomLog /var/log/apache2/logs/gitlab.example.com_forwarded.log common_forwarded
CustomLog /var/log/apache2/logs/gitlab.example.com_access.log combined env=!dontlog
CustomLog /var/log/apache2/logs/gitlab.example.com.log combined
apache2 was crashing because the folder/directory was missing ...
So now gitlab is working on is domain and my drupal too like this :
gitlab.com:80
drupal.com:80
Exactrly as I wanted :)