Using FastCGI on my dedicated server (Debian).
I now have following error, sometimes (total random behavior !!).
Resulting to white page (error 500).
[Tue May 27 13:02:09 2014] [error] [client 85.68.183.29] (104)Connection reset by peer: FastCGI: comm with server "/var/www/php5.external" aborted: read failed, referer: [...]
[Tue May 27 13:02:09 2014] [error] [client 85.68.183.29] FastCGI: incomplete headers (0 bytes) received from server "/var/www/php5.external", referer: [...]
I cannot find any other errors linked to this (any PHP details, MySQL SQL error, nothing else !!!).
Any idea to prevent this ugly bug?
Should I come back to mod-php5 ??
You might try following the suggestion on this page: https://groups.google.com/d/msg/highload-php-en/4F79Pco-2eg/_tfPMiLFzg4J
Copied here for reference:
Use -idle-timeout paramater on "FastCgiExternalServer" line to solve
this problem.
My FastCgiExternalServer line: FastCgiExternalServer
/var/run/fastcgi/USERNAME-fcgi -appConnTimeout 10 -idle-timeout 250
-socket /var/run/fastcgi/USERNAME.socket -pass-header Authorization
More information in mod_fastcgi doc:
http://www.fastcgi.com/mod_fastcgi/docs/mod_fastcgi.html
I had this issue as well. I figured out that there was some recursive dependency that resulted in not enough memory being available. Resolving the recursive dependency removed the issue.
Related
I am using Dockerflow to run parallel tasks through the Google Pipelines API on Google Cloud Platform. I started a single-step task running 1389 VMs in parallel and found that 233 of the VMs were apparently doing nothing and hanging indefinitely.
I did a spot check of the serial console output and repeatedly saw the VMs running into "Getting controller config failed" errors.
When I tried logging into the VMs I received the error: "Connection Failed. We are unable to connect to the VM on port 22".
I am wondering why my VM instances are hanging, and if there is something I can do to avoid running into these issues.
I've included a snippet of the serial console output below
startupscript: +++ readlink -f /usr/share/google-genomics/startup.sh
startupscript: ++ dirname /usr/share/google-genomics/startup.sh
startupscript: + cd /usr/share/google-genomics
startupscript: + ./controller --operation_id <id> --validation_token <token> --base_path https://genomics.googleapis.com
create controller[2905]: Getting controller config
create controller[2905]: Getting controller config failed, will retry: Get <link>: Get <service_account_token_link>: net/http: timeout awaiting response headers
create controller[2905]: Getting controller config failed, will retry: Get <link>: dial tcp 74.125.26.95:443: i/o timeout
collectd[2342]: write_gcm: Asking metadata server for auth token
collectd[2342]: write_gcm: curl_easy_perform() failed: Couldn't connect to server
collectd[2342]: write_gcm: Error -1 from wg_curl_get_or_post
collectd[2342]: write_gcm: wg_transmit_unique_segment failed.
collectd[2342]: write_gcm: wg_transmit_unique_segments failed. Flushing.
there was a temporary networking issue in us-east1-b. All 3 above VMs were in us-east1-b. These minor incidents do not appear in https://status.cloud.google.com/
Serial console output for a successful run looks like:
A Feb 21 19:05:06 ggp-5629907348021283130 startupscript: + ./controller --operation_id --validation_token --base_path https://autopush-genomics.sandbox.googleapis.com
A Feb 21 19:05:06 ggp-5629907348021283130 create controller[2689]: Getting controller config
A Feb 21 19:05:36 ggp-5629907348021283130 create controller[2689]: Getting controller config failed, will retry: Get https://genomics.googleapis.com/v1alpha2/pipelines:getControllerConfig?alt=json&operationId=&validationToken=: dial tcp 173.194.212.81:443: i/o timeout
A Feb 21 19:05:43 ggp-5629907348021283130 controller[2689]: Switching to status: pulling-image
A Feb 21 19:05:43 ggp-5629907348021283130 controller[2689]: Calling SetOperationStatus(pulling-image)
A Feb 21 19:05:44 ggp-5629907348021283130 controller[2689]: SetOperationStatus(pulling-image) succeeded
The "Getting controller config failed, will retry" is fine. It succeeded upon retry. The "SetOperationStatus(pulling-image) succeeded" indicates networking is working.
In theory, you can submit any number of jobs to Pipelines API and the API will take care of queueing.
If these temporary networking hiccups become common, we may consider changing Pipelines API to somehow detect and retry.
there may have been a temporary networking issue. Can you give me some failed operation ids (or failed VM names)?
Have you tried again since then; can you reproduce the problem?
We are migrating our python scripts to lua scripts as part of apache 2.4 upgrade. One of the requirement is connecting to Sybase database and execute queries.
To do that we have developed a small code using mod_lua api to get db connection, but we haven't been successful.
We have installed the apr-util with freetds.
To get the database connection using mod_lua, mod_dbd and freetds - we followed the steps mentioned here -
http://modlua.org/api/database#dbd
To configure the DPDParams for freetds, we followed the params mentioned here
https://httpd.apache.org/docs/2.4/mod/mod_dbd.html#DBDParams
In VirtualHost of httpd.conf, we have added the following dbdparams
DBDriver freetds
DBDParams username=xxx,password=xxx,host=host-ip:port
DBDMax 10
and the lua code, just for getting a database connection is
require "apache2"
require "string"
function handle(r)
r.content_type = "text/html"
local database, err = r:dbacquire("mod_dbd")
r:err("inside handle method_1 " .. err)
return apache2.OK
end
The error we are getting in apache error log is-
[Thu Aug 25 15:28:03.198044 2016] [dbd:error] [pid 21708:tid
139621318366976] (20014)Internal error (specific information not
available): AH00629: Can't connect to freetds:
[Thu Aug 25 15:28:03.198145 2016] [dbd:error] [pid 21708:tid
139621318366976] (20014)Internal error (specific information not
available): AH00633: failed to initialise
[Thu Aug 25 15:28:03.198184 2016] [lua:error] [pid 21708:tid
139621318366976] [client 10.135.15.148:52836] inside handle method_1
Could not acquire connection from mod_dbd. If your database is
running, this may indicate a permission problem.
We are able to connect to the database using tsql from the same system, but connection from apache dbd, is not working.
We are suspecting that there might be some configuration(DBDParams) problems or that the OS may be blocking connection from apache
Could someone please help in this regard.
We found the solution. The problem was there in the DBDParams we were passing. For connecting to sybase, we were providing the connection details(host:port) in the 'host' param(host=). On futher looking into the apr-util-freetds code, we found that for Sybase connection it is the **server(server=) param where we should provide the host port connection details**.
http://www.freetds.org/reference/a00371.html#gaef0e7a5fcf2d8c8f795b2b06ce4de8b1
The DBD Params which worked for Sybase connection using freetds is -
DBDParams username=xxx,password=xxxxxx,server=h.o.s.t:port
Its a bit confusing because in sybase terms host is normally the server hostname/ip.
I've got an embedded experience gadget configured for the IBM Connections activity stream. It seems that the OAuth dance never succeeds, only returning me to the beginning of the OAuth dance (prompt for authorisation).
I see errors like this:
[2/25/14 7:17:45:197 EST] 000000c2 BasicOAuth2Me 3 JSONException parsing response
org.apache.commons.json.JSONException: Error occurred during input read.
Caused by: java.io.IOException: Unexpected character '<' on line 1, column 1
at org.apache.commons.json.internal.Tokenizer.next(Tokenizer.java:185)
at org.apache.commons.json.internal.Parser.parse(Parser.java:114)
at org.apache.commons.json.internal.Parser.parse(Parser.java:85)
at org.apache.commons.json.JSONObject.<init>(JSONObject.java:128)
at org.json.JSONObject.<init>(JSONObject.java:41)
...which may suggest that XML is returned from something instead of JSON.
I'm also seeing POST /connections/opensocial/gadgets/makeRequest HTTP/1.1
returning:
HTTP/1.1 200 OK
Date: Tue, 25 Feb 2014 12:54:12 GMT
Server: IBM_HTTP_Server
X-Powered-By: Servlet/3.0
X-LConn-Auth: true
X-UA-Compatible: IE=EmulateIE7
Cache-Control: no-cache
Expires: Tue, 25 Feb 2014 12:54:12 GMT
Pragma: no-cache
Content-Disposition: attachment;filename=p.txt
Content-Length: 1029
Content-Type: application/json;charset=UTF-8
Content-Language: en-US
throw 1; < 'invalid javascript' > {"http://appfusions.demos.ibm.com:2990/jira/rest/api/2/issue/DEMO-12/transitions?expand=transitions.fields":{"rc":200,"body":"","oauthApprovalUrl":"http://appfusions.demos.ibm.com:2990/jira/plugins/servlet/af/oauth/authorize?client_id=8821f0cfd9264c4ed08b00861016886c&realGadgetUri=http%3A%2F%2Fappfusions.demos.ibm.com%3A2990%2Fjira%2Frest%2Fgadgets%2F1.0%2Fg%2Fcom.appfusions.jira.connections.jira-immersive-for-connections%3Aconnections-gadgets%2Fgadgets%2Fjira-ee.xml&redirect_uri=https%3A%2F%2Fconnectionsww.demos.ibm.com%2Fconnections%2Fopensocial%2Fgadgets%2Foauth2callback&response_type=code&state=WIn_8DLhmOeRc8aIYomi0j2XGIAAd7Ke3j2E4y9SbXpD6aZcmU_sKbJpOh4CEuJKr1NLEEsm8xAB-gzYz9laubsztcW-I-YC8gfq67kGbW2wxYhFh6PCaqEZgAhzD7sfRGsj_bBLykeVpSCRZCI7h8pbO9UbtiBfKDKUDwyqm2t35GWi0DV3KcWZlnKso97-z8Xlzj5AysY5ifWux0nJwHCLJ70Fq4wV738SQ5ZDExqSQkaiGY-w7NlJRHRCVP0wJzIEWP2fsSkNMbN8ui4ULLD1kR0w4T_VlBYS1QXbjvXJ6XUwEc7xhusC2ixvDzhBjjLOmbrwqOdMQnFRrxilX--Q9mI","DataHash":"qgeopmcf02p09qc016cepu22fo"}}
Here's a full trace.log -- I'm currently logging to trace with these packages:
*=info:
com.ibm.social.pim.resource.*=all:
com.ibm.lconn.homepage.services.widget.*=all:
com.ibm.lconn.news.*=all:
com.ibm.lconn.events.*=all:
com.ibm.lotus.search.engine.*=all:
com.ibm.lotus.search.index.*=all:
com.ibm.lconn.mailin.*=all:
com.ibm.lotus.connections.core.notify.*=all:
com.ibm.lconn.scheduler.*=all:
org.apache.shindig.*=all
I'd love to know a good way of debugging that JSONException as well as more into on the makeRequest POST.
Any clues on how to debug further are greatly appreciated.
Update
I've updated the trace.log after Paul's logging suggestion and I'm now seeing exceptions while storing OAuth information:
[2/27/14 7:16:17:062 EST] 0000014e RetryConnecti 3 Exception caught during attempt
org.springframework.dao.DataIntegrityViolationException: SqlMapClient operation; SQL [];
--- The error occurred in resources/news/oauth/OAuth2TokenSQL.xml.
--- The error occurred while applying a parameter map.
--- Check the insert-OAUTH2_TOKEN-InlineParameterMap.
--- Check the statement (update failed).
--- Cause: com.ibm.db2.jcc.am.SqlIntegrityConstraintViolationException: Assignment of a NULL value to a NOT NULL column "TBSPACEID=5, TABLEID=14, COLNO=4" is not allowed.. SQLCODE=-407, SQLSTATE=23502, DRIVER=4.13.80; nested exception is com.ibatis.common.jdbc.exception.NestedSQLException:
--- The error occurred in resources/news/oauth/OAuth2TokenSQL.xml.
--- The error occurred while applying a parameter map.
--- Check the insert-OAUTH2_TOKEN-InlineParameterMap.
--- Check the statement (update failed).
--- Cause: com.ibm.db2.jcc.am.SqlIntegrityConstraintViolationException: Assignment of a NULL value to a NOT NULL column "TBSPACEID=5, TABLEID=14, COLNO=4" is not allowed.. SQLCODE=-407, SQLSTATE=23502, DRIVER=4.13.80
at org.springframework.jdbc.support.SQLErrorCodeSQLExceptionTranslator.doTranslate(SQLErrorCodeSQLExceptionTranslator.java:228)
at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:72)
at org.springframework.orm.ibatis.SqlMapClientTemplate.execute(SqlMapClientTemplate.java:212)
at org.springframework.orm.ibatis.SqlMapClientTemplate.insert(SqlMapClientTemplate.java:397)
at com.ibm.lconn.news.data.dao.impl.ibatis.CrudDao.insert(CrudDao.java:394)
at com.ibm.lconn.news.data.dao.impl.ibatis.oauth.OAuth2TokenDao.insert(OAuth2TokenDao.java:61)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:60)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)
at java.lang.reflect.Method.invoke(Method.java:611)
The OAuth provider appears to be supplying OAuth tokens correctly from what I can see -- also in trace.log
I've installed tcpflow on the IBM Connections machine. Tracking the server side OAuth flow on the IBM Connections machine using...
root$ tcpflow -p -c -i eth0 port 2990 | grep -oE '(GET|POST|HEAD) .* HTTP/1.[01]|Host: .*' > cnx-outgoing-2990.log
I've found a 404 error on the OAuth provider when IBM Connections POSTs to the access_token URL -- will concentrate on that :)
I'm using Apache + passenger for my rails application, The application is running fine in production mode except when the browser displays error message after every 2-3 clicks in development mode which goes off when i refresh the page.I'm using metal in my rails application which i see could be the reason but not sure if it is the only reason.
Error message on browser:
Internal Server Error
The server encountered an internal error or misconfiguration and was unable to complete your request.
Please contact the server administrator, [no address given] and inform them of the time the error occurred, and anything you might have done that may have caused the error.
More information about this error may be available in the server error log.
Apache/2.2.16 (Ubuntu) Server at localhost Port 80
Apache log Error:
[Thu Dec 01 16:55:23 2011] [error] [client 127.0.0.1] Premature end of script headers: view_transactions, referer: http://localhost/admin_portal/1014978061/view_transactions
[ pid=660 thr=3078097552 file=ext/apache2/Hooks.cpp:822 time=2011-12-01 16:55:23.563 ]: The backend application (process 796) did not send a valid HTTP response; instead, it sent nothing at all. It is possible that it has crashed; please check whether there are crashing bugs in this application.
[ pid=796 thr=-608495398 file=utils.rb:176 time=2011-12-01 11:25:23.878 ]: *** Exception NoMethodError in application (undefined method `new' for Rails:Module) (process 796, thread #<Thread:0xb77631b4>):
from /home/brijendra/aptana_studio_3_workspace/wantsa/vendor/plugins/newrelic_rpm/lib/new_relic/control.rb:467:in `new_instance'
from /home/brijendra/aptana_studio_3_workspace/wantsa/vendor/plugins/newrelic_rpm/lib/new_relic/control.rb:54:in `instance'
from /home/brijendra/aptana_studio_3_workspace/wantsa/vendor/plugins/newrelic_rpm/lib/new_relic/agent.rb:167:in `logger'
I would be eager to post any other output if required and also suggestions to fix this issue.
I fixed this error by changing cache as true in my development.rb and it worked fine.
Ok I thought I was close to getting passenger and Apache working. I notice that some gem files were not installed after navigating to the url to see if my rails app was working. Passanger error page let me know what gems were missing so I got them installed.
Now going to the URL I get a 500 Apache internal error page with no helpful info so I checked out the log file on the server and here is what I see.
Rails Error: Unable to access log file. Please ensure that /home/mydirectory/dev/vb/log/production.log exists and is chmod 0666. $
Rack: /home/mydirectory/dev/vb: symbol lookup error: /usr/local/rvm/gems/ruby-1.9.2-p0#prodset/gems/sqlite3-ruby-1.2.4/lib/sqlite$
[Tue Dec 07 20:12:17 2010] [error] [client 64.58.208.22] Premature end of script headers:
[ pid=20653 thr=140618873321280 file=ext/apache2/Hooks.cpp:816 time=2010-12-07 20:12:17.617 ]: The backend application (proce$
Rack: /home/mydirectory/dev/vb: symbol lookup error: /usr/local/rvm/gems/ruby-1.9.2-p0#prodset/gems/sqlite3-ruby-1.2.4/lib/sqlite$
[Tue Dec 07 20:12:43 2010] [error] [client 64.58.208.22] Premature end of script headers:
Rack: /home/mydirectory/dev/vb: symbol lookup error: /usr/local/rvm/gems/ruby-1.9.2-p0#prodset/gems/sqlite3-ruby-1.2.4/lib/sqlite$
[Tue Dec 07 20:13:25 2010] [error] [client 64.58.208.22] Premature end of script headers:
[ pid=21932 thr=140618873321280 file=ext/apache2/Hooks.cpp:816 time=2010-12-07 20:13:25.168 ]: The backend application (proce$
Rack: /home/mydirectory/dev/vb: symbol lookup error: /usr/local/rvm/gems/ruby-1.9.2-p0#prodset/gems/sqlite3-ruby-1.2.4/lib/sqlite$
[Tue Dec 07 20:13:31 2010] [error] [client 64.58.208.22] Premature end of script headers:
[ pid=20623 thr=140618873321280 file=ext/apache2/Hooks.cpp:816 time=2010-12-07 20:13:31.266 ]: The backend application (proce$
Rails Error: Unable to access log file. Please ensure that /home/mydirectory/dev/vb/log/production.log exists and is chmod 0666. $
Rack: /home/mydirectory/dev/vb: symbol lookup error: /usr/local/rvm/gems/ruby-1.9.2-p0#prodset/gems/sqlite3-ruby-1.2.4/lib/sqlite$
[Tue Dec 07 20:24:56 2010] [error] [client 64.58.208.22] Premature end of script headers:
[ pid=20622 thr=140618873321280 file=ext/apache2/Hooks.cpp:816 time=2010-12-07 20:24:56.442 ]: The backend application (proce$
Rack: /home/mydirectory/dev/vb: symbol lookup error: /usr/local/rvm/gems/ruby-1.9.2-p0#prodset/gems/sqlite3-ruby-1.2.4/lib/sqlite$
anyone have any suggestions on what I should look at next. I have tried running bundler and also using rvm to install sqlite3 and I still have the same issue.
thanks again for any help
Did you checked the suggestion on the first line of the error log?
Rails Error: Unable to access log file. Please ensure that /home/mydirectory/dev/vb/log/production.log exists and is chmod 0666.