webhdfs not working on HDP sandbox - webhdfs

I am getting an error when I execute the following command on Hortonworks sandbox HDP 2.3_1:
curl -i "http://localhost:50075/webhdfs/v1/queryresult/part-m-00000?op=OPEN
HTTP/1.1 400 Bad Request
Content-Type: application/json; charset=utf-8
Content-Length: 161
Connection: close
{"RemoteException":{"exception":"IllegalArgumentException","javaClassName":"java.lang.IllegalArgumentException","message":"java.net.UnknownHostException: null"}}
When I change the port to 50070, I get a message "curl: (7) couldn't connect to host".
webhdfs property is enabled in my hdsf-site.xml and its a single node hadoop cluster.
<property>
<name>dfs.webhdfs.enabled</name>
<value>true</value>
<final>true</final>
</property>

Does /queryresult/part-m-00000 exist? Try hadoop dfs -ls /queryresult/queryresult/part-m-00000 and see if you can see the file. If you can check the permissions on the file. They should have read ability for the user making the webhdfs query.

First the 50075 is not correct the 50070 is default, but still won't work, because some strange redirection to the sandbox.hortonworks.com.
To fix it, I added to the "hosts" (for Windows located here C:\Windows\System32\drivers\etc): file the folowing entry
127.0.0.1 sandbox.hortonworks.com
After this my PC managed to deal with this redirect. Maybe you will need to restart http client, in my case it was chrome

As per https://hadoop.apache.org/docs/r1.0.4/webhdfs.html, it is better to change to actual hostname of the machine.
This command works for me (hdp 2.5):
curl -i "sandbox.hortonworks.com:50075/webhdfs/v1/data/xyz.json?op=OPEN"
I couldn't get it to work with localhost.

Related

Configuring a Data Virt resource adapter to handle an F5 redirect

How do I configure the resource adapter and/or the vdb for a url that sits behind an F5? Suppose that my resource adapter and vdb are configured to read data from
https://foo.org/data?cat='pricing'&page=1&rows=20
If this is a direct hostname then Data Virt reads the data correctly. If it is an F5 then I get an ArrayIndexOutOfBoundsException because the InputStream size is zero.
I verified that the authentication configuration works correctly, so it's not authentication-related.
If I curl the above url (when behind F5) then I get a failed 302 and no results. If I curl -L then I get static html error page (generated apparently because the server did not receive the required parameters). If I curl -L -b cookies.txt then I get the expected data. So basically, my challenge it to apply the equivalent of curl -L and -b cookies.txt options to a Data Virt resource adapter and/or vdb.
The web services translator directly does not support 302 (redirection), however it uses CXF underneath to make the connections. So, configure cxf configuration file on web service as defined in examples here 1 look at Configuring Https, then add the redirect configuration to this file as described at 2
<http:client AutoRedirect="true" Connection="Keep-Alive"/>
http://teiid.github.io/teiid-documents/master/content/admin/Web_Service_Data_Sources.html
http://cxf.apache.org/docs/client-http-transport-including-ssl-support.html

curl: (6) Couldn't resolve host, implementing api versioning in rails

After implementing api versioning in rails app, not able to set version through curl command as follows:
i am using 'myapp.com:3000/api/movies' instead of 'localhost:3000/api/movies'
On running following coommand to set version
root#localhost /vagrant/Projects/MyMovieStore (master) $ curl -H 'Accept: application/vnd.example.v1' http://myapp.com:3000/api/movies_all
Error:
curl: (6) Couldn't resolve host 'myapp.com'
The error is not about versioning, It's about that your machine doesn't recognise myapp.com as an alias for localhost.
Edit /etc/hosts file and add the following:
127.0.0.1 myapp.com
This should do the trick.
As Ahmed mentioned you have to add DNS for this new URL by editing the /etc/hosts file and mapping the ip 127.0.0.1 to the myapp.com URL. But, as you can see I don't see you have introduced any versioning to your URL. A versioned URL would look something as follows,
myapp.com:3000/api/v1/movies

Cloud9 Rails API challenge: allowed network but still cannot render console

I am following along with this Treehouse Build a Rails API course. I am using Cloud9 IDE which I think is related to the problem. The step I am stuck on is creating a new list via the API in the console.
At first I was receiving an error message saying "cannot render console" from the network. After googling this, I whitelisted the network in the development.rb file. The error message is different and so I believe I properly whitelisted it (or at least did something). But it does not have the desired effect and the error message seems to contradict itself.
Started POST "/api/lists" for 123.456.78.999 at 2015-09-14 14:53:33 +0000 Cannot render console from 123.456.78.999! Allowed networks: 123.456.78.999, 127.0.0.0/127.123.123.123, ::1
I've changed the numbers because I am not savvy about security. The point (and to me, the mystery) is that it says I can't render the console from a network that is apparently allowed (i.e., 123.456.78.999 === 123.456.78.999).
I would appreciate any advice on what could be the problem. I had no trouble following along with the tutorial until this step; the commands I entered led to the same result as in the videos.
This is the most recent command I've entered:
curl -i -H "Accept: application/json" -H "Content-type: application/json" -X POST -d '{"title":"The Title will go here"}' https://myapp-me.c9.io/api/lists
Entering that URL instead of "localhost" has worked for me thus far but it could also be related to the problem.
Thank you for any clues and happy to post additional code if needed.
please add in config/enviroments/development.rb
config.web_console.whitelisted_ips = '0.0.0.0/0.0.0.0'
To whitelists 2 IPs > Consider this example:
config.web_console.whitelisted_ips = %w( 127.0.0.1 10.0.0.9 )
To whitelist an entire IP subnet > Consider this example:
config.web_console.whitelisted_ips = %w( 127.0.0.1 10.0.0.0/24 )
always add 127.0.0.1, or else the local machine won't have access.
If anyone is still viewing this question in 2022 like I am, then you can edit config/environments/development.rb to account for the entire private network like so:
Rails.application.configure do
.
.
.
config.web_console.permissions = '123.456.78.0/16'
end
More information here.
Part of the problem is that you need to be within the "sudo su" shell in order to run Ruby on Rails applications. So, make sure that you run sudo su and you become root. The second problem is that you MUST run your application on port 8080 so when you run the server make sure the -p 8080 flag is set:
bundle update (optional)
bundle install (optional)
sudo su
rails s -p 8080
New terminal --> Preview --> Preview running application

unable to start rabbitmq-server

I installed rabbitmq using homebrew. I am trying to start rabbitmq server but I always get this error which I am unable to figure out why!
I have erlang installed and there is no other application running on the same port.
$ rabbitmq-server
{error_logger,{{2013,2,11},{22,37,49}},"Can't set short node name!\nPlease check your configuration\n",[]}
{error_logger,{{2013,2,11},{22,37,49}},crash_report,[[{initial_call,{net_kernel,init,['Argument__1']}},{pid,},{registered_name,[]},{error_info,{exit,{error,badarg},[{gen_server,init_it,6,[{file,"gen_server.erl"},{line,320}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,227}]}]}},{ancestors,[net_sup,kernel_sup,]},{messages,[]},{links,[]},{dictionary,[{longnames,false}]},{trap_exit,true},{status,running},{heap_size,610},{stack_size,24},{reductions,249}],[]]}
{error_logger,{{2013,2,11},{22,37,49}},supervisor_report,[{supervisor,{local,net_sup}},{errorContext,start_error},{reason,{'EXIT',nodistribution}},{offender,[{pid,undefined},{name,net_kernel},{mfargs,{net_kernel,start_link,[[rabbitmqprelaunch1593,shortnames]]}},{restart_type,permanent},{shutdown,2000},{child_type,worker}]}]}
{error_logger,{{2013,2,11},{22,37,49}},supervisor_report,[{supervisor,{local,kernel_sup}},{errorContext,start_error},{reason,shutdown},{offender,[{pid,undefined},{name,net_sup},{mfargs,{erl_distribution,start_link,[]}},{restart_type,permanent},{shutdown,infinity},{child_type,supervisor}]}]}
{error_logger,{{2013,2,11},{22,37,49}},std_info,[{application,kernel},{exited,{shutdown,{kernel,start,[normal,[]]}}},{type,permanent}]}
{"Kernel pid terminated",application_controller,"{application_start_failure,kernel,{shutdown,{kernel,start,[normal,[]]}}}"}
Crash dump was written to: erl_crash.dump
Kernel pid terminated (application_controller) ({application_start_failure,kernel,{shutdown,{kernel,start,[normal,[]]}}})
btw, erl -sname abc gives the same output
Update:
This is what I have in /etc/hosts
127.0.0.1 localhost
255.255.255.255 broadcasthost
check your computer name and your short host name or alias name in /etc/hosts, match this
Check your computer name [wendy#nyc123]$
nyc123 is your computer name
Check your short hostname
[wendy#nyc123]$ hostname -s
[wendy#nyc123]$ nyc456
This error could happen because your computer name and short host name didn't match. To match this, you can change the computer hostname or alias name.
Change computer host name
[wendy#nyc123]$ hostname nyc456
close your terminal and open again
[wendy#nyc456]$
the computer name has changed
or
Change alias name in /etc/hosts
127.0.0.1 nyc123.com nyc123
save and check again
[wendy#nyc123]$ hostname -s
[wendy#nyc123]$ nyc123
Restart your rabbitmq!
[root#nyc123]$ rabbitmq-server start</p>
RabbitMQ 3.6.0. Copyright (C) 2007-2015 Pivotal Software, Inc.</p>
## ## Licensed under the MPL. See http://www.rabbitmq.com/</p>
## ##</p>
########## Logs: /var/log/rabbitmq/rabbitmq#nyc123.com.log</p>
###### ## /var/log/rabbitmq/rabbitmq#nyc123.com-sasl.log</p>
##########</p>
Starting broker... completed with 6 plugins.</p>
I looked for a similar error on google, and it looks like it can happen if your /etc/hosts file is in the wrong format. Try fixing it and see if that helps.
References:
http://www.ejabberd.im/node/18
Explanation on RabbitMQ Mailing list
Edit: For completeness, it seems like setting a long name (of the form abc#abc) worked.
Found the answer here:
control rabbitmq 'name' not 'sname'
Set your machine name to something simple and make it an alias to locahost
I also encountered this problem yesterday and found the root cause:
I had changed my system's hostname to a "long" name, pm3(hc desktop).
If your server's hostname is long or invalid, Linux can still work and no error message is prompted to you. As you just modify the /etc/hostname file and reboot. However, the rabbitmq server may not work and give this "short-name" error message to you.
I changed hostname back to "pm3", rebooted and everything went well.
I solved this issue changing the computer name (on windows 8.1). The problem was that the name had a strange character (é) spanish letter. My computer name was Andrés and I changed it to Andres, restarted my computer and everything worked well. I think that Rabbit could not recognize that name (Andrés) for that strange character.
remove old style config file /etc/rabbitmq/rabbitmq.config
and use rabbitmq.conf
with listeners.tcp.default = 5672
after that restart rabbitmq server again
In my case that solved the issue in ec2 instance

Uwsgi server must execute in script path else I get a http 500 response

I can't understand why I am getting a http 500 response. When I run my uwsgi server from the path that my python script is defined in, everything is perfect. As soon as I run it from another path, I get a 500 response.
Here is my uwsgi script (I set the protocol to http just for testing purposes):
/home/baz/.virtualenvs/python-flask-benchmark/bin/uwsgi
--master
--socket 127.0.0.1:3031
--pythonpath "/home/baz/Personal/Github/benchmark-node_vs_python/python"
--file "/home/baz/Personal/Github/benchmark-node_vs_python/python/app.py"
--callable app
--processes 20
--virtualenv "/home/baz/.virtualenvs/python-flask-benchmark"
--enable-threads
--protocol http
Does anyone know why? It seems like it is ignoring the --pythonpath argument.
Okay, figured it out. It was a path error whereby my application could not find a file to open. The problem was that this was not logged to Uwsgi, which was just silent. The reason was because this was a flask application, which needs to be instructed to pass all exception errors to uwsgi by doing this: app.config['PROPAGATE_EXCEPTIONS'] = True

Resources