Twitter API authorization fails CORS preflight in browser - twitter

I'm trying to do the 3-legged authorization necessary to call the Twitter APIs in a browser. The process starts with getting a request token by POSTing a signed request to /oauth/request_token (this is also how sign in with Twitter begins).
My problem is that before the browser will POST to the Twitter API endpoint, it wants to preflight the request with an OPTIONS method. This preflight request is always returning status 400 (Bad Request).
Here's an example that you can cut and paste into a browser console that supports the Fetch API:
fetch('https://api.twitter.com/oauth/request_token', { method: 'POST', mode: 'cors', headers: new Headers({ authorization: 'xxx' }), body: 'oauth_callback=http%3A%2F%2Flocalhost%2F' });
On Chrome, the preflight request looks like this (Firefox is similar):
OPTIONS /oauth/request_token HTTP/1.1
accept:*/*
accept-encoding:gzip, deflate, sdch
accept-language:en-US,en;q=0.8
access-control-request-headers:authorization, content-type
access-control-request-method:POST
cache-control:no-cache
origin:null
pragma:no-cache
user-agent:Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.116 Safari/537.36
And the preflight response looks like this:
HTTP/1.1 400 Bad Request
content-length: 0
date: Tue, 08 Mar 2016 22:21:37 GMT
server: tsa_a
x-connection-hash: 529e3d8338caeb980077637d86db5df1
Note that problem is not that I didn't specify a real authorization header in the example above. The value of the authorization header is not used in the preflight request.
If I print out the components of my POST request to the console and assemble the pieces into a curl command (which doesn't preflight) then I can get a request token. But if I try to simulate the preflight request in curl, I haven't been able to get that to work:
$ curl -v -X OPTIONS -H "access-control-request-headers:authorization,content-type" -H "access-control-request-method:POST" -H "origin:http://example.com" https://api.twitter.com/oauth/request_token
* Trying 199.59.148.20...
* Connected to api.twitter.com (199.59.148.20) port 443 (#0)
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:#STRENGTH
* successfully set certificate verify locations:
* CAfile: /opt/local/share/curl/curl-ca-bundle.crt
CApath: none
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-ECDSA-AES128-GCM-SHA256
* ALPN, server accepted to use http/1.1
* Server certificate:
* subject: C=US; ST=CA; L=San Francisco; O=Twitter, Inc.; OU=Twitter Security; CN=api.twitter.com
* start date: Aug 11 00:00:00 2015 GMT
* expire date: Aug 15 12:00:00 2016 GMT
* subjectAltName: api.twitter.com matched
* issuer: C=US; O=DigiCert Inc; OU=www.digicert.com; CN=DigiCert SHA2 High Assurance Server CA
* SSL certificate verify ok.
> OPTIONS /oauth/request_token HTTP/1.1
> Host: api.twitter.com
> User-Agent: curl/7.47.1
> Accept: */*
> access-control-request-headers:authorization,content-type
> access-control-request-method:POST
> origin:http://example.com
>
< HTTP/1.1 400 Bad Request
< content-length: 0
< date: Tue, 08 Mar 2016 23:06:44 GMT
< server: tsa_a
< x-connection-hash: 66174829ef6d3f5e5ec641ac080ad19c
<
* Connection #0 to host api.twitter.com left intact
What am I missing that will let me do a successful CORS preflight to https://api.twitter.com/oauth/request_token?

So the unsatisfying resolution appears to be that the Twitter API does not support CORS. This seems a little astonishing to me, as it means that the API cannot be used from a browser.
That policy decision is probably related to their OAuth implementation, which is vulnerable to anyone with access to the calling platform. Maybe that was okay back in 2010, but most of the other major internet players have figured out how to do client-based authorization.

I recently faced this issue. I didn't want to spend lots of time creating node server, so I used Netlify's serverless functions. It is super easy and straight forward + free
I followed this blog, it worked perfectly fine for me: https://www.digitalocean.com/community/tutorials/nodejs-solve-cors-once-and-for-all-netlify-dev

A workaround and it is a long way around is to build a proxy server that you run with node or something else, I've done this a few times now, this is a good starter repo for someone running into this issue. It has the downside of being React specific, but you could always rip out the react ui and just setup with whatever you need: hcra twitter build.
Its forked from a Create React App/Node Express boilerplate by Mars Hall
You will need to clone it and run git fetch and then checkout twitter-req branch.

One workaround would be to create a NodeJS server for the same and call the Node API requesting Twitter API from front-end.

Check this workaround: https://stackoverflow.com/a/43881141
If you don’t control the server your frontend JavaScript code is
sending a request to, and the problem with the response from that
server is just the lack of the necessary Access-Control-Allow-Origin
header, you can still get things to work—by making the request through
a CORS proxy.
In short, prefix the request so the full URI looks something like this:
https://cors-anywhere.herokuapp.com/https://api.twitter.com/oauth/request_token
Unfortunately, you will not be able to use most Twitter libraries this way, but you can continue using fetch with the cors-anywhere prefix.
Edit: The mentioned cors-anywhere web service is no longer permanently available, but instead can be temporarily "activated" for development use.

Related

Microsoft Graph API - 503 Service Unavailable with "UnknownError"

Our application performs lots of calls to MS Graph API, mainly calls to messages, calendars and contacts. Due to huge amount of calls we experience throttling and sometime 503 Service Unavailable error. Sometimes description of 503 Service Unavailable error is following:
ErrorCode: "UnknownError", ErrorMessage: ""
Usually those errors disappear after one or several retries.
Starting on 5th of November the amount of such "UnknownError" errors increased dramatically. I couldn't find any MS Graph API service update description that could lead to those changes. What is a cause for the increase of those errors? How can I examine it?
Having the same issue, and it's obviously not related to the throttling like the previous answer suggested.
The sorting seems to be causing the issue for some reason. Almost as if there was a bad message that the API was failing to parse & throwing the exception.
Here is a full trace:
➜ workturbo curl -v -H "Authorization: Bearer $TOKEN" https://graph.microsoft.com/beta/me/messages\?%24select=changeKey\&%24filter=isDraft+eq+false+and+lastModifiedDateTime+lt+2020-02-01T23%3A19%3A57Z\&%24orderby=lastModifiedDateTime+desc\&%24skip=0\&%24top=5
* Trying 20.190.132.119...
* TCP_NODELAY set
* Connected to graph.microsoft.com (20.190.132.119) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/cert.pem
CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server did not agree to a protocol
* Server certificate:
* subject: CN=graph.microsoft.com
* start date: Jul 1 21:55:17 2020 GMT
* expire date: Jul 1 21:55:17 2022 GMT
* subjectAltName: host "graph.microsoft.com" matched cert's "graph.microsoft.com"
* issuer: C=US; ST=Washington; L=Redmond; O=Microsoft Corporation; OU=Microsoft IT; CN=Microsoft IT TLS CA 2
* SSL certificate verify ok.
> GET /beta/me/messages?%24select=changeKey&%24filter=isDraft+eq+false+and+lastModifiedDateTime+lt+2020-02-01T23%3A19%3A57Z&%24orderby=lastModifiedDateTime+desc&%24skip=0&%24top=5 HTTP/1.1
> Host: graph.microsoft.com
> User-Agent: curl/7.64.1
> Accept: */*
> Authorization: Bearer <OBFUSCATED>
>
< HTTP/1.1 503 Service Unavailable
< Cache-Control: private
< Content-Type: application/json
< request-id: 3903e2c4-325b-4925-8544-20383e2a2944
< client-request-id: 3903e2c4-325b-4925-8544-20383e2a2944
< x-ms-ags-diagnostic: {"ServerInfo":{"DataCenter":"West US","Slice":"SliceC","Ring":"5","ScaleUnit":"003","RoleInstance":"AGSFE_IN_17"}}
< Strict-Transport-Security: max-age=31536000
< Date: Tue, 21 Jul 2020 19:30:23 GMT
< Content-Length: 198
<
{
"error": {
"code": "UnknownError",
"message": "",
"innerError": {
"date": "2020-07-21T19:30:23",
"request-id": "3903e2c4-325b-4925-8544-20383e2a2944"
}
}
* Connection #0 to host graph.microsoft.com left intact
}* Closing connection 0
Please take a read of our guidance on Throttling. https://learn.microsoft.com/en-us/graph/throttling
You need to respect the retry-after header that gets returned with calls. Otherwise you'll get 429s and sometimes 503s like you are getting.
If you are using our Microsoft Graph SDKs, it has a built in throttling handler.

SSL certificate is not valid on iOS

I have a simple backend for my iOS application. It used to work perfectly but recently SSL connection started to fail only on iOS.
The strange part is that it started fail on Friday at some point and then started work again. Now I can't use my backend on iOS.
All certificate checks I did shows that everything is ok and TLS 1.2 is working. But iOS (and only iOS, macOS is fine) says that SSL cert is invalid.
Tried to renew certificate - didn't help. Certificate is not self-signed (it is from letsencrypt.org)
This is a test link:
https://api.dartoapp.com:9001/station/test1
SSL Lab report:
https://www.ssllabs.com/ssltest/analyze.html?d=api.dartoapp.com
NSURLRequest error:
Error Domain=NSURLErrorDomain Code=-1200 "An SSL error has occurred and a secure connection to the server cannot be made." UserInfo={NSURLErrorFailingURLPeerTrustErrorKey=<SecTrustRef: 0x1d430dec0>, NSLocalizedRecoverySuggestion=Would you like to connect to the server anyway?, _kCFStreamErrorDomainKey=3, _kCFStreamErrorCodeKey=-9802, NSErrorPeerCertificateChainKey=(
"<cert(0x1400cec00) s: api.dartoapp.com i: Let's Encrypt Authority X3>"
), NSUnderlyingError=0x1d084bc40 {Error Domain=kCFErrorDomainCFNetwork Code=-1200 "(null)" UserInfo={_kCFStreamPropertySSLClientCertificateState=0, kCFStreamPropertySSLPeerTrust=<SecTrustRef: 0x1d430dec0>, _kCFNetworkCFStreamSSLErrorOriginalValue=-9802, _kCFStreamErrorDomainKey=3, _kCFStreamErrorCodeKey=-9802, kCFStreamPropertySSLPeerCertificates=(
"<cert(0x1400cec00) s: api.dartoapp.com i: Let's Encrypt Authority X3>"
)}}, NSLocalizedDescription=An SSL error has occurred and a secure connection to the server cannot be made., NSErrorFailingURLKey=https://api.dartoapp.com:9001/station/test1, NSErrorFailingURLStringKey=https://api.dartoapp.com:9001/station/test1, NSErrorClientCertificateStateKey=0}
curl output:
~ ⟩ curl -kvI https://api.dartoapp.com:9001/station/test1
* Trying 54.154.203.139...
* TCP_NODELAY set
* Connected to api.dartoapp.com (54.154.203.139) port 9001 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:#STRENGTH
* successfully set certificate verify locations:
* CAfile: /etc/ssl/cert.pem
CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use h2
* Server certificate:
* subject: CN=api.dartoapp.com
* start date: Mar 11 18:18:58 2018 GMT
* expire date: Jun 9 18:18:58 2018 GMT
* issuer: C=US; O=Let's Encrypt; CN=Let's Encrypt Authority X3
* SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7ffeac806c00)
> HEAD /station/test1 HTTP/2
> Host: api.dartoapp.com:9001
> User-Agent: curl/7.54.0
> Accept: */*
>
* Connection state changed (MAX_CONCURRENT_STREAMS updated)!
< HTTP/2 200
HTTP/2 200
< content-type: application/json; charset=utf-8
content-type: application/json; charset=utf-8
< content-length: 471
content-length: 471
< date: Sun, 11 Mar 2018 19:38:24 GMT
date: Sun, 11 Mar 2018 19:38:24 GMT
Ok, so the issue was that my server wan't sending intermediate cert. If was OK before but recent LetsEncrypt changes made it mandatory.

Getting "error reading X.509 key or certificate file" by using curb, while curl uses the same certificate w/o errors

On Ubuntu trusty-64 with rvm and ruby 2.0.0-p353 using curb 0.8.5 I try to make post request.
When I perform it in command line using curl it makes request and returns ok, but when I do the same request in ruby, it fails with error reading X.509 key or certificate file message. In both cases I use just the same certificate.
vagrant#cabinet:~/app$ curl -k --cert certs/ds_admin.pem --cert-type PEM https://pim.somewhere.ru:5543/authentication/user/login/? --data "{\"login\":\"+71111111150\",\"password\":\"qwerty123\"}" -v
* Hostname was NOT found in DNS cache 151.209.250.186
* Trying 151.209.250.186...
* Connected to pim.somewhere.ru (151.209.250.186) port 5543 (#0)
* successfully set certificate verify locations:
* CAfile: none
CApath: /etc/ssl/certs
* SSLv3, TLS handshake, Client hello (1):
* SSLv3, TLS handshake, Server hello (2):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS handshake, Server finished (14):
* SSLv3, TLS handshake, Client key exchange (16):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSL connection using AES128-SHA
* Server certificate:
* subject: C=RU; O=BE; OU=test env; CN=*.somewhere.ru
* start date: 2013-02-27 15:09:21 GMT
* expire date: 2015-11-24 15:09:21 GMT
* issuer: C=RU; O=BE; OU=test env; CN=*.somewhere.ru
* SSL certificate verify result: self signed certificate (18), continuing anyway.
> POST /authentication/user/login/? HTTP/1.1
> User-Agent: curl/7.35.0
> Host: pim.somewhere.ru:5543
> Accept: */*
> Content-Length: 47
> Content-Type: application/x-www-form-urlencoded
>
* upload completely sent off: 47 out of 47 bytes
* SSLv3, TLS handshake, Hello request (0):
* SSLv3, TLS handshake, Client hello (1):
* SSLv3, TLS handshake, Server hello (2):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS handshake, Request CERT (13):
* SSLv3, TLS handshake, Server finished (14):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS handshake, Client key exchange (16):
* SSLv3, TLS handshake, CERT verify (15):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
< HTTP/1.1 200 OK
< Content-Length: 325
< Content-Type: application/json; charset=utf-8
* Server Microsoft-HTTPAPI/2.0 is not blacklisted
< Server: Microsoft-HTTPAPI/2.0
< Date: Wed, 30 Jul 2014 06:15:58 GMT
<
* Connection #0 to host pim.somewhere.ru left intact
{"Token":"re7hCmYB4hCjJRphLS4dK58nNAkIvHZYVbY9ie94vICPbJadauLclDjPKFLz9lGCz6jjex3GFTL7NmxBofOP70bx6Abin+UrXZj\/N5PEFi3jYR9LIaIxgz5AXYoLxr1kismBZgjvQYCIndhr8lwNpw==|2014-07-30T08:15:58","User":{"FirstName":"First","LastName":"Last","Login":"+71111111150","UserId":"UAS100004","UserNativeId":"1-156LVK","UserSysName":"siebel"}}
In Rails console by using curb
From: /home/vagrant/app/app/models/uas/query.rb # line 49 Uas::Query.client:
44: def self.client(url)
45: url = "#{Rails.configuration.uas_url}/#{url}"
46:
47: Curl::Easy.new(url) do |curl|
48: curl.use_ssl = 1
=> 49: binding.pry
50: curl.cert = Rails.configuration.uas_sertificate
51: curl.ssl_verify_peer = false
52: curl.ssl_verify_host = false
53: curl.verbose = true
54: end
55: end
[1] pry(Uas::Query)> c
Request post user/login/?
{:login=>"+71111111150", :password=>"qwerty123"}
* Hostname was NOT found in DNS cache
* Trying 151.209.250.186...
* Connected to pim.somewhere.ru (151.209.250.186) port 5543 (#0)
* found 164 certificates in /etc/ssl/certs/ca-certificates.crt
* error reading X.509 key or certificate file
* Closing connection 0
Curl::Err::SSLConnectError: Curl::Err::SSLConnectError
I checked Rails.configuration.uas_sertificate - it's the certificate and could be read by File.read(Rails.configuration.uas_sertificate)
How can I fix it?
In my vagrant box I got installed two libcurl packages:
libcurl4-openssl-dev
libcurl4-gnutls-dev
from which the second one was used for compiling curb gem native extension. I guess it was used due to it was installed after the libcurl4-openssl-dev package.
After I removed libcurl4-gnutls-dev from packages installation list and rebuilt vagrant box, curb had accepted the certificate.

Why can't I get Twitter's OAuth Test to work?

I'm at the bottom of the learning curve for using the REST API v1.1. I am attempting to use cURL to test my Twitter App.
Here's what I've done so far on Twitter's dev apps webpages (starting at https://dev.twitter.com/apps), and a description of the stumbling block which has completely defeated me:
I created an App, then given it "read, write and direct messages" permissions. Then I regenerated the API keys and OAuth tokens.
I went to the app management page for my app and clicked "Test OAuth".
I want to test it with something simple, so under Request Settings, I choose Request Type "GET" and Request Query "account/verify_credentials".
Then I click "see OAuth Signature for this request".
Among the data that is presented to me, I see the curl command to use. It looks like this (edited to hide the keys and the signature):
curl --get 'https://api.twitter.com/1.1/' --data 'account%2Fverify_credentials=' --header 'Authorization: OAuth oauth_consumer_key="XXXXXXXXXXXXXXXXXXXX", oauth_nonce="f187b1a421600e88ffebeed19b6c1fbf", oauth_signature="XXXXXXXXXXXXXXXXXX", oauth_signature_method="HMAC-SHA1", oauth_timestamp="1399538823", oauth_token="XXXXXXXXXXXXXXXXXXXXXXXX", oauth_version="1.0"' --verbose
So, I copy-and-paste this IMMEDIATELY (i.e. so that the timestamp does not expire) to my terminal (I'm using Debian), and I get the following response (again, edited to remove keys and signature), where you can see down the bottom it says, once again, "Invalid or expired token":
* About to connect() to api.twitter.com port 443 (#0)
* Trying 199.16.156.40... connected
* Connected to api.twitter.com (199.16.156.40) port 443 (#0)
* successfully set certificate verify locations:
* CAfile: none
CApath: /etc/ssl/certs
* SSLv3, TLS handshake, Client hello (1):
* SSLv3, TLS handshake, Server hello (2):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS handshake, Server finished (14):
* SSLv3, TLS handshake, Client key exchange (16):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSL connection using RC4-SHA
* Server certificate:
* subject: C=US; ST=California; L=San Francisco; O=Twitter, Inc.; OU=Twitter Security; CN=api.twitter.com
* start date: 2014-04-08 00:00:00 GMT
* expire date: 2014-10-10 23:59:59 GMT
* subjectAltName: api.twitter.com matched
* issuer: C=US; O=VeriSign, Inc.; OU=VeriSign Trust Network; OU=Terms of use at https://www.verisign.com/rpa (c)10; CN=VeriSign Class 3 Secure Server CA - G3
* SSL certificate verify ok.
> GET /1.1/?account%2Fverify_credentials= HTTP/1.1
> User-Agent: curl/7.21.0 (i486-pc-linux-gnu) libcurl/7.21.0 OpenSSL/0.9.8o zlib/1.2.3.4 libidn/1.15 libssh2/1.2.6
> Host: api.twitter.com
> Accept: /
> Authorization: OAuth oauth_consumer_key="XXXXXXXXXXXXXXXXXX", oauth_nonce="f187b1a421600e88ffebeed19b6c1fbf", oauth_signature="XXXXXXXXXXXXXXXXXXXXXX", oauth_signature_method="HMAC-SHA1", oauth_timestamp="1399538823", oauth_token="XXXXXXXXXXXXXXXXXXXXXXXXXXX", oauth_version="1.0"
>
< HTTP/1.1 401 Unauthorized
< content-length: 25
< content-type: text/plain
< date: Thu, 08 May 2014 08:47:38 UTC
< server: tfe
< set-cookie: guest_id=v1%3A139953885863095383; Domain=.twitter.com; Path=/; Expires=Sat, 07-May-2016 08:47:38 UTC
< strict-transport-security: max-age=631138519
<
Invalid or expired token
* Connection #0 to host api.twitter.com left intact
* Closing connection #0
* SSLv3, TLS alert, Client hello (1):
What confuses me completely is that I have submitted Twitter's own curl command back to Twitter and it has rejected it!!!!
Any ideas??? I am completely stuck at this point.
Firstly, You will need to read the verify_credentials API document
In the link above, choose your created application and hit "Generate Oauth Signature" button.
You will see that while perform an OAuth request, you will need following 4 things:
Consumer key
Consumer secret
Access token
Access token secret
Also, do not put "account/verify_credentials" to Request query field, instead fill Request URI
with "https://api.twitter.com/1.1/account/verify_credentials.json".
This time the displayed curl should work fine :)

Twitter search API: "Unauthorized" through HTTParty but successful via Curl

Working on our Rails application, I'm trying to retrieve tweets using the search API v1.1 through HTTParty. I've carefully gone through all steps and unit tested our code with the Twitter examples to setup the right OAuth authorization, so I'm very confident all code is working properly. However, no matter what I try, I keep getting a "{"errors":[{"message":"Could not authenticate you","code":32}]}"" response.
Using the OAuth Tool, I generated the request that I need. I pasted the nonce and timestamp into my code, to replicate the exact same request in our application. This is the resulting log:
opening connection to api.twitter.com...
opened
<- "GET /1.1/search/tweets.json?q=from:plindelauf&include_entities=true HTTP/1.1\r\nAuthorization: OAuth oauth_consumer_key=\"DOP3re9nlTPeyemlmcqg\", oauth_nonce=\"05d29110d282b24e516dae1ae2b54e1a\", oauth_signature=\"k9gHhrHD86D5sB7zAlXCKsaeNZ8%3D\", oauth_signature_method=\"HMAC-SHA1\", oauth_timestamp=\"1370691632\", oauth_token=\"288762088-DfqCgEIVvM56MSMZWjLuLl6oo3oaVkI0Z8ATKu3f\", oauth_version=\"1.0\"\r\nConnection: close\r\nHost: api.twitter.com\r\n\r\n"
-> "HTTP/1.1 401 Unauthorized\r\n"
-> "content-length: 63\r\n"
-> "content-type: application/json; charset=utf-8\r\n"
-> "date: Sat, 08 Jun 2013 11:41:41 UTC\r\n"
-> "server: tfe\r\n"
-> "set-cookie: guest_id=v1%3A137069170192515887; Domain=.twitter.com; Path=/; Expires=Mon, 08-Jun-2015 11:41:41 UTC\r\n"
-> "strict-transport-security: max-age=631138519\r\n"
-> "Connection: close\r\n"
-> "\r\n"
reading 63 bytes...
-> ""
-> "{\"errors\":[{\"message\":\"Could not authenticate you\",\"code\":32}]}"
read 63 bytes
Conn close
Note that even though the above dump does not show it, I have verified that the request is sent via https.
Then executing the CURL command from the OAuth Tool works just fine:
curl --get 'https://api.twitter.com/1.1/search/tweets.json' --data 'include_entities=true&q=from%3Aplindelauf' --header 'Authorization: OAuth oauth_consumer_key="DOP3re9nlTPeyemlmcqg", oauth_nonce="05d29110d282b24e516dae1ae2b54e1a", oauth_signature="k9gHhrHD86D5sB7zAlXCKsaeNZ8%3D", oauth_signature_method="HMAC-SHA1", oauth_timestamp="1370691632", oauth_token="288762088-DfqCgEIVvM56MSMZWjLuLl6oo3oaVkI0Z8ATKu3f", oauth_version="1.0"' --verbose
* About to connect() to api.twitter.com port 443 (#0)
* Trying 199.16.156.8...
* connected
* Connected to api.twitter.com (199.16.156.8) port 443 (#0)
* SSLv3, TLS handshake, Client hello (1):
* SSLv3, TLS handshake, Server hello (2):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS handshake, Server finished (14):
* SSLv3, TLS handshake, Client key exchange (16):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSL connection using RC4-SHA
* Server certificate:
* subject: C=US; ST=California; L=San Francisco; O=Twitter, Inc.; OU=Twitter Security; CN=api.twitter.com
* start date: 2013-04-08 00:00:00 GMT
* expire date: 2013-12-31 23:59:59 GMT
* subjectAltName: api.twitter.com matched
* issuer: C=US; O=VeriSign, Inc.; OU=VeriSign Trust Network; OU=Terms of use at https://www.verisign.com/rpa (c)09; CN=VeriSign Class 3 Secure Server CA - G2
* SSL certificate verify ok.
> GET /1.1/search/tweets.json?include_entities=true&q=from%3Aplindelauf HTTP/1.1
> User-Agent: curl/7.24.0 (x86_64-apple-darwin12.0) libcurl/7.24.0 OpenSSL/0.9.8r zlib/1.2.5
> Host: api.twitter.com
> Accept: */*
> Authorization: OAuth oauth_consumer_key="DOP3re9nlTPeyemlmcqg", oauth_nonce="05d29110d282b24e516dae1ae2b54e1a", oauth_signature="k9gHhrHD86D5sB7zAlXCKsaeNZ8%3D", oauth_signature_method="HMAC-SHA1", oauth_timestamp="1370691632", oauth_token="288762088-DfqCgEIVvM56MSMZWjLuLl6oo3oaVkI0Z8ATKu3f", oauth_version="1.0"
>
< HTTP/1.1 200 OK
< cache-control: no-cache, no-store, must-revalidate, pre-check=0, post-check=0
< content-length: 14040
< content-type: application/json;charset=utf-8
< date: Sat, 08 Jun 2013 11:42:20 GMT
< expires: Tue, 31 Mar 1981 05:00:00 GMT
< last-modified: Sat, 08 Jun 2013 11:42:20 GMT
< pragma: no-cache
< server: tfe
< set-cookie: lang=en
< set-cookie: guest_id=v1%3A137069174010710877; Domain=.twitter.com; Path=/; Expires=Mon, 08-Jun-2015 11:42:20 UTC
< status: 200 OK
< strict-transport-security: max-age=631138519
< x-access-level: read
< x-frame-options: SAMEORIGIN
< x-rate-limit-limit: 180
< x-rate-limit-remaining: 179
< x-rate-limit-reset: 1370692640
< x-transaction: 4b4022555f454e2b
< x-xss-protection: 1; mode=block
...[the whole response body with all found tweets]...
I've also tried to see if it makes a difference when I send the query parameters separately, but that does not seem to be the case, since this curl command works just fine as well:
curl --get 'https://api.twitter.com/1.1/search/tweets.json?include_entities=true&q=from%3Aplindelauf' --header 'Authorization: OAuth oauth_consumer_key="DOP3re9nlTPeyemlmcqg", oauth_nonce="05d29110d282b24e516dae1ae2b54e1a", oauth_signature="k9gHhrHD86D5sB7zAlXCKsaeNZ8%3D", oauth_signature_method="HMAC-SHA1", oauth_timestamp="1370691632", oauth_token="288762088-DfqCgEIVvM56MSMZWjLuLl6oo3oaVkI0Z8ATKu3f", oauth_version="1.0"' --verbose
This has been driving me crazy for many days now! Can someone please help?!
I found the problem! I did not percent encode the query parameters in my HTTP request, like I did when creating the signature. Note that I was sending the query string q=from:plindelauf&include_entities=true. This needed to explicitly be include_entities=true&q=from%3Aplindelauf. Darn!

Resources