i'm trying to get data of "Export potential" platinum for South Africa from
https://exportpotential.intracen.org/en/markets/gap-chart?fromMarker=i&exporter=710&whatMarker=k&what=711019&toMarker=j
with Python requests.
I need the following data for each country.
Example data image
I think that I can get data from GET response, but if I explore Network tab in browser debugger, I see GET-request
Request URL: https://exportpotential.intracen.org/api/epis/targeted/from/product/to/exporter/marker/k/id/711019
Unfortunantly, the response is encrypted.
6iCAiWaW1XZJbKsS5qQ/xi/BAjjuW+be41Kv9tsJK8QuAo076JqB45lqPYR76rLsJjmRXBnrxExBXs5tJSfCeYpAPOrWE1raR+UyDJtujAgezfZV22PH16UI+RhwvBA26hcd8FvxLCCA1/DPDNNT6xKGVXi0Ykm7ih2mFcR3N5mq...
I am assuming it is encrypted with RSA algorithm because it looks like the following:
https://karatejb.blogspot.com/2020/02/aspnet-core-encrypt-and-decrypt-request.html
It looks like the encrypted message in my case
Probably I can use private-key and decrypt this data, but I don't know where find this key and I'm not sure about the algorithm. The question is not related to ITC site only because I met similar encrypted responces from many other sites.
There are two questions:
How to decipher this data with Python?
Where can I find the private-key to decrypt it? For purposes of simplicity I am going to use this facility: https://gchq.github.io/CyberChef/#recipe=RSA_Decrypt('-----BEGIN%20RSA%20PRIVATE%20KEY-----','','RSA-OAEP','SHA-1').
Related
I'm trying to POST "mutlipart\form-data" to my EC2 instance through AWS API Gateway, but I couldn't find a way to this. There is a way to post data using "application/x-www-form-urlencoded" and Mapping Tamplate to convert it to JSON but still posting a binary data like an image file is missing I guess. Is there anything I'm missing ?
EDIT:
I have found another way:
I convert the image to base64 string then POST it as with content type "application/x-www-form-urlencoded". By this way I'm sending whole image as string. After I got the message I can convert it back to image in PHP. Only down side of this I could find is when I convert image to base64 its size gets a bit bigger. Other than that, I couldnt find any other downside. If there is could you please share with me ?
Api Gateway team here.
Binary data isn't supported at the moment, but it's on our backlog. Several customers have requested this.
Some customers have had success using the base64 util in the mapping templates which may get it working for you: http://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-mapping-template-reference.html#util-template-reference
Other than that you'll have to wait for official support.
Edit
Binary support is finally here!!
Context: I'm updating my WordPress plugin to authenticate against the YouTube v3 API using a server key that has to be requested and entered by the user of the plugin.
Problem: I would like to perform validation of some kind on that key before using it, but can't seem to find documentation of the format a Google API server key adheres to. Based on (a very limited) number of examples it seems as though a key is:
is 39 characters long
is case-sensitive
consists of letters, numbers and at least dashes
So the question, obviously: Is this documented somewhere? Can anyone confirm or expand?
thanks,
frank
I couldn’t find any published key format either. Maybe because they want to keep the freedom to change the format in the future. If you want to be on the safe side, you should probably just do sanity checks well above the observed format. For example <=1024 bytes and non-control ascii characters, or even base64, or just don’t do any validation at all and let Google do that.
How about taking the key and passing it to a server-side script that attempts to use the key for some call. Then if it works return a success, else fail and call this async for the validation. Just seems more reliable than trying to decoded or anticipate the format of the hash.
My application has a weird problem. I have a login webservice which is used to authenticate the users, it works well for everyone except for a tester who is in Lebanon. For her, the request always fails. It turns out that the json response is not getting parsed for her.
My first guess was that her network place is using a proxy server that converts json to html, so I asked her to switch to cellular network but this isn't solving the problem either.
Please refer to the debug message in the screenshot below.
Any suggestions on what must be wrong will be greatly helpful.
You'd really need the exact data that was received. JSON parsing is totally independent of any localisation. On the other hand, whatever service produced the JSON data may not. There is a good chance that being in the Lebanon, that customer receives non-ASCII data (which should be just fine), while other customers don't. It is possible that the server sends that data not in UTF-8 but say in some Windows encoding. That would be fine for ASCII but not for non-ASCII data. Or it could be that the server figures out that full UTF-8 is needed and not ASCII, and transmits a byte order marker, which is not legal JSON and would likely create the error message that you received.
To reproduce, I'd try to set up things so that non-ASCII data would be used. For example a username or password with non-ASCII data.
I'm building an XML-based webservice in Rails to serve as the backend for an iPhone app, and I'm wondering how I can best achieve an auth scheme that will let me use both GET and POST requests -- i.e. one that doesn't require auth sent in the body of an XML payload.
The wrinkle here is that I'm not using regular HTTP auth. Instead, I'm creating a SHA1 digest of the iPhone's hardware ID (concatenated w/ a "secret" string pre-digest) along with the unhashed ID. I validate it on the server by attempting to re-create the digest w/ the hardware ID from the request and matching it against the hashed hardware ID from the request.
My question is this: should I create my service so that every action on every resource expects a payload of POSTed XML containing the security context in a common XML structure, or is there a better way to do it?
In other words, I'd like to use GET for things like /show, /index, etc. But as my app currently stands, I can't do that, since I need to send an XML payload containing the security context.
Perhaps there's a good way to achieve effectively the same thing with headers a la Google's web API's?
Every security context looks like this:
<request-wrapper>
<security-context>
<username>joefoo</username>
<hardware-id>AE7D128BCA9206E59901</hardware-id>
<hashed-hardware-id>cfd7983850301f97f6fdc26b553d1b6170f18bde</hashed-hardware-id>
</security-context>
...
(remainder of request payload)
...
</request-wrapper>
This is my first XML service in Rails, so I'd appreciate any general practice advice in this vein as well.
Thanks!
Your authentication scheme is subject to replay attacks if the "secret string" stays the same over the lifetime of the device.
Additionally, the "secret key" (if it is embedded in your application) can be dumped via strings (or other tool) breaking your scheme entirely.
I would instead use an asymmetric key to setup a one-time secret key, and then use it to hash a counter or something. If you need the hardware id for some reason, hash it plus the counter. This is basically a dumbed down SSL implementation, so you might as well just do that frankly (generating your own certificate, and doing the rare mutual authentication; but still...).
Remember, inventing your own security scheme is almost always a bad idea.
I'm thinking that it might be best to simply use custom headers for this and then access them in my controller filters w/ things like:
request.headers['username']
request.headers['hardware-id']
request.headers['hashedhardware-id']
Any thoughts on whether this is a good/bad idea?
How about creating an SHA1 digest of the entire XML request, instead of just the hardware-id? That way you're making replay attacks a lot harder. Sure, without a timestamp and (possibly) a nonce to make each request unique, a hacker could still replay the exact same request multiple times (maybe using up account credits or whatever), but at least they couldn't take the digest from an existing request and change the request details to make it do whatever they wanted.
Suggested steps:
Take your XML (without any hashed-hardware-id in it) and turn it into a byte array.
Create an SHA1 digest of the XML byte array.
Base-64 encode the XML byte array and the SHA1 digest byte array (separately).
Send the base-64-encoded XML as one request parameter, and the base-64-encoded signature as the other, either using GET or POST.
I am currently putting together a rails-based web application which will only serve and receive data via json and xml. However, some requirements contain the ability to upload binary data (images).
Now to my understanding JSON is not entirely meant for that... but how do you in general tackle the problem of receiving binary files/data over those two entrypoints to your application?
I suggest encoding the binary data in something like base64. This would make it safe to use in XML or JSON format.
http://en.wikipedia.org/wiki/Base64
maybe you could have a look on Base64 algorithm.
This is used to "transform" everything to ascii char.
You can code and decode it. It's used for webservices, or even on dotnet Serialization.
Hope this helps a little.
Edit: I saw "new post", while posting, someone was faster.Rails base64
If you are using Rails and json and xml than you are using HTTP. "POST" is a part of HTTP and is the best way to transform binary data. Base64 is a very inefficient way of doing this.
If your server is sending data, I would recommend putting a path to the file on the server in the XML or JSON. That way your server doesn't have to base64 encode the data and your client, which already supports HTTP GET, can pull down the data without decoding it. (GET /path/to/file)
For sending files, have your server and/or client generate a unique file name and use a two step process; the client will send the xml or json message with fileToBeUploaded: "name of file.ext" and after sending the message will POST the data with the aforementioned filename. Again, client and server won't have to encode and decode the data. This can be done with one request using a multi-part request.
Base64 is easy but will quickly chew up CPU and/or memory depending on the size of the data and frequency of requests. On the server-side, it's also not an operation which is cached whereas the operation of your web server reading the file from disk is.
If your images are not too large, putting them in the database with a RoR :binary type makes a lot of sense. If you have database replicas, the images get copied for free to the other sites, there's no concern about orphaned or widowed images, and the atomic transaction issues become far simpler.
On the other hand, Nessence is right that Base64, as with any encoding layer, does add network, memory and CPU load to the transactions. If network bandwidth is your top issue, make sure your web service accepts and offers deflate/gzip compressed connections. This will reduce the cost of the Base64 data on the network layer, albeit at the cost of even more memory and CPU load.
These are architectural issues that should be discussed with your team and/or client.
Finally, let me give you a heads up about RoR's XML REST support. The Rails :binary database type will become <object type="binary" encoding="base64">...</object> XML objects when you render to XML using code like this from the default scaffolding:
def show
#myobject = MyObject.find(:id)
respond_to do |format|
format.xml { render => #myobject }
end
end
This works great for GET operations, and the PUT and POST operations are about as easy to write. The catch is the Rails PUT and POST operations don't accept the same tags. This is because the from_xml code does not interpret the type="binary" tag, but instead looks for type="binaryBase64". There is a bug with a patch at the Rails lighthouse site to correct this.