What is the best way (considering both security and template re-use) to provide secrets like a host/port and external service username/password when the consumer doesn't expect or support a ValueProvider? By consumer I mean what is typically an IO class, such as MongoDbIO, ElasticsearchIO, etc.
I've already seen and successfully used this method: https://henrysuryawirawan.com/posts/dataflow-secret-manager/ but in that example, and some apache-provided IOs, the IO accepts the parameters in a ValueProvider. The apache ElasticsearchIO does not, neither does the MongoDbIO. The RedisIO does (via the RedisConnectionConfiguration). SpannerIO does.
So if the IO we want to use does not, is providing the config statically at template compile time the only option, and how is this typically done? Providing them dynamically seems safer security-wise, especially through something like Secret Manager.
Yes, SecretManager / KMS are likely the best ways to deal here. But unfortunately, your options are limited if ValueProviders are not allowed.
You may use Flex Templates to work around the need of ValueProviders.
We have a probably useful example in DataflowTemplates: KMSUtils.java, which expands options using KMS during runtime.
Related
I would like to have a policy where the decision has to be made by querying a field against a system of record which doesnot have a http api. Is there a way for a policy to call an external program and use its return code to decide whether to allow/deny?
Looking at https://www.openpolicyagent.org/docs/latest/policy-reference/#built-in-functions, I am not finding a function to use. Is there a function I am missing?
OPA does not provide a way to directly call external programs in the policy evaluation.
There are maybe a few things I'd recommend considering:
Call the external program before doing an evaluation with OPA that requires the data, and provide the result from the program as part of the input.
Implement a custom builtin function. Check out https://www.openpolicyagent.org/docs/latest/extensions/ for a couple of options
(depending on how you are integrating OPA).
You could implement a lightweight HTTP service which wraps making the call to the external program (eg simple python or golang app that exec's the other program and returns the data for use by the http.send builtin)
I have used Spring Rabbitmq XML configuration in my project. To create the RabbitConnectionFactoryBean, we provide the ssl.properties file resource with below properties
keyStore=file:/secret/keycert.p12
trustStore=file:/secret/trustStore
keyStore.passPhrase=secret
trustStore.passPhrase=secret
The passPhrases are hardcoded values. We are worried that any one who gains access to the system can read this file and misuse it. It is true that the system admin and OS should protect these files, but this can be considered a security threat when untrusted user logs in.
In this link Gary suggests to use Java configuration and we can use that to create the RabbitConnectionFactoryBean and maybe read the encrypted passwords from system and use the setter to set them in the bean.
But since we are already using XML configuration, is there any other ways to secure the passPhrases in the properties file?
Will this same configuration cause similar security issue in the PRODUCTION environment.?
Kindly help me on how to achieve security on the above.
Not sure if that is legal to say in the public, but I'll try.
There is some security token approach, when you start your project it requests such a token. An admin (or security representative) comes, inserts some special flesh drive, enter passwords. Your project reads properties from that device, populates all the properties and starts properly. That admin pulls flesh drive from USB and goes away. No one see password for your application!
The other solution you could consider is something like Config Server. So, your properties are stored somewhere outside of the current machine.
You also can really follow encryption way as well: http://cloud.spring.io/spring-cloud-static/Finchley.RELEASE/multi/multi__spring_cloud_context_application_context_services.html#_encryption_and_decryption
I've searched for this a bit on Stack, but I cannot find a definitive answer for https, only for solutions that somehow include http or unencrypted parameters which are not present in my situation.
I have developed an iOS application that communicates with MySQL via Apache HTTPS POSTS and php.
Now, the server runs with a valid certificate, is only open for traffic on port 443 and all posts are done to https://thedomain.net/obscurefolder/obscurefile.php
If someone knew the correct parameters to post, anyone from anywhere in the world could mess up the database completely, so the question is: Is this method secure? Let it be known nobody has access to the source code and none of the iPads that run this software are jailbreaked or otherwise compromised.
Edit in response to answers:
There are several php files which alone only support one specific operation and depend on very strict input formatting and correct license key (retreived by SQL on every query). They do not respond to input at all unless it's 100% correct and has a proper license (e.g. password) included. There is no actual website, only php files that respond to POSTs, given the correct input, as mentioned above. The webserver has been scanned by a third party security company and contains no known vulnerabilities.
Encryption is necessary but not sufficient for security. There are many other considerations beyond encrypting the connection. With server-side certificates, you can confirm the identity of the server, but you can't (as you are discovering) confirm the identity of the clients (at least not without client-side certficates which are very difficult to protect by virtue of them being on the client).
It sounds like you need to take additional measures to prevent abuse such as:
Only supporting a sane, limited, well-defined set of operations on the database (not passing arbitrary SQL input to your database but instead having a clear, small list of URL handlers that perform specific, reasonable operations on the database).
Validating that the inputs to your handler are reasonable and within allowable parameters.
Authenticating client applications to the best you are able (e.g. with client IDs or other tokens) to restrict the capabilities on a per-client basis and detect anomalous usage patterns for a given client.
Authenticating users to ensure that only authorized users can make the appropriate modifications.
You should also probably get a security expert to review your code and/or hire someone to perform penetration testing on your website to see what vulnerabilities they can uncover.
Sending POST requests is not a secure way of communicating with a server. Inspite of no access to code or valid devices, it still leaves an open way to easily access database and manipulating with it once the link is discovered.
I would not suggest using POST. You can try / use other communication ways if you want to send / fetch data from the server. Encrypting the parameters can also be helpful here though it would increase the code a bit due to encryption-decryption logic.
Its good that your app goes through HTTPS. Make sure the app checks for the certificates during its communication phase.
You can also make use of tokens(Not device tokens) during transactions. This might be a bit complex, but offers more safety.
The solutions and ways here for this are broad. Every possible solution cannot be covered. You might want to try out a few yourself to get an idea. Though I Suggest going for some encryption-decryption on a basic level.
Hope this helps.
While developing an desktop application that needs to access twitter API , one must somehow pass the API key (application specific consumer key and consumer secret ) for the application to the user. Twitter's API TOS states that the application's API key cannot be publicly available and if that happens, they reset it. When that application is under GPL , meaning the developer needs to provide the source code to the user, how that user would be able to obtain the API key without it being publicly available ? Is there a standard way to handle this issue ?
Thanks.
Edit:
To clarify the situation, I was storing them in plain text in my code for cree.py so far as a conscious decision. But yesterday Twitter support team contacted me that they have reseted my key and their reasoning was the following :
C. You should not solicit another developer's consumer keys or consumer secrets especially if they will be stored or used for actions outside of that developer's control. Keys and secrets that are compromised will be reset by Twitter. For example, online services that ask for these values in order to provide a "tweet-branding" service are not allowed.
https://dev.twitter.com/terms/api-terms
If an application's keys are posted publicly, it allows for external parties to hijack the application's API access. This presents an enormous abuse risk, and as such we've reset your API keys. Please take care to ensure that these keys are not posted publicly again.
Thanks,
Twitter API Policy
Well, TTYtter evidently uses the honour system:
# yes, this is plaintext. obfuscation would be ludicrously easy to crack,
# and there is no way to hide them effectively or fully in a Perl script.
# so be a good neighbour and leave this the fark alone, okay? stealing
# credentials is mean and inconvenient to users. this is blessed by
# arrangement with Twitter. don't be a d*ck. thanks for your cooperation.
$oauthkey = (!length($oauthkey) || $oauthkey eq 'X') ?
"XXXXXXXXXXXXXXXXXXXXX" : $oauthkey;
$oauthsecret = (!length($oauthsecret) || $oauthsecret eq 'X') ?
"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX" : $oauthsecret;
(I have replaced the actual keys with Xs, to make it a little less likely that anyone will go to the trouble to abuse them, but rest assured that they are present in full in the actual source!)
Also, I don't see anything in the Rules of the Road actually requiring you to keep these things secret: the closest thing I see is the statement "Keys and secrets that are compromised will be reset by Twitter."; they never actually say what "compromised" means, though.
I might be dense here, but why don't you store them in a configuration file, the Windows registry etc and get them from there? Then distribute the application without the file, and you're done.
Maybe another solution would be to use a server, the server interacts with the twitter api, and the you request information to your server with your desktop application
Like that, the API key is only stored on the server, and not any user can get it.
I'm trying to find my way around the OAuth spec, its requirements and any implementations I can find and, so far, it really seems like more trouble than its worth because I'm having trouble finding a single resource that pulls it all together. Or maybe it's just that I'm looking for something more specialized than most tutorials.
I have a set of existing APIs--some in Java, some in PHP--that I now need to secure and, for a number of reasons, OAuth seems like the right way to go. Unfortunately, my inability to track down the right resources to help me get a provider up and running is challenging that theory. Since most of this will be system-to-system API usage, I'll need to implement a 2-legged provider. With that in mind...
Does anyone know of any good tutorials for implementing a 2-legged OAuth provider with PHP?
Given that I have securable APIs in 2 languages, do I need to implement a provider in both or is there a way to create the provider as a "front controller" that I can funnel all requests through?
When securing PHP services, for example, do I have to secure each API individually by including the requisite provider resources on each?
Thanks for your help.
Rob, not sure where you landed on this but wanted to add my 2 cents in case anyone else ran across this question.
I more or less had the same question a few months ago and hearing about "OAuth" for the better part of a year. I was developing a REST API I needed to secure so I started reading about OAuth... and then my eyes started to roll backwards in my head.
I probably gave it a good solid day or 2 of skimming and reading until I decided, much like you, that OAuth was confusing garbage and just gave up on it.
So then I started researching ways to secure APIs in general and started to get a better grasp on ways to do that. The most popular way seemed to be sending requests to the API along with a checksum of the entire message (encoded with a secret that only you and the server know) that the server can use to decide if the message had been tampered with on it's way from the client, like so:
Client sends /user.json/123?showFriends=true&showStats=true&checksum=kjDSiuas98SD987ad
Server gets all that, looks up user "123" in database, loads his secret key and then (using the same method the client used) re-calculates it's OWN checksum given the request arguments.
If the server's generated checksum and the client's sent checksum match up, the request is OK and executed, if not, it is considered tampered with and rejected.
The checksum is called an HMAC and if you want a good example of this, it is what Amazon Web Services uses (they call the argument 'signature' not 'checksum' though).
So given that one of the key components of this to work is that the client and server have to generate the HMAC in the same fashion (otherwise they won't match), there have to be rules on HOW to combine all the arguments... then I suddenly understood all that "natural byte-ordering of parameters" crap from OAuth... it was just defining the rules for how to generate the signature because it needed to.
Another point is that every param you include in the HMAC generation is a value that then can't be tampered with when you send the request.
So if you just encode the URI stem as the signature, for example:
/user.json == askJdla9/kjdas+Askj2l8add
then the only thing in your message that cannot be tampered with is the URI, all of the arguments can be tampered with because they aren't part of the "checksum" value that the server will re-calculate.
Alternatively, even if you include EVERY param in the calculation, you still run the risk of "replay attacks" where a malicious middle man or evesdropped can intercept an API call and just keep resending it to the server over and over again.
You can fix that by adding a timestamp (always use UTC) in the HMAC calculation as well.
REMINDER: Since the server needs to calculate the same HMAC, you have to send along any value you use in the calculation EXCEPT YOUR SECRET KEY (OAuth calls it a consumer_secret I think). So if you add timestamp, make sure you send a timestamp param along with your request.
If you want to make the API secure from replay attacks, you can use a nonce value (it's a 1-time use value the server generates, gives to the client, the client uses it in the HMAC, sends back the request, the server confirms and then marks that nonce value as "used" in the DB and never lets another request use it again).
NOTE: 'nonce' are a really exact way to solve the "replay attack" problem -- timestamps are great, but because computers don't always have in-sync timestamp values, you have to allow an acceptable window on the server side of how "old" a request might be (say 10 mins, 30 mins, 1hr.... Amazon uses 15mins) before we accept or reject it. In this scenario your API is technically vulnerable during the entire window of time.
I think nonce values are great, but should only need to be used in APIs that are critical they keep their integrity. In my API, I didn't need it, but it would be trivial to add later if users demanded it... I would literally just need to add a "nonce" table in my DB, expose a new API to clients like:
/nonce.json
and then when they send that back to me in the HMAC calculation, I would need to check the DB to make sure it had never been used before and once used, mark it as such in the DB so if a request EVER came in again with that same nonce I would reject it.
Summary
Anyway, to make a long story short, everything I just described is basically what is known as "2-legged OAuth". There isn't that added step of flowing to the authority (Twitter, Facebook, Google, whatever) to authorize the client, that step is removed and instead the server implicitly trusts the client IF the HMAC's they are sending match up. That means the client has the right secret_key and is signing it's messages with it, so the server trusts it.
If you start looking around online, this seems to be the preferred method for securing API methods now-adays, or something like it. Amazon almost exactly uses this method except they use a slightly different combination method for their parameters before signing the whole thing to generate the HMAC.
If you are interested I wrote up this entire journey and thought-process as I was learning it. That might help provide a guided thinking tour of this process.
I would take a step back and think about what a properly authenticated client is going to be sending you.
Can you store the keys and credentials in a common database which is accessible from both sets of services, and just implement the OAuth provider in one language? When the user sends in a request to a service (PHP or Java) you then check against the common store. When the user is setting up the OAuth client then you do all of that through either a PHP or Java app (your preference), and store the credentials in the common DB.
There are some Oauth providers written in other languages that you might want to take a look at:
PHP - http://term.ie/oauth/example/ (see bottom of page)
Ruby - http://github.com/mojodna/sample-oauth-provider
.NET http://blog.bittercoder.com/PermaLink,guid,0d080a15-b412-48cf-b0d4-e842b25e3813.aspx