f5 LTM irule - can a pool name be generated in an irule - f5

I need to setup a configuration for many similar environments. Each will have a different hostname that follows a pattern, e.g. env1, env2, etc.
I can use a pool per environment and a single virtual server with an irule that selects a pool based on hostname.
What I'd prefer to do is dynamically generate and select the pool name based on the requested hostname rather than listing out every pool in the switch statement. It's easier to maintain and automatically handles new environments.
The code might look like:
when HTTP_REQUEST {
pool [string tolower [HTTP:host]]
}
and each pool name matches the hostname.
Is this possible? Or is there a better method?
EDIT
I've expanded my hostname pool selection. I'm now trying to include the port number. The new rule looks like:
when HTTP_REQUEST {
set lb_port "[LB::server port]"
set hostname "[string tolower [getfield [HTTP::host] : 1]]"
log local0.info "Pool name $hostname-$lb_port-pool"
pool "$hostname-$lb_port-pool"
}
This is working, but I'm seeing no-such-pool errors in the logs because somehow a port 0 request is coming into the pool. It seems to be the first request and the followed by the request with the legitimate port.
Wed Feb 17 20:39:14 EST 2016 info tmm tmm[6519] Rule /Common/one-auto-pool-select-by-hostname-port <HTTP_REQUEST>: Pool name my.example.com-80-pool
Wed Feb 17 20:39:14 EST 2016 err tmm1 tmm[6519] 01220001 TCL error: /Common/one-auto-pool-select-by-hostname-port <HTTP_REQUEST> - no such pool: my.example.com-0-pool (line 1) invoked from within "pool "$hostname-$lb_port-pool""
Wed Feb 17 20:39:14 EST 2016 info tmm1 tmm[6519] Rule /Common/one-auto-pool-select-by-hostname-port <HTTP_REQUEST>: Pool name my.example.com-0-pool
What is causing the port 0 request? And is there any workaround? e.g. could I test for port 0 and select a default port or ignore it?
ONE MORE EDIT
Rebuilt the virtual server, and now the error has gone. The rebuild of the VS was just to rename it though. I'm fairly sure I recreated the settings exactly the same.

Yes, you can specify the pool name in a string. What you have there would work as long as you have a pool with that same name. Though it doesn't show an example of doing it this way, you can also check out the pool wiki page on DevCentral for more information.
As an aside, in my environment I generally create pools with the suffix _pool to distinguish them from other objects when looking at config files. So in my iRules, I would do something like this (essentially the same thing):
when HTTP_REQUEST {
pool "[string tolower [HTTP::host]]_pool"
}

The simple case mentioned by Michael works. I'd recommend removing the port value if present:
when HTTP_REQUEST {
pool "pool_[string tolower [getfield [HTTP::host] : 1]]_[LB::server port]"
}
Keep in mind that clients might send a partial hostname. If the DNS search path is set to example.org then the client might hit shared/ which maps to shared.example.org, but the HTTP::host header will just have shared. Some API libraries may append the port number even if it's on the default port. Simple code might not send a Host header. Malicious code might send completely bogus Host headers. You could trap these cases with catch.
You can also use a datagroup to map hostnames to pools. This allows multiple hosts to use the same pool. Sample code:
when HTTP_REQUEST {
set host [string tolower [getfield [HTTP::host] ":" 1]]
if { $host == "" } {
# if there's no Host header, pull from virtual server name
# we use: pool_<virtualserver>_PROTOCOL
set host [getfield [virtual name] _ 2]
} elseif { not ($host contains ".") } {
# if Host header does not contain a dot, assume example.org
set host $host.example.org
}
set pool [class match -value $host[HTTP::uri] starts_with dg_shared.example.org]
if { $pool ne ""} {
set matched [class match -name $host[HTTP::uri] starts_with dg_shared.example.org]
set log(matched) $matched
set log(pool) $pool
if { [catch { pool $pool } ] } {
set log(reason) "Failed to Connect to Pool"
call hsllog log
call errorpage 404 $log(reason) "https://[HTTP::host][HTTP::uri]" log
}
} else {
call errorpage 404 "No Pool Found" "https://[HTTP::host][HTTP::uri]" log
}
}
when SERVER_CONNECTED {
if {!($pool ends_with "_HTTPS") } {
SSL::disable serverside
}
}
This allows host.example.org/path1 to be on a different pool than host.example.org or host.example.org/path2 by including separate entries in the datagroup. I didn't include the hsllog and errorpage procs here. They dump the log array as well as the other passed parameters.
We then disable serverside ssl for pools that don't end in _HTTPS.
Note: As with dynamically generated pool names, the BIG-IP UI does not look inside datagroups for pool references, so the interface will allow you do delete one of these pools thinking it's not in use.
We use BigIPReport to identify orphan pools:
https://devcentral.f5.com/s/articles/bigip-report

Related

Nagios/Icinga - find host by custom variable with cmd/status.cgi

I'm looking for solution how to get a hostname from Nagios/Icinga by searching it by custom variable with cmd/status.cgi.
I have a custom variable with unique specific IDs on every host. I have to get the hostname by searching on ID. There is a documentation for CGI commands but I could not find the needed functionality: https://icinga.com/docs/icinga1/latest/en/cgiparams.html
UPD: I am using python for CGI requests. Maybe there is also a library to do that.
Does anyone know, if it is possible?
For Nagios at least, this is possible. You can call the host details on the objectjson.cgi for a hostgroup and in your result.json(), you would have the custom_variables for each of the hosts. With that, you can map an ID to the hostname.
make your request to https://<your_url>/nagios/cgi-bin/objectjson.cgi?query=hostlist&details=true&hostgroup=<your_hostgroup>
{...
"data": {
"hostlist": {
"<host1>": {
....
"custom_variables": {
<custom host variables dict>
},
"<host2>": {
....
}
}
}
}
untested! using python's requests module:
hostlist = result.json().get('data').get('hostlist')
id_map = {hostlist.get(host).get('custom_variables').get('your_id_key'):host for host in hostlist.keys()}

MBeanServerConnection.invoke hangs forever

We have an app that invokes various remote methods on MBeans using MBeanServerConnection.invoke.
Occasionally one of these methods hangs.
Is there any way to have a timeout on the call? so that it will return with an exception if the call takes too long?
Or do I have to move all those calls into separate threads so they don't lock up the UI and require killing the app?
See http://weblogs.java.net/blog/emcmanus/archive/2007/05/making_a_jmx_co.html
===== Update =====
I was thinking about this stuff when I first responded, but I was on my mobile and I can't type worth a damn on it.....
This is really an RMI problem, and unless you use a different protocol, there's not much you can do, except, as you say, move all those calls into separate threads so they don't lock up the UI.
But.... if you have the option of fiddling with the target server and you can customize the connecting client, you have at least 1 option which is to customize the JMXConnectorServer on your target servers.
The standard JMXConnectorServer implementation is the RMIConnectorServer. Part of it's specification is that when you create a new instance using any of the constructors (like RMIConnectorServer(JMXServiceURL url, Map environment)), the environment map can contain a key/value pair where the key is RMIConnectorServer.RMI_CLIENT_SOCKET_FACTORY_ATTRIBUTE and the value is a RMIClientSocketFactory. Therefore, you can specify a socket factory method like this:
RMIClientSocketFactory clientSocketFatory = new RMIClientSocketFactory() {
public Socket createSocket(String host, int port) {
Socket s = new Socket(host, port);
s.setSoTimeout(3000);
}
};
This factory creates a Socket and then sets its SO_TIMEOUT using setSoTimeout, so when the client connects using this socket, all operations, including connecting, will timeout after 3000 ms.
You could also checkout the JMXMP connector and server in the jmx-optional package of the OpenDMK. (links are to my github mavenized). No built in solution, mind you, but they're super easy to extend and JMXMP is simple TCP socket based rather than RMI, so this type of customization would be trivial.
Cheers.
# Nicholas : The above code is not working.I mean request is not getting timeout after 3000. ms.
map.put(RMIConnectorServer.RMI_CLIENT_SOCKET_FACTORY_ATTRIBUTE , new RMIClientSocketFactory() {
#Override
public Socket createSocket(String host, int port) throws IOException {
if(logger.isInfoEnabled() ){
logger.info("JMXManager inside createSocket..." + host + ": port :" + port);
}
Socket s = new Socket(host, port);
s.setSoTimeout(3000);
return s;
}
});
cs = JMXConnectorServerFactory.newJMXConnectorServer(url,map,mbeanServer);
As I answered on: How to set request timeout for JMX Connector the RMI properties can help you. All the properties are on Oracle documentation site:
http://docs.oracle.com/javase/7/docs/technotes/guides/rmi/sunrmiproperties.html.
For example: -Dsun.rmi.transport.tcp.responseTimeout=60000 is a client side tcp response timeout. There are also properties for connect timeout and for server side connections.
I also am not happy how the JMX/RMI/TCP stack hides important settings from lower level protocols, and makes it not available for a single connection.

Failed to Connect with Websphere MQ SSL Channel through JNDI

My JMS client connects to WMQ through JNDI. The initial context factory used is com.ibm.mq.jms.context.WMQInitialContextFactory.
Currently, at WMQ side, there's a queue manager called TestMgr. Under this queue manager I created two channels. One is PLAIN.CHL which does not specify an SSL Cipher Spec, the other one is SSL.CHL which configured SSL Cipher Spec with RC4_MD5_US and SSL Authentication with Optional.
I have created a key store for the queue manager using IBM Key Management tool. The path of key db is [wmq_home]\qmgrs\TestMgr\ssl\key.
For channel PLAIN.CHL, I defined a queue connection factory like:
DEF QCF(PlainQCF) QMANAGER(TestMgr) CHANNEL(PLAIN.CHL) HOST(192.168.66.23) PORT(1414) TRANSPORT(client)
And under the SSL channel SSL.CHL, I defined a queue connection factory like:
DEF QCF(SSLQCF) QMANAGER(TestMgr) CHANNEL(SSL.CHL) HOST(192.168.66.23) PORT(1414) TRANSPORT(client) SSLCIPHERSUITE(SSL_RSA_WITH_RC4_128_MD5)
Now I only can create connection using the PlainQCF. But failed to look up the SSL queue connection factory. My code looks like:
Hashtable environment = new Hashtable();
environment.put(Context.INITIAL_CONTEXT_FACTORY, "com.ibm.mq.jms.context.WMQInitialContextFactory");
environment.put(Context.PROVIDER_URL, "192.168.66.23:1414/SSL.CHL");
Context ctx = new InitialContext( environment );
QueueConnectionFactory qcf = (QueueConnectionFactory) ctx.lookup("SSLQCF");
qcf.createConnection();
....
Am I missing some context properties when looking up the SSL factory? connection And then I found the code is hanging on the line new InitialContext( environment ) for a long time, almost 5 minutes, and I got CC=2;RC=2009;AMQ9208... error.
Any suggestion would be appreciated. Is it true that SSL channel can't be connected by JNDI?
#T.Rob, thanks for your reply very much. But we still want to use WMQInitialContextFactory, so I'm afraid I still need to find solution for this.
I just defined the connection factory one time. The displayed info for the SSL queue connection factory like:
InitCtx> DISPLAY QCF(SSLQCF)
ASYNCEXCEPTION(ALL)
CCSID(819)
CHANNEL(SSL.CHL)
CLIENTRECONNECTOPTIONS(ASDEF)
CLIENTRECONNECTTIMEOUT(1800)
COMPHDR(NONE )
COMPMSG(NONE )
CONNECTIONNAMELIST(192.168.66.23(1414))
CONNOPT(STANDARD)
FAILIFQUIESCE(YES)
HOSTNAME(192.168.66.23)
LOCALADDRESS()
MAPNAMESTYLE(STANDARD)
MSGBATCHSZ(10)
MSGRETENTION(YES)
POLLINGINT(5000)
PORT(1414)
PROVIDERVERSION(UNSPECIFIED)
QMANAGER(TestMgr)
RESCANINT(5000)
SENDCHECKCOUNT(0)
SHARECONVALLOWED(YES)
SSLCIPHERSUITE(SSL_RSA_WITH_RC4_128_MD5)
SSLFIPSREQUIRED(NO)
SSLRESETCOUNT(0)
SYNCPOINTALLGETS(NO)
TARGCLIENTMATCHING(YES)
TEMPMODEL(SYSTEM.DEFAULT.MODEL.QUEUE)
TEMPQPREFIX()
TRANSPORT(CLIENT)
USECONNPOOLING(YES)
VERSION(7)
WILDCARDFORMAT(TOPIC_ONLY)
The JNDI Provider should be fine because I can look up the plain connection factory successfully. Also, for my client app, I extracted the cert from the key store which created for MQ server and imported it to the trust store(cacerts) of my JRE with alias name ibmwebspheremqtestmgr.
You are correct, with 2009 error there are some log entries:
=================================================================
4/20/2012 20:24:27 - Process(13768.3) User(MUSR_MQADMIN) Program(amqzmur0.exe)
Host(xxxx_host of my MQ) Installation(mqenv)
VRMF(7.1.0.0) QMgr(TestMgr)
AMQ6287: WebSphere MQ V7.1.0.0 (p000-L111019).
EXPLANATION:
WebSphere MQ system information:
Host Info :- Windows Server 2003, Build 3790: SP2 (MQ Windows 32-bit)
Installation :- C:\IBM\WebSphereMQ (mqenv)
Version :- 7.1.0.0 (p000-L111019)
ACTION:
None.
-------------------------------------------------------------------------------
4/20/2012 20:24:27 - Process(7348.116) User(MUSR_MQADMIN) Program(amqrmppa.exe)
Host(xxxx_host of my MQ) Installation(mqenv)
VRMF(7.1.0.0) QMgr(TestMgr)
AMQ9639: Remote channel 'SSL.CHL' did not specify a CipherSpec.
EXPLANATION:
Remote channel 'SSL.CHL' did not specify a CipherSpec when the local channel
expected one to be specified.
The remote host is 'xxx_host of my app (192.168.66.25)'.
The channel did not start.
ACTION:
Change the remote channel 'SSL.CHL' on host 'xxx_host of my app (192.168.66.25)' to
specify a CipherSpec so that both ends of the channel have matching
CipherSpecs.
----- amqcccxa.c : 3817 -------------------------------------------------------
4/20/2012 20:24:27 - Process(7348.116) User(MUSR_MQADMIN) Program(amqrmppa.exe)
Host(my app host) Installation(mqenv)
VRMF(7.1.0.0) QMgr(TestMgr)
AMQ9999: Channel 'SSL.CHL' to host 'xxx_host of my app (192.168.66.25)' ended
abnormally.
====================================================================
I also got some confusion with the error log. My app staged at at a machine which is different from my MQ. But the log says the Change the remote channel 'SSL.CHL' on host 'xxx_host of my app (192.168.66.25)' to
specify a CipherSpec so that both ends of the channel have matching
CipherSpecs. How can I change the channel cipher spec on my app host?
updates on MQEnvironment...
reply the comments.
The value of MQEnvironment.sslCipherSuite is null, so it throws out NullPointerExcetpion when i put it the the env hashtable. But i tried another one environment.put(MQC.SSL_CIPHER_SUITE_PROPERTY, "SSL_RSA_WITH_RC4_128_MD5") and it still failed with 2009 error.
For JMSAdmin tool, i had changed the config to use WMQInitialContextFactory. The configuration like(JMSAdmin.config):
INITIAL_CONTEXT_FACTORY=com.ibm.mq.jms.context.WMQInitialContextFactory
PROVIDER_URL=192.168.66.23:1414/SYSTEM.DEF.SVRCONN
The rest configuration leaves as default.
Kindly note, here i use the default channel SYSTEM.DEF.SVRCONN so that i can logon to admin console. If I change the channel to the SSL oneSSL.CHL, I also can't logon to admin console. The error happened here is just like the one in my client app.
Another clarification, in my client, i use follow code can connect to connect qmgr(TestMgr) successfully through channel SSL.CHL.
MQConnectionFactory factory = new MQConnectionFactory();
factory.setTransportType(JMSC.MQJMS_TP_CLIENT_MQ_TCPIP);
factory.setQueueManager("TestMgr");
factory.setSSLCipherSuite("SSL_RSA_WITH_RC4_128_MD5");
factory.setPort(1414);
factory.setHostName("192.168.66.23");
factory.setChannel("SSL.CHL");
MQConnection connection = (MQConnection) factory.createConnection();
And now the problem is just like you said, that's the initial context failed connect to qmgr through SSL channel. The option(use plain channel for initial context and ssl channel for connection factory) you provided works too. But I still want to know how to get initial context with ssl channel work. Thanks for you patience very much. Your updates will be appreciated.
thanks
I never really liked com.ibm.mq.jms.context.WMQInitialContextFactory very much. It stores the managed objects on a queue. So in order to lookup the connectionFactory, which tells JMS how to connect to the QMgr, it is first necessary to connect to the QMgr to make the JNDI call. Therefore, before you can debug the SSL connection, you need to know whether the underlying JNDI provider is working.
If you want to skip the MQ-based JNDI provider and just use the filesystem, see the updated version of Bobby Woolf's article here. If you want to continue with com.ibm.mq.jms.context.WMQInitialContextFactory, read on but be prepared to provide more configuration info.
When you run the JMSAdmin tool, do you display the objects after creating them? For example, here is one of my JMSAdmin.bat scripts:
# Connection Factory for Client mode
# Delete the Connection Factory if it exists
DELETE CF(JMSDEMOCF)
# Define the Connection Factory
DEFINE CF(JMSDEMOCF) +
SYNCPOINTALLGETS(YES) +
SSLCIPHERSUITE(NULL_SHA) +
TRAN(client) +
HOST(127.0.0.1) CHAN(SSL.SVRCONN) PORT(1414) +
QMGR( )
# Display the resulting definition
DISPLAY CF(JMSDEMOCF)
This deletes the object (because JMSAdmin doesn't have a define with replace option) then defines the object, then displays it. Do you in fact see both objects defined? Can you connect and interactively display them both? Can you update your question with the contents displayed?
If so, then what does the JNDI provider configuration look like with each sample program? The 2009 indicates that there is at least a connection to the QMgr being made, so it is important to determine whether the thing that suffering the broken connection is your app or the JNDI provider. To diagnose that requires the config info you are using for the JNDI provider and whether it is the same in the working and failing cases. If not, how do they differ?
Once you know whether it's the app or the JNDI provider that is causing the problem (or switch to another JNDI provider that doesn't require an MQ connection such as the filesystem initial context) then it will be possible to determine the next steps.
The article linked above has samples of code and managed object scripts that use a filesystem JNDI provider. You may notice my scripts pasted in above use the same QMgr name. That's because I wrote that part of the article. When I want to switch to SSL using those same samples, I just update the connectionFactory to point to the SSL channel and it works.
Here are the other bits from the sample that I've modified:
java -Djavax.net.debug=ssl ^
-Djavax.net.ssl.trustStore=key2.jks ^
-Djavax.net.ssl.keyStore=key2.jks ^
-Djavax.net.ssl.keyStorePassword=???????? ^
-Djavax.net.ssl.trustStorePassword=???????? ^
-cp "%CLASSPATH%" ^
com.ibm.examples.JMSDemo -pub -topic JMSDEMOPubTopic %*
Note: The ^ is Windows version of line continuation.
Then if there are problems, I follow the debugging scenario I described in this SO answer. Note that the app will require a truststore, even if you have SSLCAUTH(OPTIONAL) on your channel. This is because the app must always validate the QMgr's certificate, even if the app does not present its own certificate. In my case I was using SSLCAUTH(REQUIRED) so my app needed both a keystore and a truststore. Your question mentions that the QMgr has a keystore but does not say what you did for the application.
Finally, a 2009 will usually generate an entry in the QMgr error logs. If you continue to get the problem, please update your question with those log entries.
UPDATE:
Responding to the comments, the JMSAdmin tool is part of the WMQ package. However, WMQ it comes with jars for filesystem context and LDAP context. The WMQInitialContextFactory is optional and is delivered as SupportPac ME01. When using WMQInitialContextFactory with the JMSAdmin tool (or the JMSAdmin GUI or with WMQ Explorer) it is necessary to configure the PROVIDER_URL with the host, port and channel. For example:
PROVIDER_URL: <Hostname>:<port>/<SVRCONN Channel Name>
192.168.66.23:1414/SSL.SVRCONN
So after reviewing your post again, I realized that you did provide the config info for WMQInitialContextFactory. I was looking for a JMSADmin.config file but you have it in the environment hash table. And that is where the problem is. You are attempting to use the SSL channel for both the WMQInitialContextFactory and the connection factory. This is what is causing the lookup to fail. The WMQInitialContextFactory first makes a Java connection to the QMgre in order to look in the queue to obtain the administered objects such as QCF. In order to do that, it needs to know the ciphersuite that the channel is set up for in order to negotiate the handshake. Right now, the *only * place that ciphersuite is recorded is in the QCF definition.
Try adding the following line:
environment.put(MQEnvironment.sslCipherSuite, "SSL_RSA_WITH_RC4_128_MD5");
As per this Infocenter page, that should tell the context factory classes what ciphersuite to use. Of course, they also need to know where the trust store is (and possibly keystore if the channel has SSLCAUTH(RQUIRED) set) so you still need to get those values in the environment. You can use the command-line variables or try loading them into the environment using code. You'll need both -Djavax.net.ssl.trustStore=key2.jks and -Djavax.net.ssl.trustStorePassword=????????.
The other option is to continue to use the plaintext channel for the WMQInitialContextFactory and the SSL channel for the application. If the plaintext channel has an MCAUSER for a non-privileged user ID, it can be restricted to only connect to the QMgr and access the queue that contains the administered objects. With those restrictions, anyone will be able to read the administered objects using that channel but not the application queues or administrative queues.

How to block bad unidentified bots crawling my website?

How can I resist the bad unidentified bots to crawl my website? Some bad bots whose name is not present in cPanel of Apache are badly accessing my website bandwidth.
I had tried robots.txt on batgap.com/robots.txt and also blocked with .htaccess but there is no improvement in bandwidth usage. I don't know the IP of those bots so unable to block them by IP address. These bots are consuming too much bandwidth of site and hence a result I need to increase it from server.
I'm from Incapsula and we deal with bad bots on a regular basis.
We've recently release a bot-related research that provides insights of the scope of the problem ( http://www.incapsula.com/the-incapsula-blog/item/225-what-google-doesnt-show-you-31-of-website-traffic-can-harm-your-business ) and in light of this data I have to agree with #Leonard Challis - you simply can not handle bot protection manually.
Having said that, there are bot protection solutions, even Free ones (us included) that can help you with bad bots.
BTW - Just like you mentioned, one byproduct of bad bots visits is a loss of bandwidth.
We`ve recently became aware of just how surprisingly HUGE bot-related bandwidth usage really is.
This is an interesting topic by itself.
We believe that by avoiding bad bot traffic, hosting providers can actually greatly improve their efficiency (hopefully using this to drop cost or to improve services). Once you imagine Social and Business implication of this you can understand the real scope of this bad bot problem that goes way beyond the immediate damage done.
I block 'bad bots' by using PHP.
I filter in IP address primarily, then by User-Agent secondarily.
I make the 'bad bot' wait for up to 999 seconds, then return a very small web page.
Usually (always) the internet connection times-out and zero (0) bytes are returned.
Best of all I have delayed them for a few minutes before the get to the next victim.
http://gelm.net/How-to-block-Baidu-with-PHP.htm
Unfortunately robots.txt is sometimes ignored by these "bad bots", though if the problem is more things like genuine search engine spiders that you don't want to see they ought to take it in to account. I presume with CPanel you can get in to the web server (apache) logs? In there you can look for two things: the IP and the User-Agent. You can find the culprits in there and add them to your robots.txt and .htaccess. Note that .htaccess rules denying IP addresses are far better that just relying on robots.txt because you are taking the choice out of the bot creator's hands.
If you know specific bots which are doing this you should be able to get IP addresses and user-agents from forums, but if it's a more general thing then really I'm afraid it's more of a manual job.
There are other methods that can be used with varying effect, such as mod_security (http://www.askapache.com/htaccess/modsecurity-htaccess-tricks.html) but this will mean you'll have to access your web server configuration.
Finally, you can check the links that are pointing to your web site (using the link: option on google). Sometimes if you have links on spammy forums or the like this can increase the chances of bots coming to get you. Maybe you can look at the referer URL in the apache logs - but this is all based on a lot of presumptions and you'd probably be lucky if it had a great effect.
Block Unwanted Robots/Spiders visitors via PHP
Instructions:
Place the following PHP Code in the beginning of your index.php file.
The idea here is to place the code in the main site's PHP home page, the main entry point of the site.
If you have other PHP files that are accessed directly via an URL (not including PHP include or require support type files), then place the code in the beginning of those files.
For most PHP sites and PHP CMS sites, the root's index.php file is the file that is the main entry point of the site.
Keep in mind that your site statistics, i.e. AWStats, will still log the hits under Unknown robot (identified by 'bot' followed by a space or one of the following characters _+:,.;/-), but these bots will be blocked from accessing your site's content.
<?php
// ---------------------------------------------------------------------------------------------------------------
// Banned IP Addresses and Bots - Redirects banned visitors who make it past the .htaccess and or robots.txt files to an URL.
// The $banned_ip_addresses array can contain both full and partial IP addresses, i.e. Full = 123.456.789.101, Partial = 123.456.789. or 123.456. or 123.
// Use partial IP addresses to include all IP addresses that begin with a partial IP addresses. The partial IP addresses must end with a period.
// The $banned_bots, $banned_unknown_bots, and $good_bots arrays should contain keyword strings found within the User Agent string.
// The $banned_unknown_bots array is used to identify unknown robots (identified by 'bot' followed by a space or one of the following characters _+:,.;/\-).
// The $good_bots array contains keyword strings used as exemptions when checking for $banned_unknown_bots. If you do not want to utilize the $good_bots array such as
// $good_bots = array(), then you must remove the the keywords strings 'bot.','bot/','bot-' from the $banned_unknown_bots array or else the good bots will also be banned.
$banned_ip_addresses = array('41.','64.79.100.23','5.254.97.75','148.251.236.167','88.180.102.124','62.210.172.77','45.','195.206.253.146');
$banned_bots = array('.ru','AhrefsBot','crawl','crawler','DotBot','linkdex','majestic','meanpath','PageAnalyzer','robot','rogerbot','semalt','SeznamBot','spider');
$banned_unknown_bots = array('bot ','bot_','bot+','bot:','bot,','bot;','bot\\','bot.','bot/','bot-');
$good_bots = array('Google','MSN','bing','Slurp','Yahoo','DuckDuck');
$banned_redirect_url = 'http://english-1329329990.spampoison.com';
// Visitor's IP address and Browser (User Agent)
$ip_address = $_SERVER['REMOTE_ADDR'];
$browser = $_SERVER['HTTP_USER_AGENT'];
// Declared Temporary Variables
$ipfound = $piece = $botfound = $gbotfound = $ubotfound = '';
// Checks for Banned IP Addresses and Bots
if($banned_redirect_url != ''){
// Checks for Banned IP Address
if(!empty($banned_ip_addresses)){
if(in_array($ip_address, $banned_ip_addresses)){$ipfound = 'found';}
if($ipfound != 'found'){
$ip_pieces = explode('.', $ip_address);
foreach ($ip_pieces as $value){
$piece = $piece.$value.'.';
if(in_array($piece, $banned_ip_addresses)){$ipfound = 'found'; break;}
}
}
if($ipfound == 'found'){header("location: $banned_redirect_url"); exit();}
}
// Checks for Banned Bots
if(!empty($banned_bots)){
foreach ($banned_bots as $bbvalue){
$pos1 = stripos($browser, $bbvalue);
if($pos1 !== false){$botfound = 'found'; break;}
}
if($botfound == 'found'){header("location: $banned_redirect_url"); exit();}
}
// Checks for Banned Unknown Bots
if(!empty($good_bots)){
foreach ($good_bots as $gbvalue){
$pos2 = stripos($browser, $gbvalue);
if($pos2 !== false){$gbotfound = 'found'; break;}
}
}
if($gbotfound != 'found'){
if(!empty($banned_unknown_bots)){
foreach ($banned_unknown_bots as $bubvalue){
$pos3 = stripos($browser, $bubvalue);
if($pos3 !== false){$ubotfound = 'found'; break;}
}
if($ubotfound == 'found'){header("location: $banned_redirect_url"); exit();}
}
}
}
// ---------------------------------------------------------------------------------------------------------------
?>

Twitter oAuth callbackUrl - localhost development

Is anyone else having a difficult time getting Twitters oAuth's callback URL to hit their localhost development environment.
Apparently it has been disabled recently. http://code.google.com/p/twitter-api/issues/detail?id=534#c1
Does anyone have a workaround. I don't really want to stop my development
Alternative 1.
Set up your .hosts (Windows) or etc/hosts file to point a live domain to your localhost IP. such as:
127.0.0.1 xyz.example
where xyz.example is your real domain.
Alternative 2.
Also, the article gives the tip to alternatively use a URL shortener service. Shorten your local URL and provide the result as callback.
Alternative 3.
Furthermore, it seems that it works to provide for example http://127.0.0.1:8080 as callback to Twitter, instead of http://localhost:8080.
I just had to do this last week. Apparently localhost doesn't work but 127.0.0.1 does Go figure.
This of course assumes that you are registering two apps with Twitter, one for your live www.mysite.example and another for 127.0.0.1.
Just put http://127.0.0.1:xxxx/ as the callback URL, where xxxx is the port for your framework
Yes, it was disabled because of the recent security issue that was found in OAuth. The only solution for now is to create two OAuth applications - one for production and one for development. In the development application you set your localhost callback URL instead of the live one.
Callback URL edited
http://localhost:8585/logintwitter.aspx
Convert to
http://127.0.0.1:8585/logintwitter.aspx
This is how i did it:
Registered Callback URL:
http://127.0.0.1/Callback.aspx
OAuthTokenResponse authorizationTokens =
OAuthUtility.GetRequestToken(ConfigSettings.getConsumerKey(),
ConfigSettings.getConsumerSecret(),
"http://127.0.0.1:1066/Twitter/Callback.aspx");
ConfigSettings:
public static class ConfigSettings
{
public static String getConsumerKey()
{
return System.Configuration.ConfigurationManager.AppSettings["ConsumerKey"].ToString();
}
public static String getConsumerSecret()
{
return System.Configuration.ConfigurationManager.AppSettings["ConsumerSecret"].ToString();
}
}
Web.config:
<appSettings>
<add key="ConsumerKey" value="xxxxxxxxxxxxxxxxxxxx"/>
<add key="ConsumerSecret" value="xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"/>
</appSettings>
Make sure you set the property 'use dynamic ports' of you project to 'false' and enter a static port number instead. (I used 1066).
I hope this helps!
Use http://smackaho.st
What it does is a simple DNS association to 127.0.0.1 which allows you to bypass the filters on localhost or 127.0.0.1 :
smackaho.st. 28800 IN A 127.0.0.1
So if you click on the link, it will display you what you have on your local webserver (and if you don't have one, you'll get a 404). You can of course set it to any page/port you want :
http://smackaho.st:54878/twitter/callback
I was working with Twitter callback url on my localhost. If you are not sure how to create a virtual host ( this is important ) use Ampps. He is really cool and easy. In a few steps you have your own virtual host and then every url will work on it. For example:
download and install ampps
Add new domain. ( here you can set for example twitter.local) that means your virtual host will be http://twitter.local and it will work after step 3.
I am working on Win so go under to your host file -> C:\Windows\System32\Drivers\etc\hosts and add line: 127.0.0.1 twitter.local
Restart your Ampps and you can use your callback. You can specify any url, even if you are using some framework MVC or you have htaccess url rewrite.
Hope This Help!
Cheers.
Seems nowadays http://127.0.0.1 also stopped working.
A simple solution is to use http://localtest.me instead of http://localhost it is always pointing to 127.0.0.1 And you can even add any arbitrary subdomain to it, and it will still point to 127.0.0.1
See Website
When I develop locally, I always set up a locally hosted dev name that reflects the project I'm working on. I set this up in xampp through xampp\apache\conf\extra\httpd-vhosts.conf and then also in \Windows\System32\drivers\etc\hosts.
So if I am setting up a local dev site for example.com, I would set it up as example.dev in those two files.
Short Answer: Once this is set up properly, you can simply treat this url (http://example.dev) as if it were live (rather than local) as you set up your Twitter Application.
A similar answer was given here: https://dev.twitter.com/discussions/5749
Direct Quote (emphasis added):
You can provide any valid URL with a domain name we recognize on the
application details page. OAuth 1.0a requires you to send a
oauth_callback value on the request token step of the flow and we'll
accept a dynamic locahost-based callback on that step.
This worked like a charm for me. Hope this helps.
It can be done very conveniently with Fiddler:
Open menu Tools > HOSTS...
Insert a line like 127.0.0.1 your-production-domain.com, make sure that "Enable remapping of requests..." is checked. Don't forget to press Save.
If access to your real production server is needed, simply exit Fiddler or disable remapping.
Starting Fiddler again will turn on remapping (if it is checked).
A pleasant bonus is that you can specify a custom port, like this:
127.0.0.1:3000 your-production-domain.com (it would be impossible to achieve this via the hosts file). Also, instead of IP you can use any domain name (e.g., localhost).
This way, it is possible (but not necessary) to register your Twitter app only once (provided that you don't mind using the same keys for local development and production).
edit this function on TwitterAPIExchange.php at line #180
public function performRequest($return = true)
{
if (!is_bool($return))
{
throw new Exception('performRequest parameter must be true or false');
}
$header = array($this->buildAuthorizationHeader($this->oauth), 'Expect:');
$getfield = $this->getGetfield();
$postfields = $this->getPostfields();
$options = array(
CURLOPT_HTTPHEADER => $header,
CURLOPT_HEADER => false,
CURLOPT_URL => $this->url,
CURLOPT_RETURNTRANSFER => true,
CURLOPT_SSL_VERIFYPEER => false,
CURLOPT_SSL_VERIFYHOST => false
);
if (!is_null($postfields))
{
$options[CURLOPT_POSTFIELDS] = $postfields;
}
else
{
if ($getfield !== '')
{
$options[CURLOPT_URL] .= $getfield;
}
}
$feed = curl_init();
curl_setopt_array($feed, $options);
$json = curl_exec($feed);
curl_close($feed);
if ($return) { return $json; }
}
I had the same challenge and I was not able to give localhost as a valid callback URL. So I created a simple domain to help us developers out:
https://tolocalhost.com
It will redirect any path to your localhost domain and port you need. Hope it can be of use to other developers.
set callbackurl in twitter app : 127.0.0.1:3000
and set WEBrick to bind on 127.0.0.1 instead of 0.0.0.0
command : rails s -b 127.0.0.1
Looks like Twitter now allows localhost alongside whatever you have in the Callback URL settings, so long as there is a value there.
I struggled with this and followed a dozen solutions, in the end all I had to do to work with any ssl apis on local host was:
Go download: cacert.pem file
In php.ini * un-comment and change:
curl.cainfo = "c:/wamp/bin/php/php5.5.12/cacert.pem"
You can find where your php.ini file is on your machine by running php --ini in your CLI
I placed my cacert.pem in the same directory as php.ini for ease.
These are the steps that worked for me to get Facebook working with a local application on my laptop:
goto apps.twitter.com
enter the name, app description and your site URL
Note: for localhost:8000, use 127.0.0.1:8000 since the former will not work
enter the callback URL matching your callback URL defined in TWITTER_REDIRECT_URI your application
Note: eg: http://127.0.0.1/login/twitter/callback (localhost will not work).
Important enter both the "privacy policy" and "terms of use" URLs if you wish to request the user's email address
check the agree to terms checkbox
click [Create Your Twitter Application]
switch to the [Keys and Access Tokens] tab at the top
copy the "Consumer Key (API Key)" and "Consumer Secret (API Secret)" to TWITTER_KEY and TWITTER_SECRET in your application
click the "Permissions" tab and set appropriately to "read only", "read and write" or "read, write and direct message" (use the least intrusive option needed for your application, for just and OAuth login "read only" is sufficient
Under "Additional Permissions" check the "request email addresses from users" checkbox if you wish for the user's email address to be returned to the OAuth login data (in most cases check yes)

Resources