For a few years now I observe a strange behaviour, most likely triggered by enduser's email clients using my webapp (Ruby on Rails system, doesn't matter tho).
I am running a mid sized business and send out thousands of mails to my customers each month who buy leads from me.
The emails include two links, one to buy the lead and the other to give feedback. There is a dynamic part in both URLs which is a UUID, example:
offer/968ec0c1-e105-4c70-95b2-fd0c799b58f3
and
feedback/968ec0c1-e105-4c70-95b2-fd0c799b58f3
Every now and then, my webapp gets accessed at the same time on both links (which makes me confident it is not the user since it is the very same second they get accessed) but with different dynamic parts in the url, so i see in my logs
offer/NGVjZjA0YT
and
feedback/NGVjZjA0YT
It is always a random string with a length of 10 chars.
So this is not a big deal since it happens only 1-2 times per week and as far as I can tell no user is really affected by this, but still I wonder what's behind this. Did any of you experience a similar thing?
Maybe an email client wants to crawl / load a preview, seeing an uuid pattern in the url and changes it because whatever?!
I disabled link click tracking in the email sending provider (sendgrid), just as a side note. So they won't / shouldn't replace the email links. Also experienced this when sending links via AWS SES.
Im just curious. Any ideas or experiences? Thanks in advance & have a great day!
Related
We have been noticing a large number (~3400) of fake signups over the last year and have not been able to determine where they are coming from. Common parameters:
They often come from weird, yet validly formatted, email address (many in the .ru TLD or from thefmail.com)
Some use cyrillic or arabic characters in their name (we are basically focused only on US English speakers by our content)
They do NOT trigger the Intercom.io javascript for account signup notifications
They somehow defeat reCaptcha 3
They sometimes use URLs for their username
They don't confirm (devise_confirmable)
We've been handling these by disabling the accounts, and there's obviously a few items above we could use to identify these before they even get created, but I was wondering if someone's cracked this nut already or if there's some simple best practices (pwned db check?) that might cut this down to a dull roar or out entirely.
The two big "I don't get its" are bypassing the JS and defeating recaptcha. Is this just mechanical-turking?
Do you know whether these users have been created by the same IP address? (probably not). Are the accounts sporadically created or are they created in batches? The gem rack-attack could be used to mitigate this issue, especially if at least one of these conditions is satisfied. It also comes with some sort of fail2ban filter which could be helpful, as it is designed to detect suspicious requests from 'misbehaving' clients.
I also can't understand how they possibly bypassed recaptcha.
I'm not sure that this is a stack overflow appropriate question. If not, I'd appreciate a pointer to a more appropriate forum, as I haven't been able to find one.
I have a small website project that gets a few hundred daily unique users and on average I get one or two people to create an account per day. Yesterday I noticed that more users were signing up (like about 50) and today another 150 users signed up. Wonderful, right? Except that then I noticed that while the emails look legitimate, all of the usernames ended in same letters. My site requires that email be confirmed before a user gets any additional access and none of these accounts have confirmed their email. There is no apparent regularity to the creation of these accounts other than that it is happening with slowly increasing frequency.
My first question is, what is the most effective way to prevent this with the least user impact. The only thing that I can think of is adding a captcha step as part of account registration. I really dislike captcha so if anyone has a better idea for a general solution to this I'd appreciate it.
I'm also interested in this: What could this malicious user be gaining by doing this? It's not yet anything other than a minor nuisance to me. The accounts are easily identifiable and they're not (yet) being created at a rate that could represent anything like a denial of service attack. The only thing I can think is that they're trying to confirm that these emails are registered on my site. But I can't think why that would be useful. Also, if the email addresses are real, they're using my site to spam those email, but the spam is a registration confirmation for my site. So I guess they might eventually get my email provider to shut me down if they keep this up.
Thanks in advance for any help, even if that's a redirect to a different forum.
Other possibly useful information:
My site is hosted on Azure using asp.net mvc5 with identity framework
I believe that the emails are legitimate because my email provider
shows a very small bounce rate (like 1%) on these emails.
There are 2 more options which are SMS-Confirmation(by limiting the phone number), and IP restriction.
I have a site that logs a "hit (via saving a record to a Hits table that captures the date/time and IP of the machine whenever the detail page is loaded)" whenever a user brings up a detail page for a particular item so that admins can see how many hits that particular item gets. We get random instances where items are being hit multiple times/day in twos. So, in the data, it looks like a user is viewing an item, but the site is logging their hit twice in the database (same item, same date/time, same IP Address, etc.). Most hits are only being recorded once, and all my testing has lead to assurance the site is working appropriately. I'm noticing that particular IP Addresses are causing double hits. When I do Reverse IP searches, all the "double hits" are tied to IP Addresses that trace back to Amazonaws in northern Virginia, on the other side of the country. Our site is used locally, and the single hits are coming from IPs that trace back to local areas. Is there a bot hitting my site from afar? Should I block Amazonaws in Azure (which is where my site is hosted) or is that going to lock out genuine users? Is there a way I can detect whether a hit is genuine in my code (my site is in .Net MVC)? Has anyone faced a similar situation in the past?
Note: This IS RELEVANT to software engineering because a part of the question is asking how I can verify in my code that a hit is genuine!!!!!!!!!!!!!!!!!!
Basically, what I found out (no thanks to the elitist user who downvoted my question and offered no contribution) is that, my hit counter is being inflated by web crawlers. The quick and dirty solution is to implement a robots.txt file to block crawlers from hitting that page. Of course, that comes with the sacrifice that my client's site will no longer come up, should the public do a google search for the product being offered.
One alternative is the hidden link method; in which we put a hidden page on the site that no human user would ever access. When a bot hits that page, we record the IP in a "blacklist" table. Then, before our real hit counter logs a hit, it checks the user's IP against the blacklist.
Another alternative is to implement a blacklist of known User-Agents used by bots. We check the user's credentials against that list in order to determine whether a user is a bot.
Neither of these solutions are 100% though.
These are fairly adequate responses to my question. Of course, since this is StackExchange (or StackOverflow or StackYourMomma or whatever it is), people are just going to downvote your question and act like you're beneath a response because you didn't follow all the little bull crap rules that come along with being a member of the SE/SO/SYM community.
I need to develop an application which should help me in getting all the status,messages from different servers like Twitter,facebook etc in my application and also when i post a message it should gets updated in all the services. I am using authlogic for authentication. Can anyone suggest me what gems/plug-ins i can use..
I need API help to get all the tweets/messages to be displayed in my application and also ways to post the messages to the corresponding services by posting it from my application. Can anyone help me from design point.
Walk through what you'd want to do in your head. Imagine the working site, imagine your webapp working before you start. So your user logs in (handled by authlogic) and sees a textbox called "What are you doing right now?". The user fills in a status message and clicks "post". The status message appears at the top of their previously posted messages.
Start with the easy part. Create a class that posts to two services. Use the twitter gem and rfacebook to post to two already defined services. In the future, you'll want to let the user associate services to their account and you would iterate through the associated services and post the message to each. Once you have this working, you can refactor or polish the UI a bit to round out this feature. I personally would do the "add a social media account to my profile" feature towards the end.
Harder is the reading of the data (strangely enough) because you're going to have to figure out how to store it. You could store nothing but I suspect you'd run into API limits just searching all the time (could design around this). I would keep a little cache of posts associated to the user's social media account. In this way, the data model would look like this:
A user has many social media accounts.
A social media account has many posts. (cache)
Of course, now you need to schedule the caching of the posts. This could be done manually, based on an event (like when they login) or time based. So when the update happens, you load up the posts for that social media account and the user will see the posts the next time they hit the page. For real-time push to the client's browser while they stare at the screen, use faye (non-trivial) and ajax to pull the new posts to the top of the social media stream view.
The time based one is tricky because you'd either have to have a cron job run or have rails handle it all with a gem like clockwork. But then you have to leave rails running. I've also solved this by having a class in /lib do all the work and a simple web call kicks off the update. But it wasn't in a multi-user use case. So that might not work. In any case, you'll want to have some nice reusable code for these problems since update requests can come from many different sources.
You'll also have to deal with the API limits. When pulling down content from twitter, you won't get everything. That will just have to be known by the user or you'll have to indicate a "break in time" somehow.
The UI should be pretty easy (functionally anyway), because you know which source the post/content is coming from. It'd be easy to throw a little icon next to the post to display which social media site it's coming from.
Anyway, good luck, sounds like a fun project.
I'm going to attempt to create an open project which compares the most common MP3 download providers.
This will require a user to enter a track/album/artist name i.e. Deadmau5 this will then pull the relevant prices from the API's.
I have a few questions that some of you may have encountered before:
Should I have one server side page that requests all the data and it is all loaded simultaneously. If so, how would you deal with timeouts or any other problems that may arise. Or should the page load, then each price get pulled in one by one (ajax). What are your experiences when running a comparison check?
The main feature will to compare prices, but how can I be sure that the products are the same. I was thinking running time, track numbers but I would still have to set one source as my primary.
I'm making this a wiki, please add and edit any issues that you can think of.
Thanks for your help. Look out for a future blog!
I would check amazon first. they will give you a SKU (the barcode on the back of the album, I think amazon calls it an EAN) If the other providers use this, you can make sure they are looking at the right item.
I would cache all results into a database, and expire them after a reasonable time. This way when you get 100 requests for Britney Spears, you don't have to hammer the other sites and slow down your application.
You should also make sure you are multithreading whatever requests you are doing server side. Curl for instance allows you to pull multiple urls, and assigns a user defined callback. I'd have the callback send a some data so you can update your page with as the results come back. GETTUNES => curl callback returns some data for each url while connection is open that you parse it on the client side.