For my portfolio software I have been using fetchmail to read from a Google email account over IMAP and life has been great. Thanks to the miracle of idle connection supported by imap3, my triggers fire in near-realtime due to server push, much sooner than periodic polling would allow otherwise.
In my basic .fetchmailrc setup, in which a brokerage customer's account emails trade notifications to a dedicated Gmail/Google Apps box, I've had
poll imap.gmail.com proto imap user "youraddress#yourdomain-OR-gmail.com" pass "yoMama" keep nofetchall ssl idle mimedecode limit 29000 no rewrite mda "myCustomSpecialMDAhandler.sh %F %T"
Trouble is, now I need to support reading from multiple email boxes, and hand off the emails to other specialized MDA scripts I wrote. No problem, just add more poll lines to .fetchmailrc, right? Well that doesn't work when the other accounts also use imap.gmail.com. What ends up happening is that while one account reads fine (and not necessary the first one listed, though usually yes), the other is getting "socket error" all day and the emails remain untouched, unread. I can't figure out why and not even sure if it's some mechanism at imap.gmail.com or not, eg. limiting to one IMAP connection from a host. That doesn't seem right since I have kept IMAP connections to many separate Gmail & Google Apps accounts from the same client for years (like Thunderbird) and never noticed this exclusivity problem.
I haven't tried launching multiple fetchmail daemons using separate -f config files (assuming they wouldn't conflict), or deploying one or more getmail and other similar email fetchers in addition. Still trying to avoid that kind of mess--unscaleable the more boxes I have to monitor.
Don't have the reference offhand but somewhere in fetchmail's docs I recall reading that idle is not so much an imap feature as a fetchmail optional trick, which has a (nasty for me) side effect of choking off all other defined accounts from polling until the connection is cut off by some external event or timeout. So at least that would vindicate Google.
Credit to Carl's Whine Rack blog for some tips.
For now I use killall fetchmail; fetchmail -f fetcher.$[$RANDOM % $numaccounts].rc periodically from crontab to cycle reading accounts each defined individually in fetcher.1.rc, fetcher.2.rc, etc. Acceptable while email events are relatively infrequent.
Related
I have a MQTT environment like this:
there is One (gray) sensor and one Observer that are related by the topic room/temp, so far so good, sensor can publish and the Observer can get the info as it should.
the Issue I have is now: I need to block IN THE BROKER that a 2nd undesired client comes(the orange one),and start to publish into the same topic, as far as I know, MQTT is loose coupled so that observer doesn't care who is pushing the temp values, but I find a security flawless when someone hack my environment and publish non sense triggering my alarms...
any suggestion?
am using eMQTTd by the way and according to this there is nothing in the etc/emqttd.config file I can do to avoid that...
Thanks!
I only have experience with Mosquitto but, from a quick read of the document linked, it looks like there are several ways you could achieve this.
I am unclear if you are talking about an incidental problem here--i.e. bad information is being accidentally sent--or if you are protecting against an active threat.
If you are concerned with incidental overwriting of a value, then the simple clientid solution on (pg. 38) would work.
But my impression is that it would still be transmitted in the clear and thus be of little use to you if you are facing an actual adversary (hacker etc.). If that is your concern simply setup SSL and remove all non-SSL listeners. (See pg. 24). That should limit all traffic to an encrypted channel. Then if you wish add password / user authentication (pg. 38) to complete the security.
Alternatively, depending on your configuration, you could block unapproved ip addresses at the firewall level (i.e. block access to the port that your broker is listening on to all addresses except for the temperature sensor) or using eMQTTd's built in ACL facility (pg. 25). That would be less secure than a full SSL setup but depending upon your needs it might be enough.
Hi I am implementing Email Client Application. My requirement is i need to monitor all the mailboxes available in specified IMAP server. I am created separate TCP Connection for each mailboxes. But i am getting disconnected from IMAP Server. I am trying Gmail/yahoo for my testing purpose. Is there any restriction to open multiple connection from same ip to particular IMAP Server? Particularly in Gmail and Yahoo.
or is there anyway to Monitor all the mailboxes in Single Connection without using IMAP-NOTIFY seems it does not supported in both Gmail/Yahoo...
Please Help me out...
This is something which I have answered on stackoverflow before, but which is now only available via the wayback machine. The question was about how to "kill too many parallel IMAP connections". Reprinted below; the core takeaway message is that for some reason, most server administrators prefer to have smaller number of short-lived connections instead of more connections which are active over longer period of time, yet they spend most of their time silently idling in the background. What they do not get is that the IMAP protocol is designed with long-lived connections in mind, and trying to prevent that will lead to wasting resources because the clients will constantly resync mailboxes as they are hopping among them.
The original answer follows:
Nope, it's a very wrong idea. IMAP is designed so that monitoring a single mailbox takes one connection; in most IMAP server implementations, this means a single process. However, unless the client the user is using is terribly broken, all these connections enter the IDLE mode. In IDLE, the clients are passively notified about any updates to the mailbox state. If you disable these connections, the clients would have to activelly poll for changes in many mailboxes. Now decide for yourself -- what is worse, having ten processes sitting idle, or one process doing heavy polling every two minutes? Which of these solutions would consume more energy, CPU time and IO operations? That's for the number of parallel connections.
The second question was about the long-lived connections. Again, this is a critical aspect of IMAP -- each connection carries a lot of associated state information which is rather expensive to obtain. Unless your server implements certain extensions and your clients use them (ESEARCH, CONDSTORE, QRESYNC are the crucial bits), opening a mailbox can require O(n) operations. I don't know how many messages your users have, but do you really want to transfer e.g. message flags for 250k messages when you decided to kill a connection because it has been active for "too long"?
Finally, any reasonable IMAP server vendor offers a way to configure a per-user session limit on the number of concurrent processes. Using that is much better than maintaining a script for ad-hoc killing of "unused" connections.
If you would like to learn more about the synchronization process, my thesis about using IMAP on clients with flaky network and limited resources describes what the clients have to do in order to show an updated view of mailboxes to their users.
Use case is to have a server connect to thousands of users email accounts and sniff incoming mail in java preferably with java mail and spring integration/amqp/rabbit mq type scalable infrastructure.. And imap idle type connections and add server processing nodes as needed.
Single inbound channel is easy with imap idle inbound adapter.. You could configure few in XML. But if you need a persistent listener/imapidlechannel adapters queue of thousands of these adapters and Needed to add new user connection dynamically for server processing.. This would be a challenge. Also need fault taulerance that if the java listener dies or server reboots all these listeners and their configuration also reboot vs rebuilding thousands of these connections and recovery if some connections loose their idle receive capability without rebuilding all user connections for the idle receiving.
Any ideas welcome as searched a lot however could not find anything? This seems to be a significant scalability issue about e mail receive connections open.
If you want to use the IMAP IDLE command to listen for new messages using JavaMail, you'll need one thread per mailbox, which is likely to impact your scalability. Even keeping thousands of connections open might be an issue.
You don't say how quickly you need to react to new messages. Unless you have near real time requirements, it might be better to poll a subset of mailboxes every so often, eventually cycling through all the mailboxes.
You'll need to deal with the fault tolerance issues yourself, using checkpointing or transactions or whatever seems appropriate for your application.
The other option is to perhaps take a look at something like Akka with actors performing the async io. You'll need to ditch the JavaMail package and parse the imap commands yourself but there's lots of packages out there to do that. Would love to hear if you had a better solution.
I'm currently designing an application for iOS (using MonoTouch) that will have a server component running on Windows Azure. The application will essentially be a chat type application where users will generate messages within their clients and send them to the server, which will then need to forward those messages out (as quickly as practicable) to other clients that the user might be sending the messages to.
My question is - is there a recommended practice for architecting an application like this, where clients need to receive 'push' messages from the server?
I've considered a few options but would appreciate feedback.
The first option is to use Apple's Push Notifications service (APNs). I have two concerns about this - first, the clients only need to receive the messages when they're online (APNs sends messages through when the app is closed too, which I don't need or want); and second, there is a possibility that there will be a high volume of messages, which I know Apple would probably get unhappy about (perfectly fairly).
A second option I considered is using a web service (WCF-based) and having the client call this service every (say) 2-3 seconds, which is the maximum delay we could tolerate. This would seem to involve a great deal of potentially unnecessary network traffic, though ("have you got anything for me?", "no", repeated ad nauseum).
A third option is to maintain a persistent web service connection between the client and the server. When the client app starts it would call a web service method on a background thread. The server would hold the connection open (by not returning anything), and if any messages came through it would immediately return them. This connection might time out after, say, 2 minutes at which point it would be re-established. This seems to do what I want, but again, I'm concerned that there'd be a lot of connections open to the server at any moment, which could require server resources unnecessarily.
A fourth option is to use a persistent connection over TCP (or UDP, although from what I've found, Windows Azure doesn't support this). This seems to be a good option, but again, might be overkill in terms of server usage - there could potentially be hundreds or even thousands of clients connected at any moment.
A fifth option is to somehow have the server push messages directly to the client, perhaps by having the client run a mini web server or similar. However, as the app will be running on 3G and WiFi networks (beyond my control) I don't expect incoming ports will be open for this sort of thing.
If anyone has any other suggestions, or thinks one of the above options would be a good idea (or is a standard way of approaching this sort of problem) I'd be very interested to hear about it.
Thanks in advance,
John
You had a look at Pubnub http://www.pubnub.com/ ?
I'm trying to develop a mass mailing application and would like to use Delphi for it. What components or libraries would be my best options (for Delphi7). Can they handle sending about 25,000 emails? The application is not for spamming, so I'm curious if correctly authenticating and playing by the rules will add some delay to the process?
I'd use the Indy components, which are in the box. Newer versions will let you also send to gmail and other servers under SSL.
They can handle even complex situations, they limitations are likely in the SMTP server you'll use for sending. They generally have significant restrictions, but they tend to be all different.
The problem with mass mailing is not so much your program, but it is the limits that your ISP will allow.
Many ISPs severely restrict you and will not allow more than a set quota per hour, day or month that is usually quite low, e.g. 200 emails per day. Penalties are severe, and they can suspend your account without giving you a warning first.
If you (or the people using your program) want to send out 25,000 emails or so within a reasonable time, say a few hours, you will need either a dedicated email server, or need to use the services of a high volume SMTP service.
Also make sure you research into Throttling and Batching your emails.
First of all I agree with the answers marco.cantu and lkessler have given you.
If you want a higher level abstraction in a component, I have such a component available. You can find it here: RoboMailer
It can do mass mailing and also can personalize mails with additional data. It was designed for tasks just like this. I use it in a big product that sends out a lot of mails daily (surveying system with invitaion system on top). The component simplifies the task of mail sending so you don't have to worry about the pitfalls of mail composition etc... It is also equally simple to send 1 or 20.000 mails. The Demo is inlcuded. In fact the demo itself is a fully functional mass mailer :)
But it is based on ICS (Internet Component Suite), so you have to use ICS with it.
For simple internet applications I have found that Synapse library is very good. It can talk with TLS/SSL protected POP3 & SMTP while Indy cannot without paid libraries (at least from Turbo Delphi). It work very well with command line, formless applications that you can compile using Delphi, Kylix or FPC. Look at their HOWTO for examples with mail protocols.