I'm trying to develop a mass mailing application and would like to use Delphi for it. What components or libraries would be my best options (for Delphi7). Can they handle sending about 25,000 emails? The application is not for spamming, so I'm curious if correctly authenticating and playing by the rules will add some delay to the process?
I'd use the Indy components, which are in the box. Newer versions will let you also send to gmail and other servers under SSL.
They can handle even complex situations, they limitations are likely in the SMTP server you'll use for sending. They generally have significant restrictions, but they tend to be all different.
The problem with mass mailing is not so much your program, but it is the limits that your ISP will allow.
Many ISPs severely restrict you and will not allow more than a set quota per hour, day or month that is usually quite low, e.g. 200 emails per day. Penalties are severe, and they can suspend your account without giving you a warning first.
If you (or the people using your program) want to send out 25,000 emails or so within a reasonable time, say a few hours, you will need either a dedicated email server, or need to use the services of a high volume SMTP service.
Also make sure you research into Throttling and Batching your emails.
First of all I agree with the answers marco.cantu and lkessler have given you.
If you want a higher level abstraction in a component, I have such a component available. You can find it here: RoboMailer
It can do mass mailing and also can personalize mails with additional data. It was designed for tasks just like this. I use it in a big product that sends out a lot of mails daily (surveying system with invitaion system on top). The component simplifies the task of mail sending so you don't have to worry about the pitfalls of mail composition etc... It is also equally simple to send 1 or 20.000 mails. The Demo is inlcuded. In fact the demo itself is a fully functional mass mailer :)
But it is based on ICS (Internet Component Suite), so you have to use ICS with it.
For simple internet applications I have found that Synapse library is very good. It can talk with TLS/SSL protected POP3 & SMTP while Indy cannot without paid libraries (at least from Turbo Delphi). It work very well with command line, formless applications that you can compile using Delphi, Kylix or FPC. Look at their HOWTO for examples with mail protocols.
Related
Now, I want to write a program(daemon) to query IMAP server for new mails and notify clients when their new mails arrive. And I have thousands of email accounts, so I want to know how to accomplish this efficiently and easily.
From my understanding, I know two possible solutions.
query email accounts one by one, which is 'poll' style
use IMAP idle extension and event driven I/O (epoll) reference
I want to know the advantages and disadvantages of both solutions.
I also want to know is there any library(C, C++, PHP, Python) can help me to do such thing. Or how can I do this by myself.
Or a third solution?
The advantage of IMAP IDLE is reduced overhead and (probably) increased responsiveness.
The advantage of polling is that it's inherently supported by all servers, whereas IDLE isn't supported by all servers.
As for libraries, #Max points out imaplib and imaplib2 in Python. I'm sure the other languages you mention have similar libraries.
As for third option, instead of IMAP libraries, you might look into using context.io.
There might be a third option if it is your imap server. In that case you could bypass the protocol itself and monitor the servers database itself for changes. That sould be much more efficient, since it omits the whole communication layer.
I am writing an application that keeps track of content pushed around between users of a certain task. I am thinking of using WebSockets to send down new content as they are available to all users who are currently using the app for that given task.
I am writing this on Rails and the client side app is on iOS (probably going to be in Android too). I'm afraid that this WebSocket solution might not scale well. I am after some advice and things to consider while making the decision to go with WebSockets vs. some kind of polling solution.
Would Ruby on Rails servers (like Heroku) support large number of WebSockets open at the same time? Let's say a million connections for argument sake. Any material anyone can provide me of such stuff?
Would it cost a lot more on server hosting if I architect it this way?
Is it even possible to maintain millions of WebSockets simultaneously? I feel like this may not be the best design decision.
This is my first try at a proper Rails API. Any advice is greatly appreciated. Thx.
Million connections over WebSockets, using Ruby, I can't see its real if you not using clustering to spread connections between different instances to handle all the data processing.
The problem here is serializing and deserializing data.
As well you have to research of how often you will need to pull data to client from server, and if it worth to have just periodical checks using AJAX, then handling connection for whole time. Because if you do handle connection and then you not using it - it is waste of resources. WebSockets are build on top of TCP layer, and all connections are not "cheap" as well going through for OS and asking them for data available again is not the simple process, with millions connections it is something really almost impossible without using most advanced technologies in the world.
I head that Erlang is able to handle millions of connections, but I don't have details over it. As well connection is one thing, another is processing data and interaction between connections - this you might want to check, because if you have heavy processing algorithms, then you definitely need to look into horizontal scaling options over clustering solutions.
If you are implementing chat, use websockets.
If you are implementing 1 way messages in realtime use server sent events.
If you are implementing 1 way messages sent every few hours or so, use APNS.
The saying goes phone in hand, use websockets / server sent events.
Phone in pocket, use APNS.
APNS will alleviate wifi dips, tcp/ip socket hangs and many other issues. Really useful. There is the chance that it may take a little time to get through. But then again, there is the chance that websockets will take
Recent versions of iOS let you send APNS to the client without a popup message to the client so it can ask the server for more information. That along with some backgrounding implementations really improves things.
If possible, do not implement totally anonymous clients. It is very tricky to detect if a client reinstalls the app. So you'll end up sending duplicates to the client. Need to take that into account.
APNS looks trivial to implement in ruby, but I'd suggest avoiding the urge and going to using an existing gem/service out there that supports both google and apple. It is much trickier to implement than it may seem at first.
If you decide to stick with websockets, it may make sense to just leverage websockets in nginx like https://github.com/wandenberg/nginx-push-stream-module
ASIDE:
Using SMS where speed is critical is very expensive. $1/month per phone number only sends a max rate of 1 message per second. So sending 100 messages per second = $100/month plus message fees. Do note that 100 messages at a rate of 50 messages/second = $50/month. But if you want to send 1k messages, that takes 20 seconds.
Good luck
Any suggestions for components to use as a base for a scalable TCP server? I currently have an implementation that uses Indy which works well for say 100 relatively active connections or 1,000 relatively inactive connections, but the one thread per connection model limits the number of concurrent active connections that can be handled.
Let's say my goal might be 1,000 connections each processing 10 messages per second or 10,000 connections each processing 1 message per second on a good server (8-16 cores). Is this realistic? I'd really like to hear of any real-world implementations because I have found that what might work in theory does not necessarily work in practice and I do not want to be chasing a proposed solution that will not work.
Edit: IOCP would be good, but I only want to use commercial-grade classes/components, so they need to be as "professional" as Indy or IP*Works before I would think of using them. Furthermore, I have no intention of "rolling my own" solution - it would take too much time to make it commercial-grade. Lastly, I am looking for a significant improvement on what I already have. I am sure I can squeeze at least 20-50% more out of what I have (based on Indy), but I am never going to be able to handle 10,000 concurrent clients, or 10,000 messages per second, no matter how hard I try. Whether there is something out there that meets these conditions is another matter.
I have decided to accept the answer referring to the IOCP classes, even though I have not used them, because they look like the best path for investigation at this stage.
There is a project at http://voipobjects.com/ which is based on the former iopcclasses project.
It claims to handle thousands simultaneous connections:
IOCP engine is set of classes, components and routines for rapid
creation high scalable and performance TCP/UDP applications.
Application created using IOCP classes can handle thousands
simultaneous connections.
Library is written in Delphi - Delphi 7 - 2010 are supported.
Library uses IO completion ports technology. There is most powerful
technology in Win32 world for creation highly scalable and performance
TCP/UDP applications. This technology is supported in all desktop
Windows OSes except old Win9x/WinME versions.
This library is licensed under MPL1.1. Also It includes some files
from Jedi project (Winsock2 header translation).
https://bitbucket.org/voipobjects/iocpengine
My favorite Delphi network layer is ICS by Francois Piette. It's fantastically easy to understand, very scalable, and ultra-high performance. Free, and open source. Will probably scale to 1000 clients for most people, without significant effort, and without the complexity that gives me trouble when I use Indy.
I got about a 20% scalability/performance boost from switching all my stuff from Indy to ICS.
You should look at RealThinClient SDK http://www.realthinclient.com/about.htm
Well proven solution. Good support. Test results for different server solutions on the home page.
The real deciding fact is what you plan on doing on each of those transactions.
I use Indy with Network Load Balanced windows servers. One of these Delphi applications is serviced by 3 physical servers listening one public IP address where we have received millions of requests since yesterday with zero errors. Load overnight is pretty idle so the actual requests are around 350/second/server during the day and there's plenty of room for growth.
If there's not a lot of CPU/Memory needed per transaction you might get away with it on one box using Indy. It all depends on the load...as you likely can't write to 1000 different files every second.
There's other items to worry about too - like the OS supporting this amount of activity. You may need to tweak some registry settings. (see this stackoverflow question)
IOCP is the way to go for ultra-capacity servers. I have used Indy for ease of use in implementation/debugging for a very long time. I have my own IOCP implementation that I wrote years ago but never rolled it out on production as we simply haven't needed to.
My simple advice - I'd highly suggest rolling it out in Indy, using NLB as your crutch for load, and after that if you are still desiring the utmost speed, write your own IOCP implementation so you can craft it towards your specific requirements. Note that this is based on knowing nothing of the actual implementation requirements.
I've tried multiple Delphi solutions to do networking, and found that many if not all solutions add complexity and code which impacts on either performance or footprint or both. So I started searching for the lightest wrapper around the winsock API. The I (re)discovered Delphi's own TTcpClient and TTcpServer components. Used in blocking mode, and overriding TCustomTcpServer overriding the DoAccept method, I've had the best results till now.
If you expect to have a really high number of incoming connections and (small) responses to serve to (small) requests, it's highly advisable to implement I/O completion ports, as this handles incoming requests better.
I have been using ICS for the last 12 years. It is non-blocking. I support upto 2000 concurrent connections, each getting atleast 5000 bytes per second, sent as 1000 bytes every 200 msec. Never faced any problem and cpu usage of the app is very small.
Good support in forum, but not required at all.
Shekar
We intend to design a system with three "tiers".
HQ, with a single server
lots of "nodes" on a regional basis
users, with iPads.
HQ communicates 2-way with the nodes which communciate 2-way with the users. Users never communicate with HQ nor vice-versa.
The powers that be decree a Windows app from HQ (using Delphi) and a native desktop app for the users' iPads. They have no opinion on the nodes.
If there are compelling technical arguments, I might be able to beat them down from "decree" to "prefer" on the Windows program (and, for isntance, make it browser based). The nodes have no GUI, they just sit there playing middle-man.
What's the best way for these things to communicate (SOAP/HTTP/AJAX/jQuery/home-brewed-protocol-on-top-of-TCP/something-else?) Is it best to use the same protocol end to end, or different protocols for hq<-->node and node<-->iPad?
Both ends of each of those two interfaces might wish to initiate a transaction (which I can easily do if I roll my own protocol), so should I use push/pull/long-poll or what?
I hope that this description makes sense. Please ask questions if it does not. Thanks.
Update:
File size is typcially below 1MB with nothing likely to be above 10MB or even 5MB. No second file will be sent before a first file is acknowledged.
Files flow "downhill" from HQ to node to iPad. Files will never flow "uphill", but there will be some small packets of data (in addition to acks) which are initiated by user action on the iPad. These will go to the local node and then to the HQ. We are probably talking <128 bytes.
I suppose there will also be general control & maintenance traffic at a low rate, in all directions.
For push / pull (publish / subscribe or peer to peer communication), cross-platform message brokers could be used. I am not sure if there are (iOS) client libraries for Microsoft Message Queue (MSMQ), but I would also evaluate open source solutions like HornetQ, Apache ActiveMQ, Apollo, OpenMQ, Apache QPid or RabbitMQ.
All these solutions provide a reliable foundation for distributed messaging, like failover, clustering, persistence, with high performance and many clients attached. On this infrastructure message with any content type (JSON, binary, plain text) can be exchanged, and on top messages can contain routing and priority information. They also support transacted messaging.
There are Delphi and Free Pascal client libraries available for many enterprise quality open source messaging products. (I am am the author of some of them, supporting ActiveMQ, Apollo, HornetQ, OpenMQ and RabbitMQ)
Check out MessagePack: http://msgpack.org/
Also, here's more RPC discussion on SO:
RPC frameworks available?
MessagePack: fast cross-platform serializer and RPC - please share experience
ICE might be of interest to you: http://zeroc.com/index.html
They have an iOS layer: http://zeroc.com/icetouch/index.html
IMHO there are too little requisites to decide what technology to use. What data are exchanged, how often, what size? Are there request/response time constraints? etc. etc. Never start selecting a technology before you understand your needs deeply.
I would like to write a messaging system using TCP IP in Delphi 2010. I would like to hear what my best options are for using the standard delphi 2010 components/indy components for doing this.
I would like to write a server which does the listening and forwarding of messages to all machines on the network running a client.
1.) a.) clients can send a message to server to be forwarded to all other clients
b.) clients listen for messages from other senders (via server) and displays messages.
2.) a.) Server can send a message to all clients
b.) Server forwards any messages from clients to all other clients
thanks for any suggestions
NOTE: I am not writing a instant messaging or chat program. This is merely a system where users can send alerts/messages to other users - they can not reply to each other! NO commercial, shareware, etc links - please! I would like to hear about how you would go about writing this type of system and what approachs you would take, and possibly the TCP IP messaging architecture you would use. Whether it be straight Winows API, Indy components, etc, etc.
If this is Windows only, and you don't want to use 3rd party libraries, then you can skip TCP/IP and go for Mailslots.
Edit: if you want guaranteed delivery, than named pipes is a better solution. This SO question has a few nice answers with Delphi named pipe examples.
A mailslot is a mechanism for one-way
interprocess communications (IPC).
Applications can store messages in a
mailslot. The owner of the mailslot
can retrieve messages that are stored
there. These messages are typically
sent over a network to either a
specified computer or to all computers
in a specified domain. A domain is a
group of workstations and servers that
share a group name.
They don't need a Windows domain, they work over a LAN.
DelphiPages has a nice run-down on doing IPC in Delphi, including mailslots.
--jeroen
If you want save yourself some work and use ready solution:
0MQ
STOMP
AMQP
XMPP
in above order. Last two listed rather in informational purposes.
IMO 0MQ it's best bet - it's well-thought-out, lightweight, fast and reliable.
Looks like publish-subscribe messaging. kbmMW offers one.
There's an instant message client from Aid Aim:
http://www.aidaim.com/products/msgcommunicator/manual_instant-messaging-im-sdk/index.php
About using indy, such as in this demo: indy10clieservr.sourceforge.net OR using the TServerSocket and TClientSocket package
-> now with x64 demo sample from INDY
To get this working reliably is more work than you might think - about 5% effort to get it working, and 95% effort to ensure it works under all circumstances, including network dropouts, etc. If you go here (http://www.csinnovations.com/framework_delphi.htm) you can see how I implemented my inter-process communications framework. The TCP stuff comprises about 10,000 lines of code using the Indy components.