I'm looking at porting an Interbase 6 / Delphi 7 application to embedded Firebird in Delphi 2007. One of the problems we have is getting our users (often quite an unskilled bunch, really - though I love them to bits, naturally) to unblock our applications in their firewall. Windows firewall itself is fairly straightforward but often they are running McAfee or similar (they tend to buy cheap Dells with this stuff pre-installed) and it seems that each and every variation of this stuff has a slightly different user interface. sigh
Still, I'm digressing, sorry. Straight to the point; If my Delphi app connects to an embedded Firebird database, will I still need to all/open something in the user's firewall (as I currently do when installing stuff that makes a connection to 'normal' IB6)?
And if you've read this far (thanks) - can embedded Firebird be used concurrently on a machine? Let's say we have 2 applications, both of which want to use DIFFERENT databases - could the user run both of these apps simultaneously on the same machine or is there some kind of port binding that goes on under the hood, which we'd have to work around?
I have never had a problem with firewalls or McAfee with embedded firebird. (I assume this is because embedded is not really a 'server' and does not require a port to operate)
Yes you can have two apps concurrently, just keep the executables & databases in two different folders.
Even using Firebird in a non-embeded install on the local machine we have never bumped into any firewall issues in hundreds of installations. You don't even have to use TCP/IP to connect to the database. We do use TCP/IP, but using the local shared memory protocol would avoid the issue entirely.
Firebird makes an excellent embedded or semi-embedded database. We just install it in it's normal mode and it runs in the background without any user intervention 24x7 for years at a time.
As the embedded version of Firebird doesn't use TCP/IP to talk to the database, you'll be fine on single user machines. Bear in mind that Firebird Embedded is single-user and you won't be able to get two apps talking concurrently to the same database. To do that you'd need to install the Firebird server on the machine and in the connection string use localhost:C:\Data\MyDB.FDB on both apps.
I use UIB to talk to Firebird (I wrote a persistence layer for the OPF I use using it), it's thread-safe (unlike IBX) and I've found it to be appreciably faster than IBX. There's a version that comes with JVCL and a slightly later version at http://www.progdigy.com
Related
Are there major disadvantages using the embedded Firebird 3 in a multi-user application server (Delphi Webbroker) instead of the full blown server install?
The application usually has very short transactions with low data volume.
As far as I am informed accessing one database file with multiple threads through the embedded server is not problematic but user security is not available. As the application server does the rights stuff I do not need Firebird security.
But will I loose performance or things like garbage collection?
Firebird Embedded provides all the features (except network access and authentication) that a normal Firebird server provides. However, because it is in-process, any problems that cause your application to crash, will take Firebird with it and vice versa.
Other possible downsides:
Garbage collection will - as far as I know - always use the 'cooperative' model (where the connection to find old record versions, is the one that cleans it up),
You can't use other tools to access your database remotely which may make administration harder,
You can't put your database on a separate server from your web application (think of security requirements).
Personally, I would only choose Firebird Embedded if the situation calls for it. In all other situations, I will use Firebird Server.
Has somebody real experience with firebird databases over the internet?
I have a typical windows accounting/ERP software (done with delphi) that works with the firebird database server pretty well.. Now my users (300 aprox. now, but should increment) also want to work "in the cloud" (connecting from the office, from the laptop, from the house, etc.). It is a lot of work of recreating everything to a standard web application (let's say for example, HTML+CSS+JS+PHP+MYSQL), so I'm considering keeping the win client (I don't care about other OSes) but instead of the server living in the clients LANs moving it to a pair of dedicated servers that I will contract (one primary and one secondary againts failures for starting).
Searching I've come across this faq http://www.firebirdfaq.org/faq53/ that explains that the fb protocol it isn't ideal for working in the internet, but still all my users today have at least a 1MBbit/sec ADSL internet connection (I don't think that to be slow as the faq denotes).
Somebody have done this? what was the experience? how secure are fb servers for being open to the internet? how well they scale?
I know that building a "middleware" with SOAP for example will be more normal, but still the solution I'm evaluating here is much more fast and easy (still I have some work with the replication, backup, hearbreath services, but it's much less than redoing everything for the web).
Thanks! Edit: FB version: 2.5.
I had being trying to "push" the Firebird Core developers to improve the Firebird protocol to get better speed with high latency network (aka. Internet). Recently, Dmitry Yemanov published some articles in his blog about this subject (dyemanov.blogspot.com). It seems that there is margin for optimizations, and I would really like to see this coming in FB 2.5.3 and FB 3.0, although there is no warranty for this happening in those versions or anytime soon. You can vote in such improvement here: http://tracker.firebirdsql.org/browse/CORE-2530
Safety? You may try to set up a VPN. It also may help with speed, since most of the VPNs software out there (Zebedee, etc) can compress the data being transfered, helping to speed up data transfer in some cases.
Some of my customers do use Firebird traditional C/S over the internet. It is much slower compared to local network, and of course, how much slower depends basically on the link speed and latency. You can do some optimization at the client side too, using metadata cache, etc. but don't expect miracles with the currently protocol. I would say that for whole day working, using Terminal Services would be a better option for now.
The response about the scaling question
Firebird runs in production on large big iron servers : 512G of ram 100.000 concurrent users
We run Firebird to power larger systems (for 12 government agencies
and 3 banks). It has approximately 100000 end users multiplexed
through 2500 (max) pooled connections
https://plus.google.com/111558763769231855886/posts/Q1ACy1yyTgP
The protocol in Firebird 2.5 is improoved there is still room left for 3.0 but you can check
what is already done
http://asfernandes.blogspot.com/2009/07/network-latency-influence-on-firebird.html
And the future enhancements in 3.0
http://www.firebirdnews.org/?p=6953
To protect your connection i guess the best bet is ssl/ssh tunnel (it can be a opnvpn)
with high compression option
http://mapopa.blogspot.com/2010/11/securing-firebird-using-ssh-tunnel.html
FB protocol problem isn't about bandwidth, but latency. In my experience, some operations can be very slow over internet/VPN compared to LAN or local connection. I haven't examined issue further since I don't really run applications over internet connection.
However, I suggest three-tier model for application. Create own application server, which runs on database server/same network. Let the clients talk with application server and you get maximum performance.
There are some N-tier application/middleware frameworks for Delphi:
RemObjects SDK and DataAbstract
RealThinClient
kbmMW
Delphi's own DataSnap
MidWare
With those you can get data compression, encryption, binary messages (faster than SOAP) etc.
You can implement TCP/IP packets encryption/decryption directly in the firebird engine itself.
Personnaly, i have downloaded the Firebird 2.5 source code and injected secure tunnelization code directly in his low level communication layer (the INET socket layer). Now, encryption/decryption is done directly by the firebird engine for each TCP/IP packet both at the server side and the client side (fbclient.dll).
Then there is no need to re-structure the client application except adding one line of code that provide the secret key you choose to crypt communication to the fbclient.dll. The same secret key must be declared in the firebird.conf file of your server installation.
I have also implemented a proxy negociation solution in the fbclient.dll in order to allow to TCP/IP packets to pass throught any proxy server (like Microsoft ISA Server for example).
For us, this architecture is functional for more than one year in a real production system.
kbmMW CodeGear Edition is free but without source. It can be used for commercial apps.
Download it after registering at: https://portal.components4developers.com
In case you see certificate errors (you shouldnt but I know we have heard that some actually do), accept and ignore them. The site is valid despite the cert.error.
kbmMW CodeGear Edition contains a subset of kbmMW Professional Edition, but supports the following Delphi database API's:
Borland Database Engine
DBExpress
kbmMemTable
SQLite3
It supports binary, binary over HTML, XML and SOAP protocols in communication with clients.
It contains everything you need incl.
unified remote custom method invocation
unified remote dataset query, execute and data change resolving
unified database meta data handling and creation (tables, fields, indexes, generators/sequencers)
optional automatic proxying of requests to another server and proxying results back to original requester
full native XML DOM and SAX support
full dataset briefcase support as CSV, or binary data
advanced but simple to use wizard for creating new application server services
THere is one caveat though. Newest version of kbmMW CodeGear Edition always only supports newest Delphi version. You can still download older kbmMW CodeGear Editions matching older Delphi releases.
kbMMW Professional Edition and kbmMW Enterprise Edition do not have such limitations, and currently supports D7, D2006, D2007, D2010, DXE, DXE2 along with Embarcadero C++ counterparts.
best regards
Kim Madsen
www.components4developers.com
I have a Client/Server application written Delphi. Essentially all the application is doing is transferring xml data streams between a server application and connected clients. I am currently using the Indy TIdTCPServer component. But the server side application keeps crashing on some of my installments. And it has been extremely difficult to debug. So I am wondering if there is some "architecture" I should be utilizing which does all the tcp/ip connection management and database connection pooling, allowing me to concentrate on the business logic.
Here are more details:
clients must maintain a "persistent" connection. There are times when the server must notify and send data to all connected clients.
clients are connecting from laptop computers using wireless aircards. So network "drops" are pretty common.
Backend database is SqlServer.
There can be upward of 100 computers simultaneously connected at a time.
When the server gets a new connection (TCPServer.OnConnect) I instantiate my own object containing it own SqlServer database connection. When tcp connections are dropped I in turn free these objects (and associated database connection).
Client application have a TTimer built into them. They routinely send heartbeats to the server. And if they "drop"/"lose" their connection they automatically establish a new connection once the network is back.
Anyone have any suggestions on the best approach/architecture here?
I presume the Indy component would work, but at the same time feel I am "reinventing the wheel" with respect to managing the connections.
Three component sets I am aware of that will take care of the nitty gritty technical aspects of client server applications for you:
kbmMW: http://components4developers.com/
Asta: http://www.astatech.com/index.asp
RemObjects: http://www.remobjects.com/
You may have to rework your applications to take advantage of the way these component sets work, but assuming you have properly separated layers that shouldn't be too much of a hassle and will buy you the advantage of well tested and widely used code for your client server work.
If you want some light TCP/IP components, take a look at our SynCrtSock unit.
You'll find low-level classes to create IP Client and Servers.
We implemented both TCP/IP and UDP/IP in one of our applications.
There is also a THttpServer class, which implement a HTTP/1.1 server. Therefore it follows the HTTP/1.1 connection management. There is also an optional compression, and using HTTP/1.1 on a port other than 80 is not a bad idea. And what is good with HTTP/1.1 is that it can pass through firewalls, and can be easily be VPNed or hosted on another HTTP server (like IIS or Apache) with a proxy. There is even a FastCGI class, if you need such a server under a linux-based solution.
Of course, a THttpClientSocket class does the same on the client class.
We use these classes to add HTTP/1.1 connection to our Open Source SQLite3 RESTful framework - http://synopse.info/forum/viewforum.php?id=2
See http://synopse.info/fossil/artifact?name=722e896e3d7aad1fe217b0e2e7903483e66d66d1 for the SynCrtSock unit. Open source, work from Delphi 7 to Delphi 2010.
Misha Charrett's CSI Application Framework covers pretty much exactly what you're asking for.
It's an open source Delphi framework that at its heart is a distributed message passing and threading framework that allows XML message passing from both client to server and server to client.
It can handle disconnections/reconnections, high client numbers and there's an optional virtual database library that will handle SQL server (or you could just use same SQL Server access you're using now).
It's not particularly well known yet but I can tell you that it's been actively developed over the last few years and that the author Misha is very keen to assist anyone who's interested in using it in their application.
Well, it would probably require a complete rewrite of much of your C/S code, but instead of using the Indy components, you could try to use a COM+ solution instead. Basically, you would create a COM+ component that will be installed on the server and your client applications will connect to this client and call the functions of this component directly. It will have transaction management which will be handled by Windows itself and the same is true about handling transactions. It's also technically possible to create events, which would allow the server to do callbacks to the client, although that would make things a bit more complicated.
I don't think this solution would work out for you, though, unless you have a lot of experience with COM development in Windows and/or you're brave enough to try something different.
In the past, I had a similar problem where hundreds of clients had to connect to a single server, doing all kinds of database transactions. It has a steep learning curve but me and my team managed to get things working and once we understood the technique, it resulted in a very stable and reliable solution which did manage to have up to 500 users simultaneously doing updates and other actions in a one-time extreme stress-test. But again, the learning curse is steep, so it might not be the solution you're looking for.
(Still, COM+ will use a lot of functionality that's build-in into Windows, like transaction management, database pooling and whatever more.)
If you use Indy each connection will equal a thread.
Anyway, I suggest for connecting to MSSQL to use SDAC from Devart http://www.devart.com/sdac/ and for the connection layer to use HPScktSrvr based on I/O Completion Port from http://www.torry.net/authorsmore.php?id=7131 (I don't know though what changes it will need for TThread changes in newer VCL).
You build your client class arround THPServerClient, you set your new class as the server ClientClass and the framework will create automatically new clients for you.
You may also want to have a look at the ICS/Midware combo: http://www.overbyte.be/
The situation is simple. I've created a complex Delphi application which uses several different techniques. The main application is a WIN32 module but a few parts are developed as .NET assemblies. It also communicates with a web service or retrieves data from a specific website. It keeps most of it's user-data inside an MS Access database with some additional settings inside the Registry. In-memory, all data is converted inside an XML document, which is occasionally saved to disk as backup in case the system crashes. (Thus allowing the user to recover his data.) There's also some data in XML files for read-only purposes. The application also executes other applications and wants for those to finish. All in al, it's a pretty complex application.
We don't support Citrix with this application, although a few users do use this application on a Citrix server. (Basically, it allows those users to be more mobile.) But even though we keep telling them that we don't support Citrix, those customers are trying to push us to help them with some occasional problems that they tend to have.
The main problem seems to be an occasional random exception that seems to pop up on Citrix systems. Never at the same location and often it looks related to some memory problems. We've p[lenty of error reports already and there are just too many different errors. So I know solving all those will be complex.
So I would like to go a bit more generic and just want to know about the possible issues a Delphi (2007) can have when it's run on a Citrix system. Especially when this application is not designed to be Citrix-aware in any way. We don't want to support Citrix officially but it would be nice if we can help those customers. Not that they're going to pay us more, but still...
So does anyone know some common issues a Delphi application can have on a Citrix system?
Does anyone know about common issues with Citrix in general?
Is there some Silver Bullet or Golden Hammer solution somewhere for Citrix problems?
Btw. My knowledge about Citrix is limited to this Wikipedia entry and this website... And a bit I've Googled...
There were some issues in the past with Published Delphi Applications on Citrix having no icon in the taskbar. I think this was resolved by the MainFormOnTaskbar (available in D2007 and higher). Apart from that there's not much difference between Terminal Server and Citrix (from the Application's perspective), the most important things you need to account for are:
Users are NEVER administrator on a Terminal or Citrix Server, so they no rights in the Local Machine part of the registry, the C drive, Program Folder and so on.
It must be possible for multiple users on the same system to start your application concurrently.
Certain folders such as the Windows folder are redirected to prevent possible application issues, this is also means that API's like GetWindowsFolder do not return the real windows folder but the redirected one. Note that this behaviour can be disabled by setting a particular flag in the PE header (see delphi-and-terminal-server-aware).
Sometimes multiple servers are used in a farm which means your application can run on any of these servers, the user is redirected to the least busy server at login (load balancing). Thefore do not use any local database to store things.
If you use an external database or middleware or application server note that multiple users will connect with the same computername and ip address (certain Citrix versions can use Virtual IP addresses to address this).
Many of our customers use our Delphi applications on Citrix. Generally speaking, it works fine. We had printing problems with older versions of Delphi, but this was fixed in a more recent version of Delphi (certainly more recent than Delphi 2007). However, because you are now running under terminal services, there are certain things which will not work, with or without Citrix. For example, you cannot make a local connection to older versions of InterBase, which use a named pipe without the GLOBAL modifier. Using DoubleBuffered would also be a really bad idea. And so on. My suggestion is to look for advice concerning Win32 apps and Terminal Services, rather than looking for advice on Delphi and Citrix in particular. The one issue which is particular to Citrix that I'm aware of is that you can't count on having a C drive available. Hopefully you haven't hard-coded any drive letters into your code, but if you have you can get in trouble.
Generally speaking, your application needs to be compatible with MS Terminal Services in order to work with XenApp. My understanding is that .NET applications are Terminal Services-compatible, and so by extension should also work in a Citrix environment. Obviously, as you're suffering some problems, it's not quite that simple, however.
There's a testing and verification kit available from http://community.citrix.com/citrixready that you may find helpful. I would imagine the Test Kit and Virtual Lab tools will be of most use to you. The kit is free to use, but requires sign-up.
Security can be an issue. If sensitive folders are not "sandboxed" (See Remko's discussion about redirection), the user can break out of your app and run things that they shouldn't. You should probe your app to see what happens when they "shell out" of your app. Common attack points are CHM Help, any content that uses IE to display HTML, and File Open/Save dialogs.
ex: If you show .chm help, the user can right-click within a help topic, View Source. That typically opens Notepad. From there, they can navigate the directory structure. If they are not properly contained, they may be able to do some mischief.
ex: If they normally don't have a way to run Internet Explorer, and your app has a clickable URL in the about box or a "visit our web site" in the Help menu, voila! they have access to the web browser. If unrestrained, they can open a command shell by navigating to the windows directory.
Is it possible to emulate incoming messages using Indy (if it's of any importance: I'm using Indy 10 and Delphi 2009)? I want to be able to create these messages locally and I want Indy to believe that they come from specific clients in the network. All the internal Indy handling (choice of the thread in which the message is received and stuff like that) should be exactly the same as if the message would have arrived over the network.
Any ideas on that? Thanks in advance for any tips.
What you want to do has nothing to do with Indy, as you would need to do this on a much lower level. The easiest way to make Indy believe that messages come from a specific client is to inject properly prepared packets into the network stack. Read up on TCP Packet Injection on Google or Wikipedia. EtterCap is one such tool that allows to inject packets into established connections. However, this is definitely going into gray areas, as some of the tools are illegal in some countries.
Anyway, all of this is IMHO much too complicated. I don't know what exactly you want to do, but a specially prepared client or server is a much better tool to emulate certain behaviour while developing server or client applications. You can run them locally, or if you need to have different IP addresses or subnets you can do a lot with virtual machines.
Indy doesn't have any built-in mechanisms for this but thinking off the top of my head I would recommend building a small test application (or a suite) that runs locally on your development machine and connects to your Indy server application to replay messages.
It should be irrelevant to your Indy server applications if a TCP connection is made either locally or from a remote host as the mechanisms by which a server thread is created and a command processed is identical to both scenarios.
My last gig involved using Indy and all our testing was done with a similar Resender type application that would load local message files and send these to the Indy server app.
HTH and good luck!
One thing you can do would be to create virtual machines to run your test clients, that way they will not be seen as "local machine", and its fairly simple to create a complex network with VMS -- provided you have enough memory and disk space. The other advantage of testing with VM's is you can eliminate the development environment completely when its time to focus on deployment. Amazing how much time that saves alone.
VirtualPC is a free download from Microsoft and works fairly well. VMWare has another option, but costs a little more to get started. For development purposes, I prefer the desktop versions but the server versions also work well. You will still need to have a license to install the virtual OS. MSDN membership is probably the cheapest way to go, and allows you to build test environments for other flavors of the OS.
Indy has abstract stack mechanism for crossplatform support (IDStack.pas) I think u can hack the stack for windows (IdStackWindows.pas). It is a class. U can even consider to derivate it and override some functions to do the hack.