I am evaluating Apache Ignite data grid for the production usage.
One of critical requirements is to have a well defined stragegy for upgrading a large system to a binary incompatible version (usually unavoidable when using binary protocols like Ignite does). More specifically, upgrading the Ignite infrastructure independently (before or after) from the large number of Ignite client node components and/or Ignite thin clients.
So wondering what would such process look like, for situation when upgrading all the components of the system as a big-bang is not practically possible.
If your primary objective is clients which should access cluster without downtime during upgrade, I can recommend that most of those clients should be 'thin' clients, such as JDBC client, ODBC client, REST or Java/C#/C++/node.js thin clients which are under active development currently. They have no strict version checking.
So you should avoid using 'thick' clients (a.k.a. Apache Ignite client nodes) and only for operations which can't be performed by thin clients. Or use Rolling Upgrades as mentioned.
First, Ignite cares about the compatibility of its binary format and doesn’t change it without some fallback mechanisms. In particular, backward compatibility of the persistent storage (ability to upgrade and continue to use the same DB files) is preserved.
Second, to upgrade a production cluster there is a Rolling Upgrade feature offered by GridGain (built on top of Ignite). See this for details: https://docs.gridgain.com/docs/rolling-upgrades.
Related
I am working on a Spring boot project which uses Hazelcast as Cache. I am using the community edition of that. I have couple of questions,
I wanted to know whether there is minimal provision provided in community edition for security features. I know that we can provide unique group name so other nodes cannot join the cluster. But is there any other way?.
I also tried with hazelcast.application.validation.token but it is not working. What is the correct way to check with this property.
Also, hazelcast communicating using TCP is not blocked by spring boot. Is there any way in spring security to add some security feature to hazelcast?
I suppose, you're using Hazelcast 4.0 or later. The property hazelcast.application.validation.token was removed in version 4.
Maybe you've already looked into this answer - it's related to Hazelcast 3.y versions. Some info is still valid though.
The basic protection approach in Hazelcast version 4 (OS) is to set different cluster names (equivalent of group name in Hazelcast 3).
You can use the advanced network feature which allows you to have separated port numbers for different protocols (member protocol, client protocol, REST, ...). Then you can use OS level protection - such as firewall - to protect these endpoints.
You can also disable binding server sockets to all network interfaces (default behavior) and control which interface is used.
I don't think the Spring security provides a feature which would help you with protecting Hazelcast endpoints, but I'm not Spring expert, so maybe I'm wrong.
Are there major disadvantages using the embedded Firebird 3 in a multi-user application server (Delphi Webbroker) instead of the full blown server install?
The application usually has very short transactions with low data volume.
As far as I am informed accessing one database file with multiple threads through the embedded server is not problematic but user security is not available. As the application server does the rights stuff I do not need Firebird security.
But will I loose performance or things like garbage collection?
Firebird Embedded provides all the features (except network access and authentication) that a normal Firebird server provides. However, because it is in-process, any problems that cause your application to crash, will take Firebird with it and vice versa.
Other possible downsides:
Garbage collection will - as far as I know - always use the 'cooperative' model (where the connection to find old record versions, is the one that cleans it up),
You can't use other tools to access your database remotely which may make administration harder,
You can't put your database on a separate server from your web application (think of security requirements).
Personally, I would only choose Firebird Embedded if the situation calls for it. In all other situations, I will use Firebird Server.
Our future architecture is to move towards docker /micro services. Currently we are using JBoss EAP 6.4 (with potential to upgrade to EAP 7) and Tomcat.
According to me JEE container is too heavy (slow, more memory, higher maintenance etc) for microservices environment. However, I was told that EAP 7 is quite fast and light weight and can be used for developing microservices. What is your input in deciding EAP 7 vs Tomcat 8 for docker/microservices? Cost and speed would be consideration.
EAP7 vs Tomcat 8 is an age old question answered multiple times here, here and here.
Tomcat is only a web container where as EAP7 is an application server that provides all Java EE 7 features such as persistence, messaging, web services, security, management, etc. EAP7 comes in two profiles - Web Profile and Full Profile. The Web Profile is much trimmer version and includes only the relevant implementations typically required for building a web application. The Full Profile is, as you'd expect, contains full glory of the platform. So using EAP 7 Web Profile will help you cut down the bloat quite a bit.
With Tomcat, you'll have to use something like Spring to bring the equivalent functionality and package all the relevant JARs with your application.
These discussions are typically helpful when you are starting a brand new project and have both Java EE or Spring resources at hand. Here are the reasons you may consider using EAP7:
You are already using EAP 6.4. Migrating to EAP 7 would be seamless. Using Docker would be just a different style of packaging your applications. All your existing monitoring, clustering, logging would continue to work. If you were to go with Tomcat, then you'll have to learn the Spring way of doing things. If you have time and resources and willing to experiment, you can go that route too. But think about what do you want to gain out of it?
EAP 7 is optimized for container and cloud deployments. Particularly, it is available as a service with OpenShift and so you know it works OOTB.
EAP 7 will give a decent performance boost in terms of latency and throughput over EAP 6.4. Read https://access.redhat.com/articles/2607521 for more details.
You may also consider TomEE. They provide Java EE stack integrated with Tomcat.
Another option, as #Federico recommended, consider using WildFly Swarm. Then you can really start customizing on what parts of the Java EE platform do you want. And your deployment model is using a JAR file.
As for packaging using Docker, they all provide a base image and you need to bundle your application in it. Here are a couple of important considerations for using a Docker image for microservices:
Size of the Docker image: A container may die unexpectedly or orchestration framework may decide to reschedule it on a different host. A bigger image size will take that much more longer to download. This means your perceived startup time of the service would be more for a bigger image. This also means dynamic scaling of the app would take longer to be effective.
Bootup time of the image: After the image is downloaded, the container may startup quickly but how long does it take for the application to be "ready"?
As a personal note, I'm much more familiar with Java EE stack than Tomcat/Spring, and WildFly continues to be favorite application server.
Besides using traditional Application servers, which are not really that heavy, you can taste different flavor of Java EE, called microcontainers.
Java EE is just a set of standards. Standard results in an API specification and everyone is then free to implement the specification. An Application Server (AS) is mainly a fine-tuned collection of this functionality. Those APIs were not brought to life for no reason. These represent functionality commonly used in projects. Application server can be viewed as a "curated set" of those functionalities. This approach has many advantages - AS has many users, therefore it is well tested over time. Wiring the functionality on your own may result in bugs.
Anyhow, a new age has come, where with Docker, the application carries its dependencies with it. The need for a full-blown application server with all the functionality ready to be served to applications is no longer required in many cases. In times past, the application server did not exactly know which services the applications deployed will need. Therefore, everything was bundled in. Some of more innovative AS like WildFly instantiated only the services required. Also, there are Java EE profiles which eased the monolith Application Server a little bit.
Right now, we usually ship the application together with it's dependencies (JDK, libraries, AS) inside a Docker - or we're heading there. Therefore, an effort to bundle exactly the right amount of is a logical choice. But, and it is a "big but", the need for the functionality of the AS is still relevant. It is still good idea to develop common functionality based on standards and common effort. It only no longer seems to be an option to distribute it as one big package, potentially leaving most of the APIs inactive. This effort has many names, be it microcontainers, uberjar creators ...
WildFly Swarm
Payara Micro
Spring Boot*
There are Java EE server so light it is doubtful to use anything else.
* Spring Boot is not based on Java EE and in default configuration present in the Getting Started guide, Tomcat is used internally.
WebSphere Liberty
Apache TomEE
The key point is, your Java EE application should be developed as an independent Java EE application. Wrapping it with "just enough" functionality is delegated onto these micro solutions. This is, at least in my humble opinion, the right way to go. This way, you will retain compatibility with both full-blown AS and micro-solutions. The uber-jar, containing all the dependencies, can be created during or after the build process.
WildFly Swarm or Payara Micro are able to "scan" the application, running only the services required. For a real-world application, the memory footprint in production can be as low as 100 MB - for a real-world application. This is probably what you want. Spring Boot can do similar things, if you require Spring. However, from my experience, Spring Boot is much more heavyweight and memory hungry than modern Java EE, because it obviously has Spring inside, so if you are seeking lightweigtness in terms of memory consumption, try Java EE, especially WildFly Swarm (or pure WildFly) and Payara Micro. Those are my favorite AS and they can be really, really small. I would say WildFly Swarm is much easier to start with, Payara micro requires more reading, but offers interesting functionality. Both can work as a wrapper - you can just wrap your current project with them after the build phase, no need to change anything.
Payara Micro even provides Docker images ready to use ! As you can see, Java EE is mature and ready to enter Docker lands :)
One of the very good and reliable resources is Adam Bien, for example in his Java EE micro/nanoservices video. Have a look.
In which mode neo4j database should be used embedded or rest server?
My main concerns are :
Performance
Horizontal scaling (HA,Clustering) - essential as application is very big.
Transactional support(in frameworks like SDN,Grails Plugin,structr etc.)
Deployment server support like amazon,GrapheneDB etc.
Easiness of switching from one to another
Scaling(size of database)
Disclaimer: I'm one of the founders of GrapheneDB.
I'm not an expert in embedded mode so my answer might be biased but I will try my best:
Embedded is more performant at this time than server
Clustering is supported in embedded as well as in server
Transactional support is available in both modes AFAIK. Spring Data, however has currently bad performance over Rest/server.
From my POV embedded has the disadvantage of being coupled to your app/server deployment.
There is one more option which you haven't brought up, which is using unmanaged server extensions.
Using extensions you can get the best of both modes:
You write your code on top of the Java API and it's executed locally, so you get extremely good performance.
You can run the server in server mode, making operations easier and also enabling you to host on a separate remote host, on any cloud environment.
GrapheneDB supports unmanaged extensions and it's the option we currently recommend for scenarios where extra performance is needed.
Has somebody real experience with firebird databases over the internet?
I have a typical windows accounting/ERP software (done with delphi) that works with the firebird database server pretty well.. Now my users (300 aprox. now, but should increment) also want to work "in the cloud" (connecting from the office, from the laptop, from the house, etc.). It is a lot of work of recreating everything to a standard web application (let's say for example, HTML+CSS+JS+PHP+MYSQL), so I'm considering keeping the win client (I don't care about other OSes) but instead of the server living in the clients LANs moving it to a pair of dedicated servers that I will contract (one primary and one secondary againts failures for starting).
Searching I've come across this faq http://www.firebirdfaq.org/faq53/ that explains that the fb protocol it isn't ideal for working in the internet, but still all my users today have at least a 1MBbit/sec ADSL internet connection (I don't think that to be slow as the faq denotes).
Somebody have done this? what was the experience? how secure are fb servers for being open to the internet? how well they scale?
I know that building a "middleware" with SOAP for example will be more normal, but still the solution I'm evaluating here is much more fast and easy (still I have some work with the replication, backup, hearbreath services, but it's much less than redoing everything for the web).
Thanks! Edit: FB version: 2.5.
I had being trying to "push" the Firebird Core developers to improve the Firebird protocol to get better speed with high latency network (aka. Internet). Recently, Dmitry Yemanov published some articles in his blog about this subject (dyemanov.blogspot.com). It seems that there is margin for optimizations, and I would really like to see this coming in FB 2.5.3 and FB 3.0, although there is no warranty for this happening in those versions or anytime soon. You can vote in such improvement here: http://tracker.firebirdsql.org/browse/CORE-2530
Safety? You may try to set up a VPN. It also may help with speed, since most of the VPNs software out there (Zebedee, etc) can compress the data being transfered, helping to speed up data transfer in some cases.
Some of my customers do use Firebird traditional C/S over the internet. It is much slower compared to local network, and of course, how much slower depends basically on the link speed and latency. You can do some optimization at the client side too, using metadata cache, etc. but don't expect miracles with the currently protocol. I would say that for whole day working, using Terminal Services would be a better option for now.
The response about the scaling question
Firebird runs in production on large big iron servers : 512G of ram 100.000 concurrent users
We run Firebird to power larger systems (for 12 government agencies
and 3 banks). It has approximately 100000 end users multiplexed
through 2500 (max) pooled connections
https://plus.google.com/111558763769231855886/posts/Q1ACy1yyTgP
The protocol in Firebird 2.5 is improoved there is still room left for 3.0 but you can check
what is already done
http://asfernandes.blogspot.com/2009/07/network-latency-influence-on-firebird.html
And the future enhancements in 3.0
http://www.firebirdnews.org/?p=6953
To protect your connection i guess the best bet is ssl/ssh tunnel (it can be a opnvpn)
with high compression option
http://mapopa.blogspot.com/2010/11/securing-firebird-using-ssh-tunnel.html
FB protocol problem isn't about bandwidth, but latency. In my experience, some operations can be very slow over internet/VPN compared to LAN or local connection. I haven't examined issue further since I don't really run applications over internet connection.
However, I suggest three-tier model for application. Create own application server, which runs on database server/same network. Let the clients talk with application server and you get maximum performance.
There are some N-tier application/middleware frameworks for Delphi:
RemObjects SDK and DataAbstract
RealThinClient
kbmMW
Delphi's own DataSnap
MidWare
With those you can get data compression, encryption, binary messages (faster than SOAP) etc.
You can implement TCP/IP packets encryption/decryption directly in the firebird engine itself.
Personnaly, i have downloaded the Firebird 2.5 source code and injected secure tunnelization code directly in his low level communication layer (the INET socket layer). Now, encryption/decryption is done directly by the firebird engine for each TCP/IP packet both at the server side and the client side (fbclient.dll).
Then there is no need to re-structure the client application except adding one line of code that provide the secret key you choose to crypt communication to the fbclient.dll. The same secret key must be declared in the firebird.conf file of your server installation.
I have also implemented a proxy negociation solution in the fbclient.dll in order to allow to TCP/IP packets to pass throught any proxy server (like Microsoft ISA Server for example).
For us, this architecture is functional for more than one year in a real production system.
kbmMW CodeGear Edition is free but without source. It can be used for commercial apps.
Download it after registering at: https://portal.components4developers.com
In case you see certificate errors (you shouldnt but I know we have heard that some actually do), accept and ignore them. The site is valid despite the cert.error.
kbmMW CodeGear Edition contains a subset of kbmMW Professional Edition, but supports the following Delphi database API's:
Borland Database Engine
DBExpress
kbmMemTable
SQLite3
It supports binary, binary over HTML, XML and SOAP protocols in communication with clients.
It contains everything you need incl.
unified remote custom method invocation
unified remote dataset query, execute and data change resolving
unified database meta data handling and creation (tables, fields, indexes, generators/sequencers)
optional automatic proxying of requests to another server and proxying results back to original requester
full native XML DOM and SAX support
full dataset briefcase support as CSV, or binary data
advanced but simple to use wizard for creating new application server services
THere is one caveat though. Newest version of kbmMW CodeGear Edition always only supports newest Delphi version. You can still download older kbmMW CodeGear Editions matching older Delphi releases.
kbMMW Professional Edition and kbmMW Enterprise Edition do not have such limitations, and currently supports D7, D2006, D2007, D2010, DXE, DXE2 along with Embarcadero C++ counterparts.
best regards
Kim Madsen
www.components4developers.com