Can we connect to multiple databases on directus simultaneously?
I wanted to connect postgres, SQL and aws at the same time on Directus
The closest the Directus team has come to discussing multiple database management has been this thread on their Github:
https://github.com/directus/directus/discussions/12699
As per the last message by a maintainer:
This is not something that's officially supported [...] Directus wasn't designed to handle this use case out of the box 👍🏻
Within that same thread a user had manually modified the usage of Knex to use pools to connect to different Databases but it seems unreliable.
Currently the best way to get the benefits of using Directus with multiple databases is to set up a server with multiple instances of Directus running, each pointing to a different database, and using NGINX to access each Directus instance under a separate sub-path.
For example if you have your app data on Postgress and create reports on a separate MySQL database, you could setup NGINX to proxy one Directus connected to Postgres to /api and another Directus connected to MySQL to /reports.
Some useful links:
https://learndirectus.com/how-to-manage-multiple-projects-in-directus/
https://github.com/directus/directus/discussions/4480
https://github.com/directus/sdk/issues/51
I'm developing an app which live-streams video and/or audio from different entities. Those entities' IDs and configurations are stored as records in my DB. My app's current architecture is something such as the following:
a CRUD API endpoint for system-wide functionalities, such as logging in or editing an entity's configuration.
N-amount of other endpoints (where N is the number of entities and every endpoint's route is defined by the specific entity's ID, like so: "/:id/api/") for each entity's specific functionalities. Each entity is loaded by the app on initialization. Those endpoints are both a REST API handler and a WebSocket server for live-streaming media received from the backend which was configured for that entity.
On top of that, there's an NGINX instance which acts as a proxy and hosts our client files.
Obviously, this isn't very scalable at the moment (a single server instance handles an ever-growing amount of entities) and requires restarting my server's instance when adding/deleting an entity - which isn't ideal. I was thinking of splitting my app's server into micro-services: one for system-wide CRUD, and N others for each entity defined in my DB. Ultimately, I'd like those micro-services to be run as Docker containers. The problems (or questions to which I don't know the answers) I'm facing at the moment are:
How does one run Docker containers dynamically, according to a DB (or programmatically)? Is it even possible?
How does one update the running Docker container to be able to reconfigure that entity during run-time?
How would one even configure NGINX to proxy those dynamic micro-services? I'm guessing I'll have to use something like Consul?
I'm not very knowledgeable, so pardon me if I'm too naive to think I can achieve such architecture. Also, if you can think of a better architecture, I'd love to hear your suggestions.
Thanks!
I have two mongodb servers. Would it be possible to specify, based on the collection, on which server the query will be performed?
Preferably, I would like to do this at the application level (Rails + mongoid)
The reason is that I'm migrating some data and have this specific collection that will need to live temporary on a different server on the same vpc.
Thanks
So I have a Rails application. It currently runs separate as front-end and back-end + database.
I need to scale it to have several back-end servers.
The backend server has Resque background workers running (spawned by user front-end requests). It also relies heavily on callbacks.
I am planning the following setup:
|front-end| --- |load-balancer (haproxy or AWS ELB)| --- Server 1 ---- Postgresql Database (+++ other DBs added via replication later if needed)
\___ Server 2 ---/
++ (other servers added in the same fashion later )
I have concerns about how to deal with putting Database on a separate machine in this case.
1) I intend to create a new empty Rails app with schema identical to initial back-end. Have it running and accepting updates / posts via HTTP and keep connected via remote SSH (to trigger :after_commit callbacks in back-end). Is it a bettergood idea?
2) I am using Postgresql and intend to switch to an enterprise DB once the need arises. Currently the need is to scale the part of back-end that does processing not the database.
3) Does this approach seem scalable?
I'm not sure I really understand your question. Generally in production applications the database layer is separate from the application layer. I can't quite tell if this pertains to you, but it's definitely an interesting watch. http://vimeo.com/33263672 . It talks about using a redis layer between the rails and db layers to facilitate queuing, and creating a zero downtime environment. It seems like it would be a better solution than using a second rails stack? I think it should look something like this;
|ELB| Web Servers |ELB| Application Servers |RRDNS| Redis Servers | PostGreSQL Servers |
If I am understanding your meaning. If not, that video link is still worth a watch :)
UPDATED 2010-11-25
A legacy stand-alone application (A1) is being re-created as a web application (A2).
A1 is written in Delphi 7 and uses a MS Access database to store the data. A1 has been distributed to ~1000 active users that we have no control over during the build of A2.
The database has ~50 tables, some which contain user data, some which contain template data (which does not need to be copied); 3-4 of these user tables are larger (<5000 records), the rest is small (<100).
Once A2 is 'live', users of A1 should be able to migrate to A2. I'm looking for a comparison of scenario's to do so.
One option is to develop a stand-alone 'update' tool for these users, and have this update tool talk to the A2 database through webservices.
Another option is to allow users to upload their Access db (~15 MB) database to our server, run some kind of SSIS package (overnight, perhaps) to get this into A2 for that user, and delete the Access db afterward.
Am I missing options? Which option is 'best' (I understand this may be somewhat subjective, but hopefully the pro's and cons for the scenario's can at least be made clear).
I'll gladly make this a community wiki if so demanded.
UPDATE 2010-11-23: it has been suggested that a variant of scenario 1 would be to have the update tool/application talk directly to the production database. Is this feasible?
UPDATE 2011-11: By now, this has been taken into production. Users upload the .zip file the .mdb is in, which is unpacked and placed in a secure location. A nightly SSIS scheduled job comes along and moves the data to staging tables, which are then moved into production through SP's.
I would lean toward uploading the complete database and running the conversion on the server.
In either case you need to write a conversion program. The real questions is how much of the conversion you deploy and run on the customers' computers. I would keep that part as simple as possible, i.e. just the upload. That way if you find any bugs or unexpected data during the conversion you can simply update the server and not need to re-deploy your conversion program.
The total amount of data you are talking about is not too large to upload, and it sounds like the majority of it would need to be uploaded in any case.
If you install a conversion program locally it would need a way to recover from a conversion that stopped part way through. That can be a lot more complicated than simply restarting an upload of the access database.
Also you don't indicate there would be any need for the web services after the conversions are done. The effort to put those services together, and keep them running and secure during the conversions would be far more than a simple upload application or web form.
Another factor is how quickly your customers would convert. If some of them will run the current application for some time period you may need to update your conversion application as the server database changes over time. If you upload the database and run the conversion on the server then only the server conversion program would need to be updated. There would not be any risk of a customer downloading the conversion program but not running it until after the server databases were updated.
We have a similar case where we choose to run the conversion on the server. We built a web page for the user to upload their files. In that case there is nothing to deploy for the new application. The only downside we found is getting the user to select the correct file. If you use a web form for the upload you can't pre-select file name for the user because of security restrictions. In our case we knew where the file was located but the customers did not. We provide directions on the upload page for the users to help them out. You could avoid this by writing a small desktop application to perform the upload for the users.
The only downside I see to writing a server based conversion is some of your template data will be uploaded that is un-needed. That is a small amount of data anyway.
Server Pros:
- No need to re-deploy the conversion due to bugs, unexpected data, or changes to the server database
- Easier to secure (possibly), there is only one access point - the upload. Of course you are accepting customer data in the form of an access database so you still can't trust anything in it.
Server Cons:
- Upload un-needed template data
Desktop Pros:
- ? I'm having trouble coming up with any
Desktop Cons:
- May need multiple versions deployed
As to talking to a server database directly. I have one application that talks to a hosted database directly to avoid creating web services. It works OK, but if given the chance I would not take that route again. The internet is dropped on a regular basis and the SQL Providers do not recover very well. We have trained our clients just to try again when that happens. We did this to avoid creating web services for our desktop application. We just reference the IP address in the server connection string. There is an entire list of security reasons not to take this route - we were comfortable with our security setup and possible risks. In the end the trade off of using the desktop application with no modifications was not worth having an unstable product.
Since a new database server to be likely one the standard database engines in the industry, why not consider linking the access application to this database server? That way you can simply send your data up to sql server that way.
I'm not really sure why you'd consider even suggest using a set of web services to a database engine when access supports an ODBC link to that database engine. So one potential upgrade path would be to simply issue a new application in access that has to be placed in the same directory as to where their current existing data file (and application) is now. Then on startup this application can simply RE link all of its tables to your existing database, plus come with a pre link set of tables to the database server. This is going to be far less work in building up some type of web services approach. I suppose part of this centers around where the database servers going to be hosted, but in most cases perhaps during the migration period, you have the database server running somewhere where everyone can get access to it. And a good many web providers allow external links to their database now.
It's also not clear that on the database server system you're going to create separate databases for each one, or as you suggest in your title it's all going to be placed into one database. Since is going to be placed into one database, then during the upsizing, an additional column that identifies the user location or however you plan to distinguish each database will be added during this upsizing process to distinguish each user set of data.
How easy this type of migration be will depend on the schema and database layout that the developers are using for the new system. Hopefully and obviously it has provisions for each user or location or however you plan to distinguish each individual user of the system. So, I don't suggest web services, but do suggest linking tables from the Access application to the instance of SQL server (or whatever server you run).
How best to do this will depend on the referential integrity and business rules that must be enforced, if there are any. For example, is there the possibility of duplicates when the databases are merged? I gather they are being merged from your somewhat cryptic statement: "And yes, one database for all, aspnet membership for user id's".
If you have no control of the 1000+ users of A1, how are you going to get them all to convert to A2?
Have you considered giving them an SQL Server Express DB to upgrade to, and letting them host the Web App on their own servers?