I want to implement Elastic Search as a microservice. We would have 2 apps, where the main app will be connected to Relational Db and the other app will have all the logic for the Elastic search. Now, in the main app, any change would emit a unique event with the payload having all the relevant info around that change.
With the Elastic Search in place, the main app will send this event to the Elastic Search service to process. I am planning to have services for indexing each index table in elastic search. This service will process the event and make the addition/update/deletion based on the event in that index.
A single event can also be processed by multiple services if the index related to that service includes the data change related to that event.
I am just worried if this would be the right way to approach. I can see a single service(class) would grow huge with too many events being processed for indexing.
I would code this in ruby, and there is a gem called elasticsearch-model which is used by most people and is mentioned in every other blog. But I don't want to get restricted by that and also it would not be possible to use it in the kind of microservice setup.
Thanks in advance.
I used elasticsearch-model gem in my current web application you can override it's CRUD methods into your ruby code as used them as REST API Architecture
or
If you don't want to use this gem then you can also create REST api for your elasticsearch and implement your desired features.
Related
I've built a database in Neo4j and use Rails with Neo4jrb as an easy way to manipulate the database. For reasons I explain below, below I call this a knowledgebase (kb) instead of database.
I starting working with a friend who wants to provide access to the kb for users of his app. So I built an API in rails so he can access it.
Now my friend and I are talking about building a membership site with subscription payments. So I figured I needed to figure out how to build Stripe subscription payments into my app. I found this great tutorial by RailsApps which in terms of functionality does exactly what I need, it relies on a gem called Payola which makes Stripe integration easy. Payola is great but it works with ActiveRecord, not Neo4j. I was thinking I'd have to figure out how to do what Payola does with Neo4j.
But now I have the idea of just building a separate app using the RailsApp+Payola approach, and then just hooking that app up to my kb's API.
The reason I say knowledgebase is because its purpose is to structure knowledge within a particular domain. I use the graph database Neo4j because the graph-based data model suits this goal, for example (object of type A) -[has a certain influence on]-> (object of type B).
So I like my idea of using a separate app for managing users and subscriptions because then I avoid mixing app specific data with domain knowledge in the database.
So I guess my concern is, what might I be missing? Will speed be a concern if this membership site has to access an API every time a page loads? Would there be unusual security concerns?
It's perfectly fine to use another database next to neo4j. You should always use the database that fits your needs I think :)
Suppose we have this scenario:
www.main.com - Main interface where admin (foo, bar, etc..) can store products, based on their own e-commerce
www.foo.com - Sample store that sells items from the "foo" store
www.bar.com - Sample store that sells items from the "bar" store
The problem is finding a way to centralize the database structure and models.
I prefer to keep every single store in separated apps (so I exclude rails engines).
For instance, if a user buys something in the "foo" store, I need to interact with the main db and update it.
How can I do this?
Rails works much better with a "one database, one app" model. There are lots of ways how you can share models across apps (gems, engines, git submodules, etc), but none of those ways is great. You end up introducing lots of overhead in your development, deployment, and testing process. You also invite lots of hidden dependencies between code, as Rails doesn't give you easy way to keep clean abstraction (for example, you wrote a helper for store Foo, and then your coworker used that helper in store Bar, and then every time you change Foo, Bar breaks).
I recommend a centralized API approach instead:
api.foobarmain.com - a app/service that provides RESTful API for all the functionality of all stores.
This app has all the db models, and it exposes them as resources in the API for other apps to interact with.
This app can have an admin UI for all the stores, if you need it. Alternatively, admin UI may be another client of an API.
www.foo.com - a full stack app that interacts with API at api.foobarmail.com
There is no shared database connection to API, everything that you need to interact has to be exposed via API.
There will be no shared code between www.foo.com and www.bar.com. Code reuse happens only by virtue of using the same API app/service.
From the perspective of www.foo.com, the model layer (in MVC) is powered by API, not by database.
You can still have its own database on www.foo.com if you need to store data specific to www.foo.com only.
www.bar.com - another full stack app that interacts with API
so on ...
Another way is using multiple schema if you are using Postgresql. I have a similar issue with a new project that I about to start.
You can use gem 'apartment' to deal with different schemas. The queries will be a bit complicated but with different schemas you will end up with one database and you can create different namespaces to respond accordingly.
You can set up so app select the correct schema based on the Domain or Subdomain.
Here is the link: https://github.com/influitive/apartment
I'm in the process of setting up a centralised data management system + CAS for a series of apps that currently have duplicated data across many databases that is a nightmare to manage.
I've got the CAS system implemented nicely with rubycas-server and rubycas-client, but I'm running into problems when it comes to actually sharing the user data and other models between client apps. For example:
CAS/Central system (share a DB) expose a base User as JSON
App1 extends user with information about favourite colours
App2 just cares about having a user
I want to be able to let App1 add the data it cares about the object returned from the API, without burdening App2 with that info.
I'm currently playing around with writing a gem for the client apps which can consume the JSON (with ActiveResource or ROAR, not sure which yet), however the client apps need to decorate/extend their models with additional data which is currently managed via ActiveRecord. So I'm not sure what the best way to go about extending those gem provided instances would be.
So far I've experimented a bit with having the gem contain a bunch of modules which the ActiveRecord models can then include, but this has issues when needing to deal with class methods (not insurmountable but any means). I've tried looking around for resources/guides for best practices in this sort of situation, but haven't been able to find anything.
tl;dr summary: Multiple client apps need to use and extend data provided by a JSON api. What's the recommended practice for doing this?
Firstly let me mention that I'm new to web-frameworks.
I have to write my first web-app for a Uni project. I spent two weeks learning Grails and Django. Started working with Rails yesterday and loved it. So I've decided to go with it and discard my work in the other frameworks.
About the app
It's supposed to be a Twitter app that utilizes Twitter's Streaming API to record tweets which match a set of specified filters. (I'm going to use the Tweetstream gem which takes care of connecting to Twitter and capturing matching tweets).
The app's web interface should have the following functionality -
Creating new requests The user inputs a set of filter parameters (keywords to track) & URL/username/password of an existing PostgreSQL or MySQL database.
When a request is created, the web-app spawns a background ruby process. This process connects to Twitter via the Tweetstream gem. It also connects to the database specified by the user to stores received tweets.
View/terminate of existing requests
The user should be able to see a list of requests that are running as background processes by visiting a URL such as /listRequests.
See further details about a process/terminate the process
The user should be able to go to URL such as /requests/1/detail to view some details (e.g how long request has been running, number of tweets captured, etc). The user should also be able to terminate the process.
My inexperience is showing as I'm unable to comprehend -
what my models should be (maybe Request should be a model. Tweet doesn't need to be a model as it's not being stored locally)
how I'm going to connect to remote databases.
how I can create background processes (backgroundrb??) and associate them with request objects so that I can terminate then when the user asks.
At the end of the day, I've got to build this myself, so I'm not asking for you to design this for me. But some pointers in the right direction would be extremely helpful and appreciated!
Thanks!
Hmm.
Since the web app is just a thin wrapper around the heavy-lifting processes, it might be more appropriate to just use something like Sinatra here. Rails is a big framework that pulls in lots of stuff that you won't need for this project, even though it will work.
Does the "background process" requirement here strictly mean a separate process, or does it just mean concurrency? TweetStream uses the EventMachine gem to handle updates as they come, which uses a separate thread for each connection. It would be quite possible to spawn the TweetStream clients from a simple Sinatra web app, keep them in a big array, have them all run concurrently with no trouble, and simply run stop on a given client when you want it to stop. No need for a database or anything.
I'm not sure exactly what your prof is looking for you to do here, but MVC doesn't really fit. It's better to work with the requirements than to mush it into a design pattern that doesn't match it :/
Even so, I <3 Rails! Definitely get on that when you're working primarily with objects being represented in a database :)
Quite a project. Most of what will be challenging is not related to rails itself, but rather the integration with background processes. backgroundrb is a bit out of fashion. The last commit on the main github project is over a year ago, so it's likely not up to snuff for Rails 3. Search around and evaluate your options. Resque is popular, but I'm not sure if your real-time needs match with its queue-based structure.
As for your app, I see only a single model, but don't call it request. That's a reserved name in rails. Perhaps a Search model, or something along that line.
Connecting to different databases is straight forward but will require direct configuration of your ActiveRecord class during operation rather than using database.yml.
The iTunes Enterprise Partner Feed is "a data feed of the complete set of metadata from iTunes and the App Store" and "is available in two different formats - either as the files necessary to build a relational database or as stand-alone flat files that are country and media dependent."
I need to consume the data from this feed (which is essentially exported into flat files) and allow linking of my own Model objects (User, Activity, etc.) to data provided by the feed (App, Developer, etc.) The data is provided as a weekly full export and a daily incremental export.
I have two ideas for ways to implement this:
Create all of the models in my rails app and write my own importer that will insert/update records directly into my app's database daily via cron using models I've created (App, Developer, etc.)
Keep this database entirely separate and open up REST API that my own app will consume
My naive approach with #1 to keep everything in the Rails app is based on the need to be able to observe changes in the data I'm getting from the EPF. For example, if an App's description is updated, I want to be able to create an Activity object via an observer to track that update.
On one hand #2 feels like a better approach because it creates a standalone API into the data that can be consumed from several different apps I create. On the other hand, I'm just not sure how I'd accomplish the data change notifications without using observers directly on my own models. Or, how I would even consume the data in an object oriented way that could still be used with my own models. It feels like a lot of duplicate work to have to query the API for, say, an App's data, create a proper Active Record object for it and then save it just so it can be linked to one of my own models.
Is there a standard way to do this that I'm totally missing? Any pointers?
EDIT: Rails engines sound interesting but it would mean that each app would still need to consume and insert the data separately. That doesn't sound so DRY. It sounds more and more like a REST API is the way to go. I just don't know how to bridge the gap from API to Active Record model.
Rails Engines might be a good fit for this. You can create a Rails Engine gem and add all of your models and rake tasks to consume the data. Then you can include this gem in any app that uses it and also create an API app which includes the gem. You should be able to create observers in your other apps that interact with the gem.
I have quite a few apps that interact with each other and this approach works well for me. I have one central app that includes all the engines that consume data and I run all of my cronjobs from this app. I use the use_db plugin which allows my app to communicate with different databases. Each engine has use_db as a dependency and I keep the database configuration inside the gem. One example:
Engine Gem = transaction_core
This gem consumes transaction data from a source and inserts it into my transaction database.
The gem is included in my central app and I pull the transaction data using a rake task on the cron
I include this gem in several other apps that need to use the transaction data. Since the engine automatically adds the models and database config to the app, there is no additional work required to use the models in the app.
I have not used observers inside an app that includes my engines, but I see no reason why it would not work. With the engine the models work as if they are in your app/models directory. Hope this helps!
Modest Rubyist has a good 4 part tutorial on Rails 3 plugins that includes Engines:
http://www.themodestrubyist.com/2010/03/05/rails-3-plugins---part-2---writing-an-engine/