I've been working on implementing an automated trading system in Grails based on Interactive Brokers' API (brief overview here: Grails - asynchronous communication with 3rd party API) for longer than I care to admit. This is a high-frequency trading strategy, so it's not as simple as placing an order for 100 shares and getting filled. There's a lot of R&D involved, so my architecture and design have been and continue to morph and evolve over time.
What has become clear over the past month or so is that the asynchronous nature of the API is killing me. I have a model of my intended position within Grails, but this does not automatically reflect the actual state at the brokerage. There is a process of creating orders, some of which get filled now, some later, and some never. There could be partial fills, canceled or rejected orders, or any number of other errors. And the asynchronous updates have turned into a nightmare of pessimistic locks, ugly relationships and dependencies between Positions, Intents, Orders, Transactions, etc. And still, with all that unelegant, smelly code, there are times when my internal model gets out of sync with the actual state of the brokerage account. And that is a very dangerous situation.
So, I'm realizing that I need some kind of async framework that will allow Grails and the IB API to maintain precisely the same state without fail. I am somewhat familiar with Gpars, Akka, Promises, and Actors but only on the surface; I have no hands-on experience with any of them. Just recently, I saw Parse's Bolt Framework Tasks and wondered if that might be a good fit. My need is not really for parallelism or multi-threading of computations or collections. All I am trying to do is make sure that the async callbacks from IB are properly reflected in the Grails domain classes at all times. And I'm hoping that the right framework will allow me to delete tons of ugly spaghetti code that I've written trying to solve this problem.
What I need is a recommendation on the right framework, model, or architecture that addresses this problem. I welcome any recommendations, whether or not I mentioned them above.
Related
After a bunch of googling, I don't really see a good way to have Orleans work with an existing Relation-Database backend.
Every example that I have found for doing this relies on adding columns to deal with concurrency and I haven't really seen any samples of how to use Orleans with, as is the typical example, the northwind database or something.
This leads me to believe that Orleans is not really intended to be used in this way (because if it was I would expect someone somewhere to have create a sample app demonstrating it by now). Am I missing something? Has anyone seen a sample project or blog post explaining how to use, say, an existing EF context with Orleans? This needs to be done without adding additional columns. I am working with data that is controlled by multiple teams in a mission critical system, so there is no way I will get approval to start adding columns to hundreds of tables.
As #Milney says, to my knowledge, there is nothing special in Orleans that would prevent you from using a normal EF DbContext, no extra columns required.
If, on the other hand, your issue is that other applications are causing concurrency issues from outside Orleans, then I think you'll need to deal with them as you would in any application (e.g. with optimistic concurrency checks).
But it's possible I'm misunderstanding your use case.
I watched some videos, read some blogs about it. SO has many questions and answers on that subject but I can not find anywhere exact answer for my question.
Almost every question and answer has a lack of usage context.
I have one middle sized, asp.net-mvc, monolith application which is running in one process on IIS. I want to (refactor and) go all the way with DDD (and CQRS without separated storage for reads and writes for now) but for me it looks like impossible mission without breaking some rules/guides/etc.
Bounded Context
For example I have more than one BCs. Each should not cross their boundaries which means should not share their storage. Right?
It is not possible if you use the whole known (everywhere scattered over the web) solution to work with NHibernate session and UoW.
Aggregate Root
Only one AR should be modified in one transaction. When others ARs are involved should introduce eventual consistency (if I remember those are Eric Evans words).
I try to do it but it is not easy in app like that. Pub/Sub not working as desired because if event is published then all subscribers are take their action within one transaction (NSB/MT does that way).
If event handlers wants to modify others ARs they should be executed in separated transactions, right?
Is it possible to deal with it in monolith application (application where whole code works in one process)?
It is not possible if you use the whole known (everywhere scattered
over the web) solution
[...]
if event is published then all subscribers are take their action
within one transaction
I think you're setting yourself useless and harmful constraints by trying to stick to some "state of the art".
Migrating an entire application to DDD + CQRS is a massive undertaking. Some areas of it don't have well-documented beaten paths yet and you'll probably have a fair bit of exploration to do. My best advice would be to stay at a reasonable distance from "the way people do things". Both in traditional ASP.Net web apps because mainstream practices often don't match the way DDD+CQRS works, and in CQRS itself because the case studies out there are few and far between and most probably very domain specific, with a tendency to advocate the use of heavy tools which may not make sense in your context.
You may need to think out of the box, adopt things incrementally and refrain from goldplating everything. You'll be better off starting with very simple implementations that suit your needs exactly than throwing a ton of tools and frameworks at your codebase.
For instance, do you really need a service bus or could a simple Observer pattern suffice ?
Regarding NHibernate, most implementations out there use a (single) Session Per Request approach, but just because it's the most popular doesn't mean it's the only one. Have you tried using multiple ISessions (one for each BC) available at a more programmable level, such as per-action, or managed entirely manually ? Conversely, have you considered sharing a database between Bounded Contexts at first and see for yourself if that's bad or not ?
We know Reflection is a quite expensive engine. But nevertheless ASP.NET MVC is full of it. And there is so much ways to use and implement additional reflection-based practices like ORM, different mappings between DTO-entities-view models, DI frameworks, JSON-parsing and many many others.
So I wonder do they all affect performance so much that it is strongly recommended to avoid using reflection as much as possible and find any another solutions like scaffolding etc? And what is the tool to perform server's load testing?
There's nothing wrong with Reflection. Just use it wisely, a.k.a cache the results so that you don't have to perform those expensive calls over and over again. Reflection is used extensively in ASP.NET MVC. For example when the controller and action names are parsed from the route, Reflection is used to find the corresponding method to invoke. Except that once found, the result is cached so that the next time someone requests same controller and action name, the method to be invoked is fetched from the cache.
So if you are using a third party framework check the documentation/source code whether it uses reflection and whether it caches the results of those calls.
And if you have to use it in your code, same rule applies => cache it.
For stress testing, this SO post gives quite a few possibilities: Stress Testing ASP.Net application.
I have thought about this question myself, and come to the following conclusions:
Most people don't spend their days resubmitting pages over and over again. The time the user spends reading and consuming pages which at worst contain a few Ajax calls is minimal when taken into context with the time spent visiting an actual website. Even if you have a million concurrant users of your application, you will generally not have to deal with a million requests at any given time.
The web is naturally based on string comparisons... there are no types in an HTTP response, so any web application is forced to deal with these kinds of tasks as a fact of everyday life. The fewer string comparisons and dynamic objects the better, but they are at their core, unavoidable.
Although things like mapping by string comparison or dynamic type checking are slow, a site built with a non-compiled, weakly-typed language like PHP will contain far more of these actions. Despite the number of possible performance hits in MVC compared to a C# console application, it is still a superior solution to many others in the web domain.
The use of any framework will have a performance cost associated with it. An application built in C# with the .NET framework will for all intents and purposes not perform as well as an application written in C++. However, the benefits are better reliability, faster coding time and easier testing among others. Given how the speed of computers has exploded over the past decade or two, we have come to accept a few extra milliseconds here and there in exchange for these benefits (which are huge).
Given these points, in developing ASP.NET MVC applications I don't avoid things such as reflection like the plague, because it is clear that they can have quite a positive impact on how your application functions. They are tools, and when properly employed have great benefits for many applications.
As for performance, I like to build the best solution I can and then go back and run stress tests on it. Maybe the reflection I implemented in class X isn't a performance problem after all? In short, my first task is to build a great architecture, and my second is to optimise it to squeeze every last drop of performance from it.
This is verbose, I apologise if it’s not in accordance with local custom.
I’m writing a web replacement for a Windows application used to move firefighters around between fire stations to fill skill requirements, enter sick leave, withdraw firetrucks from service, and so on. Rails was the desired back-end, but I quickly realised I needed a client-side framework and chose Backbone.js.
Only one user will be on it at a time, so I don’t have to consider keeping clients in sync.
I’ve implemented most of the application and it’s working well. I’ve been avoiding facing a significant shortcoming, though: server-side validations. I have various client-side procedures ensuring that invalid updates can’t be made through the interface; for instance, the user can’t move someone who isn’t working today to another station. But nothing is stopping a malicious user from creating a record outside of the UI and saving it to the server, hence the need for server-side validation.
The client loads receives all of today’s relevant records and processes them. When a new record is created, it’s sent to the server, and processed on the client if it saved successfully.
The process of determining who is working today is complex: someone could be scheduled to work, but have gone on holidays, but then been called in, but then been sent home sick. Untangling all this on the server (on each load?!) in Ruby/Rails seems an unfortunate duplication of business logic. It would also have significant overhead in a specific case involving calculating who is to be temporarily promoted to a higher rank based on station shortages and union rules, it could mean reloading and processing almost all today’s data, over and over as each promotion is performed.
So, I thought, I have all this Backbone infrastructure that’s building an object model and constraining what models can be created, why not also use it on the server side?
Here is my uncertainty:
Should I abandon Rails and just use Node.js or some other way of running Backbone on the server?
Or can I run Node.js alongside Rails? When a user opens the application, I could feed the same data to the browser and Node, and Rails would check with the server-side Backbone to make sure the proposed new object was valid before saving it and returning it to the browser.
One factory is how deeply Rails is entrenched in this application. There isn’t that much server-side Ruby for creation/deletion of changes, but I made a sort of adaptation layer for loading the data to compensate for the legacy database model. Rails is mostly just serving JSON, and CSS, Javascript, and template assets. I do have a lot of Cucumber features, but maybe only the data-creation ones would need to be updated?
Whew! So, I’m looking for reassurance: is it reasonable, like suggested in this answer, to be running both Rails and Node on the server, with some kind of inter-process communication? Or has Rails’s usefulness shrunk so much (it is pretty much a single-page application like mentioned in that answer) that I should just get rid of it entirely and suffer some rewriting to a Node environment?
Thanks for reading.
It doesn't sound like you're worried a lot about concurrency as much as being able to do push data, which both platforms are completely capable of performing. If you have a big investment in Ruby code now, and no one is complaining about its use then what might be the concern? If you just want to use Node for push and singularity of using javascript through the stack then it might be worth it to move code over to it. From your comments, I really feel it is more about what is interesting to you, but you'll have to support the language(s) of choice. If you're the only one on the team then its pretty easy to just slip into a refactor to node just because it is interesting. Just goes back to who's complaining more, you or the customer. So to summarize: Node lets you move toward a single language in your code base, but you have to worry about what pitfalls javascript on the server has currently. Ruby on Rails is nice because you have all the ability to quickly generate features and proto them out.
I know there is a lot of talk about BPM these days and I am conscious that some may see it to be a craze rather than a fundamentally important piece of software.
As someone from what most would call 'The Business', I have been doing my best to learn about BPM to ensure we continue to make decisions that not only make sense to the business, but IT as well.
I have noticed while reading that mention is made to application workflow when sometimes discussing BPM. I hadn't given this much thought until recently.
Therefore, what is the difference? When would you use one and not the other?
BPM is about the process and improving it, which takes into account users and potentially more than one application,e.g. an ERP system may have more than one application to it, though there may be other uses of the term. Note that the process could be viewed without what applications or technologies are used.
Application workflow is how an application is used to go from a to b. Here it is a specific set of code that is used and what happens over the course of an application getting from a to b. In this case, the application is front and center rather than the process.
Does that provide an answer? Another way to think of it is that multiple application workflows can make up a system which is used in a process that can have BPM applied to it.
Late to the game, but workflow is to database as BPMS is to DBMS. (Convenient how the letters line up, huh?)
IOW, BPM(S) is traditionally meant to refer to a particular framework/application that allows you to manage business processes: defining them, storing them, versioning them, measuring them, etc. This is similar to how a DBMS manages databases.
Now, a workflow is a definition, much like a database is a definition. In the former case, it is a definition of operations/work (Fufill Order), steps thereof (Send Invoice) and rules/constraints on the work (If no stock, send notice). In the latter, similar case, it is a definition of data structure (CREATE TABLE) and constraints (InvoiceTotal must be > $0.00).
I think this is a potentially confusing subject, particular as some development environments use a type of process flow model to generate user facing applications (I'm thinking about Outsystems here, for example).
But, for me, the distinction is crystal clear. Application workflow, as people talk about it, refers to a user's path through an application, i.e. the pages they complete/visit, the data they enter, etc. on their way to completing a transaction of some sort. Application orkflow is a poor term for this though, I think application flow would be more meaningful.
BPM on other hand, is about modelling and executing a workflow process. By workflow, in this context, I mean a series of discrete steps (or tasks) that have to be completed (either programmatically or via human interaction) in a certain order to complete a process. These tasks can be implemented as individual application modules (each with their own "application workflow", see above). The job of the workflow engine is to make sure that these separate steps are assigned to the right people (of groups of people) in the right sequence, and that overall the process completes in an orderly way.
I don't think there's a clear answer to this at all. These are words, as opposed to theoretical concepts. If you add the word "checklist" into the mix - that just turns out to be a linear version of a process (but you can have conditionals in checklists - making them a workflow).
I am not sure how to help in reframing this question, but it's almost as if no answer can ever be possible. My own thoughts are at https://tallyfy.com/improving-efficiency-workflow-vs-business-process-management/