I've been using perfino for a week, first thing I noticed is that there was no transactions being recorded, which is fine because all of our tools don't use any framework or special annotations that perfino could potentially detect.
When I saw POJO events transactions, I thought that I could just tell it to include all packages that we make here with a wildcard and have those analyzed.
It seems like perfino doesn't accept wildcards when adding a pojo class name.
Is there anything that I'm missing that I could do to enable this functionality?
perfino deliberately does not have wildcards for classes since that would be an easy way to introduce unacceptable overhead. A transaction should be a high level use case, creating transactions for each method call is something that perfino is not designed for. You can see method-level data from all packages in the "Method sampling" view.
However you can instrument entire class hierarchies with the "Intercept subclasses" option on the POJO transaction configuration.
Related
I am currently implementing a web application in .net core(C#) using entity framework. While working on the project, I actually encountered quite a few challenges but I will start with the one which I think are most important. My questions are as follows:
Instead of frequent loading data from the database, I am having a set of static objects which is a mirror of the data in the database. However, it is tedious and error prone when I want to ensure any changes, i.e., adding/deleting/modifying of objects are being saved to the database at real time. Is there any good example or advice that I can refer to improve my approach to do this?
Another thing is that value of some objects' properties will be changed on the fly according to the value of some other objects' properties. Something like a spreadsheet where a cell's value will be changed automatically if the value in the cell that the formula is referring to changes. I do not have a solution to do this yet. Appreciate if anyone has any example that I can refer to. But this will add another layer of complexity to sync the changes of the objects in memory to database.
At the moment, I am unsure if there is any better approach. Appreciate if anyone can help. Thanks!
Basically, you're facing a problem that's called eventual consistency. Something changes and two or more systems need to be aware at the same time. The problem here is that both changes need to be applied in order to consider the operation successful. If either one fails, you need to know.
In your case, I would use the Azure Service Bus. You can create queues and put messages on a queue. An Azure Function would handle these queue messages. You would create two queues, one for database updates, and one for the in-memory update (I think changing this to a cache service may be something to think off). Now the advantage of these queues is that you can easily drop messages on these queues from anywhere. Because you mentioned the object is going to evolve, you may need to update these objects either in the database or in memory (cache).
Once you've done that, I'd create a topic, with two subscriptions. One forwarding messages to Queue 1, and the other to Queue 2. This will solve your primary problem. In case an object changes, just send it to the topic. Both changes (database and memory) will be executed automagically.
The only problem you have now, it that you mentioned you wanted to update the database in real-time. With this scenario, you're going to have to leave that.
Also, you need to make sure you have proper alerts in place for the queues so in case you did miss a message, or your functions didn't handle it well enough, you'll receive an alert to check & correct errors.
I'm totally agree with #nineedm's and answer, but there are also other solutions.
If you introduce cache, you will always face cache revalidation problem - you have to mark cache as invalid when data were changed. Sometimes it is easy, depending on nature of cached data and how often data are changed.
If you have just single application, MemoryCache can be enough with proper specified expiration options.
If there is a cluster - you have to look at Distributed Cache solutions, for example Redis. There is MS article about that Distributed caching in ASP.NET Core
I have an old SL4/ria app, which I am looking to replace with breeze. I have a question about memory use and caching. My app loads lists of Jobs (a typical user would have access to about 1,000 of these jobs). Additionally, there are quite a few lookup entity types. I want to make sure these are cached well client-side, but updated per session. When a user opens a job, it loads many more related entities (anywhere from 200 - 800 additional entities) which compose multiple matrix-style views for the jobs. A user can view the list of jobs, or navigate to view 1 job at a time.
I feel that I should be concerned with memory management, especially not knowing how browsers might deal with this. Originally I felt this should all be 1 EntityManager and I would detachEntities when user navigates away from a job, but I'm thinking this might benefit from multiple managers by intended lifetime. Or perhaps I should create a new dataservice & EntityManager each time the user navigates to a new hash '/#/' area, since comments on clear() seems to indicate that this would be faster? If I did this, I suppose I will be using pub/sub to notify other viewmodels of changes to entities? This seems complex and defeating some of the benefits of breeze as the context.
Any tips or thoughts about this would be greatly appreciated.
I think I understand the question. I think I would use a multi-manager approach:
Lookups Manager - holds once-per session reference (lookup) entities
JobsView Manager - "readonly" list of Jobs in support of the JobsView
JobEditor Manager - One per edit session.
The Lookups Manager maintains the canonical copy of reference entities. You can fill it once with a single call to server (see docs for how). This Lookups Manager will Breeze-export these reference entities to other managers which Breeze-import them as they are created. I am assuming that, while numerous and diverse, the total memory footprint of reference entities is pretty low ... low enough that you can afford to have more than one copy in multiple managers. There are more complicated solutions if that is NOT so. But let that be for now.
The JobsView Manager has the necessary reference entities for its display. If you only displayed a projection of the Jobs, it would not have Jobs in cache. You might have an array and key map instead. Let's keep it simple and assume that it has all the Jobs but not their related entities.
You never save changes with this manager! When editing or creating a Job, your app always fires up a "Job Editor" view with its own VM and JobEditor Manager. Again, you import the reference entities you need and, when editing an existing Job, you import the Job too.
I would take this approach anyway ... not just because of memory concerns. I like isolating my edit sessions into sandboxes. Eases cancellation. Gives me a clean way to store pending changes in browser storage so that the user won't lose his/her work if the app/browser goes down. Opens the door to editing several Jobs at the same time ... without worrying about mutually dependent entities-with-changes. It's a proven pattern that we've used forever in SL apps and should apply as well in JS apps.
When a Job edit succeeds, You have to tell the local client world about it. Lots of ways to do that. If the ONLY place that needs to know is the JobsView, you can hardcode a backchannel into the app. If you want to be more clever, you can have a central singleton service that raises events specifically about Job saving. The JobsView and each new JobEditor communicate with this service. And if you want to be hip, you use an in-process "Event Aggregator" (your pub/sub) for this purpose. I'd probably be using Durandal for this app anyway and it has an event aggregator in the box.
Honestly, it's not that complicated to use and importing/exporting entities among managers is a ... ahem ... breeze. Well worth it compared to refreshing the Jobs List every time you return to it (although you'll want a "refresh button" too because OTHER users could be adding/changing those Jobs). You retain plenty of Breeze benefits: querying, validation, change-tracking, batch saves, entity navigation (those reference lists work "for free" in Breeze).
As a refinement, I don't know that I would automatically destroy the JobEditor view/viewmodel/manager when I returned to the JobsView. In my experience, people often return to the same Job that they just left. I might hold on to a view so you could go back and forth quickly. But now I'm getting tricky.
Imagine i want to parse a binary blob of data. If all comes okay, then all the logs are INFO, and user by default does not even see them. If there is an error, then user is presented with error and can view the log to see exact reason (i don't like programs that just say "file is invaid. for some reason. you do not want to know it" )
Probably most log libraries are aimed at quickly loading, classifying and keeping many many log lines per second. which by itself is questionable, as there is no comfort lazy evaluation and closures in Delphi. Envy Scala :-)
However that need every line be pre-сlassified.
Imagine this hypothetical flow:
Got object FOO [ok]
1.1. found property BAR [ok]
1.1.1. parsed data for BAR [ok]
1.2 found property BAZ [ok]
1.2.1 successfully parsed data for BAR [ok]
1.2.2 matching data: checked dependancy between BAR and BAZ [fail]
...
So, what can be desired features?
1) Nested logging (indenting, subordination) is desired then.
Something like highlighted in TraceTool - see TraceNode.Send Method at http://www.codeproject.com/KB/trace/tracetool.aspx#premain0
2) The 1, 1.1, 1.1.1, 1.2, 1.2.1 lines are sent as they happen in a info sink (TMemo, OutputDebugString, EventLog and so one), so user can see and report at least which steps are complete before error.
3) 1, 1.2, 1.2.2 are retroactively marked as error (or warning, or whatever) inheriting from most specific line. Obviously, warning superseeds info, error superseeds warning and info, etc/
4) 1 + 1.2 + 1.2.2 can be easily combined like with LogMessage('1.2.2').FullText to be shown to user or converted to Exception, to carry the full story to human.
4.1) Optionally, with relevant setup, it would not only be converted to Exception, but the latter even would be auto-raised. This probably would require some kind of context with supplied exception class or supplied exception constructing callback.
5) Multisink: info can be just appended into collapsible panel with TMemo on main form or currently active form. The error state could open such panel additionally or prompt user to do it. At the same time some file or network server could for example receive warning and error grade messages and not receive info grade ones.
6) extra associated data might be nice too. Say, if to render it with TreeView rather than TMemo, then it could have "1.1.1. parsed data for BAR [ok]" item, with mouse tooltip like "Foo's dimensions are told to be 2x4x3.2 metres"
Being free library is nice, especially free with sources. Sometimes track and fix the bug relying solely on DCUs is much harder.
Non-requiring extra executable. it could offer extra more advanced viewer, but should not be required for just any functionality.
Not being stalled/abandoned.
ability to work and show at least something before GUI is initialized would be nice too. Class constructors are cool, yet are executed as part of unit visualization, when VCL is not booted yet. If any assertion/exception is thrown from there, user would only see Runtime error 217, with all the detail lost. At least OutputDebugStreen can be used, if nothing more...
Stack tracing is not required, if needed i can do it and add with Jedi CodeLib. But it is rarely needed.
External configuration is not required. It might be good for big application to reconfigure on the fly, but to me simplicity is much more important and configuration in code, by calling constructors or such, is what really matters. Extra XML file, like for Log4J, would only make things more fragile and complex.
I glanced few mentioned here libraries.
TraceTool has a great presentation, link is above. Yet it has no info grade, only 3 predefined grades (Debug/Error/Warning) and nothing more, but maybe Debug would suit for Info replacement... Seems like black box, only saving data into its own file, and using external tool to view it, not giving stream of events back to me. But their messages nesting and call chaining seems cool. Cools is also attaching objects/collection to messages.
Log4D and Log4Delphi seems to be in a stasis, with last releases of 2007 and 2009, last targeted version Delphi 7. Lack documentation (probably okay for log4j guy, but not for me :- ) Log4Delphi even had test folder - but those test do not compile in Delphi XE2-Upd1. Pity: In another thread here Log4delphi been hailed for how simple is to create custom log appender (sink)...
BTW, the very fact that the only LOG4J was forked into two independent Delphi ports leaves the question of which is better and that both lack something, if they had to remain in split.
mORMot part is hardly separated from the rest library. Demo application required UAC escalation for use its embedded SQLite3 engine and is frozen (no window opened, yet the process never exits normally) if refused Admin grants. Another demo just started infinite stream of AV exceptions, trying to unwind the stack. So is probably not ready yet for last Delphi. Though its list of message grades is excessive, maybe even a bit too many.
Thank you.
mORMot is stable, even with latest XE2 version of Delphi.
What you tried starting were the regression tests. Among its 6,000,000 tests, it includes the HTTP/1.1 Client-Server part of the ORM. Without the Admin rights, the http.sys Server is not able to register the URI, so you got errors. Which makes perfectly sense. It's a Vista/Seven restriction, not a mORMot restriction.
The logging part can be used completely separated from the ORM part. Logging is implemented in SynCommons.pas (and SynLZ.pas for the fast compression algorithm used for archival and .map embedding). I use the TSynLog class without any problem to log existing applications (even Delphi 5 and Delphi 6 applications), existing for years. The SQLite3 / ORM classes are implemented in other units.
It supports the nesting of events, with auto-leave feature, just as you expect. That is you can write:
procedure TMyClass.MyMethod(const Params: integer);
begin
TSynLog.Enter;
// .... my method code
end;
And adding this TSynLog.Enter will be logged with indentation corresponding to the recursive level. IMHO this may meet your requirements. It will declare an ISynLog interface on the stack, which will be freed by Delphi at the "end;" code line, so it will implement an Auto-Leave feature. And the exact unit name, method name and source code line number will be written into the log (as MyUnit.TMyClass.MyMethod (123)), if you generated a .map file at compilation (which may be compressed and appended to the .exe so that your customers logs will contain the source line numbers). You have methods at the ISynLog interface level to add some custom logging, including parameters and custom state (you can log objects properties as JSON if you need to, or write your custom logging data).
The exact timing of each methods are tracked, so you are able to profile your application from the data supplied by your customer.
If you think the logs are too much verbose, you have several levels of logging, to be customized on the client side. See the blog articles and the corresponding part of the framework documentation (in the SynCommons part). You've for instance "Fail" events and some custom kind of events. And it is totally VCL-independent, so you can use it without GUI or before any GUI is started.
You have at hand a log viewer, which allow client-side profiling and nested Enter/Leave view (if you click on the "Leave" line, you'll go back to the corresponding "Enter", e.g.):
If this log viewer is not enough, you have its source code to make it fulfill your requirements, and all the needed classes to parse and process the .log file on your own, if you wish. Logs are textual by default, but can be compressed into binary on request, to save disk space (the log viewer is able to read those compressed binary files). Stack tracing and exception interception are both implemented, and can be activated on request.
You could easily add a numeration like "1.2.1" to the logs, if you wish to. You've got the whole source code of the logging unit. Feel free to ask any question in our forum.
Log4D supports nested diagnostic contexts in the TLogNDC class, they can be used to group together all steps which are related to one compound action (instead of a 'session' based grouping of log events). Multi-Sinks are called Appenders in log4d and log4delphi so you could write a TLogMemoAppender with around twentyfive lines of code, and use it at the same time as a ODSAppender, a RollingFileAppender, or a SocketAppender, configurable at run time (no external config file required).
I'm using rhino ETL for the first time in a project, and I'm very
impressed by its capabilities. I use a join-operation to match two
datasources.
Sometimes there might be missing data, so I override LeftOrphanRow to
"log" the error. So I though I would throw an exception and then at
the end of the process collect all occured exceptions using
GetAllErrors().
But as it seems the process is being aborted with the first exception.
Is that intentionally? What would be the best way to deal with
OrphanRows (especially when I would like to have a summary of all orphan rows for all operations at the end of the process)?
Seems to me that the problem is that you're trying to use exceptions to report a non-exceptional event. That's not really what exceptions are for, and certainly when you're expecting the exception to pass through a third-party library, you shouldn't rely on that library to behave in any specific way with respect to that exception.
Can you just keep a list of orphan rows somewhere, e.g. globally, and add to it whenever you encounter one in any of your join operations? Then after your EtlProcess is finished, just print the list out. You might also consider using log4net to accomplish this. Or even simply raising an event, that you subscribe to elsewhere and do whatever seems appropriate.
I have a server application made in Erlang. In it I have an mnesia table
that store some information on photos. In the spirit of "everything is a
process" I decided to wrap that table in a gen_server module, so that the
gen_server module is the only one that directly accesses the table. Querying
and adding information to that table is done by sending messages to that process
(which has a registered name). The idea is that there will be several client
processes querying information from that table.
This works just fine, but that gen_server module has no state. Everything it
requires is stored in the mnesia table. So, I wonder if a gen_server is perhaps
not the best model for encapsulating that table?
Should I simply not make it a process, and instead only encapsulate the table
through the functions in that module? In case of a bug in that module, that
would cause the calling process to crash, which I think might be better, because
it would only affect a single client, as opposed to now, when it would cause the
gen_server process to crash, leaving everyone without access to the table (until
the supervisor restarts it).
Any input is greatly appreciated.
I guess according to Occam's razor there is no need for this gen_server to exist, especially since there is absolutely no state stored in it. Such process could be needed in situations when you need access to the table (or any other resource) to be strictly sequential (for example you might want to avoid any aborted transactions at cost of a bottleneck).
Encapsulating access to the table in a module is a good solution. It creates no additional complexity, while providing proper level of abstraction and encapsulation.
I'm not sure I understand why you've decided to encapsulate a table with a process. Mnesia is designed to mediate multiple concurrent accesses to tables, both locally and distributed across a cluster.
Creating an API module that performs all the particular table access operations and updates is a good idea as the API functions will convey your intent better in the code that calls them. It will be more readable than putting the mnesia operations directly into the calling code.
An API module also gives you the option to switch from mnesia to some other storage system later if you need to. Using mnesia transactions inside your API module protects you from some programming errors as mnesia will roll-back operations that crash. The API module will always be available to callers and allows any number of callers to perform operations concurrently, whereas a gen_server based API has a point of failure, the process, that can render the API unavailable.
The only thing a gen_server based API gives you over a pure-functional API is to serialize access to the table - which is an unusual requirement and unless you specifically need it, it will be a performance killer.
It may be a good idea to handle a mnesia table using single gen_server process when you want to use dirty access and avoid transactions. This approach might be faster than txs, but as usually you need to benchmark it.