I am using iTextPDF version 5.4.2 and i am having thread contention issues when there is heavy load. I am using IBM JDK 6
This is the problem, when there are multiple independent threads trying to generate different pdf files i see a contention on SecureRandom.nextBytes , since this is synchronized all the threads are waiting to lock this object. Below is the Thread dump
*"WebContainer : 0" daemon prio=3 tid=0x007ffc00 nid=0x91 waiting for monitor entry [0xc823d000]
java.lang.Thread.State: BLOCKED (on object monitor)
at java.security.SecureRandom.nextBytes(SecureRandom.java:433)
- waiting to lock <0xdf4b7430> (a java.security.SecureRandom)
at java.util.UUID.randomUUID(UUID.java:162)
at com.itextpdf.text.pdf.PdfPCell.<init>(PdfPCell.java:123)
at com.itextpdf.text.pdf.PdfPRow.<init>(PdfPRow.java:136)
at com.itextpdf.text.pdf.PdfPTable.<init>(PdfPTable.java:260)
at com.itextpdf.text.pdf.PdfPCell.<init>(PdfPCell.java:251)
at com.itextpdf.text.pdf.PdfPRow.<init>(PdfPRow.java:136)
at com.itextpdf.text.pdf.PdfPTable.<init>(PdfPTable.java:260)
at com.itextpdf.text.pdf.PdfPCell.<init>(PdfPCell.java:251)
at com.itextpdf.text.pdf.PdfPRow.<init>(PdfPRow.java:136)
at com.itextpdf.text.pdf.PdfPTable.<init>(PdfPTable.java:260)
at com.itextpdf.text.pdf.PdfPCell.<init>(PdfPCell.java:251)
at com.itextpdf.text.pdf.PdfPRow.<init>(PdfPRow.java:136)
at com.itextpdf.text.pdf.PdfPTable.adjustCellsInRow(PdfPTable.java:1377)
at com.itextpdf.text.pdf.PdfPTable.getRows(PdfPTable.java:1364)
at com.itextpdf.text.pdf.ColumnText.goComposite(ColumnText.java:1702)
at com.itextpdf.text.pdf.ColumnText.go(ColumnText.java:881)
at com.itextpdf.text.pdf.ColumnText.go(ColumnText.java:876)
at com.itextpdf.text.pdf.ColumnText.go(ColumnText.java:865)
at com.itextpdf.text.pdf.PdfDocument.addPTable(PdfDocument.java:2566)
at com.itextpdf.text.pdf.PdfDocument.add(PdfDocument.java:723)
at com.itextpdf.text.Document.add(Document.java:278)*
Can you please help in avoiding this issue ?
Sid
I expect the contention is not for the lock, but for random bytes. It is hard for me to tell exactly what setup you're using, but I suspect that the SecureRandom class utilizes the (slow) secure random services provided by the OS and that you are exhausting the entropy pool. When it is exhausted, the call to read random data will block until more data is available. the other threads attempting to get random data will get stuck on the lock, making it look like the lock is the problem.
I think the simplest fix for this problem would be to upgrade to the latest version of iTextPdf. In this version, the ID has been changed from a UUID to an integer generated from a counter which doesn't block.
Another potential fix if you cannot upgrade would be to change the security provider used in your environment to provide a less secure but faster SecureRandom service.
Related
I think this is a popular antipattern that happens either standalone, for example activeJob local task with async, or coming from controllers, because then the strategy of the server must be taken into account.
My question is, what cautions should one take in the code when forking inside a thread (think inside of a ActiveJob task) and then even threading it?
The main worries I have seen online are:
Needs to lose and reopen the database connections after the fork. It seems that nowadays activeRecord takes care of it, doesn't it?
Access to the common Logger could be complicated. Somehow it seems to work.
Concurrent was expected to be problematic too but current versions are patched to detect that a fork has happened and threads are dead. Still it seems that one needs to make sure of doing, at the end of the forked process, a fine shutdown of any Rails::Concurrent pool that could have active or pending jobs. I think that it is enough
ActiveJob::Base.queue_adapter.shutdown
but perhaps it could miss some tasks that have not started or tasks under other Concurrent queue. In fact I think it already happens if one uses Concurrent::Future in a controller managed by the puma webserver. Generically I try to insert
Concurrent::global_io_executor.shutdown
Concurrent::global_io_executor.wait_for_termination
Extra problems I have found are resource-related: the postgres server is not ready to manage so many connections by default. Perhaps it could be sensible to reduce the size of the connection pool before the fork. And the inotify watcher gem also exhausts resource, when launched in development. Production is fine in both cases.
TL;DR; - I'm against doing it but many of us do it anyway and ignore the fact that it's unsafe... things break too rarely.
It is a simple fact that calling fork in a multi-threaded process may cause the new child to crash / deadlock / spin and may also cause other (harder to isolate) bugs.
This has nothing to do with Ruby, this is related to the locking mechanisms that safeguard critical sections and core process functionality such opening/closing files, allocating memory and any user created mutex / spinlock, etc'.
Why is it risky?
When calling fork the new process inherits all the state of the previous process but only the thread that called fork (all other threads do not exist in the new process).
This means that if any of the other threads was inside a critical section (i.e., allocating memory, opening a file, etc'), that critical section would remain locked for the lifetime of the new process, possibly causing deadlocks or unexpected errors.
Why do we ignore it?
In practical terms, the risk of something seriously breaking is often very low and most developers hadn't both encounter the issue and recognized its cause. Open files can be manually (if not automatically) closed, which leaves us mostly with the question of critical sections.
We can often reset our own critical sections which leaves mostly the system's critical sections...
The system's core critical sections that can be effected by fork are not that many. The main one is the memory allocator which can hardly ever break. Often the malloc implementation has multiple "arenas", each with its own critical section and it would be a long-shot to hit the system's underlying page allocation (i.e., mmap).
So is it safe?
No. Things still break, they just break rarely and when they do it isn't always obvious. Also, a parent process can sometime catch some of these errors and retry / recuperate and there are other ways to handle the risks.
Should I do it?
I wouldn't recommend to do it, but it depends. If you can handle an error, sure, go ahead. If not, that's a no.
Anyway, it's usually much better to use IPC to forward a message to a background process so that process perform any required fork / task.
The pattern can occur naturally when a Rails controller is combined with a webserver. The situation is different depending if the webserver is threaded, forked or evented, but the final conclusion is the same; that it is safe.
Fork + fork and thread + fork should not present problems of multiple access to the database or multiple execution of the same code, as only the current thread is active in the children.
Event + fork could be a source of troubles if the event machine is still active in the forked thread. Fortunately most designs generate a separate thread for the control of the event loop.
I have seen a lot of argument about epoll accepted new fd and spawn new thread for read and write on it's own thread doesn't scale well? But how it doesn't scale well? What if every connection has heavy processing like:-
doing database transaction
doing heavy algorithm work
waiting for other things to completed.
If my purpose definitely just want to do the thing inside the program(no more fancy routing to other connection to do stuff), and do not spawn new thread for read/write io. It might be hanging forever just because of one function waiting for something right? If this is the case how epoll scale well if do not spawn new thread?
epoll_wait(...);
// available to read now
recv(....);
// From here if i don't spawn thread, the program will be hanging. What should I do?
processing algorithm work.....// At least 3 secs to do the job.
continue;
AFAIU, epoll(7) does not spawn new threads by itself (see also pthreads(7)...). You need some other call (using pthread_create(3) or the underlying clone(2) system call used by pthread_create...) to create threads.
Read more about the C10K problem (which today should be called C100K) and some pthread tutorial. But it looks like your program could be compute-intensive, not IO-bound. So the bottleneck might be computer power (then you cannot get scalability with just multi-threading on a single computer node; you need distributed computing)
Threads are quite heavy resources. So you want to have some thread pool and have only a few dozens of active (i.e. runnable) threads. See this.
Be also aware of other multiplexing system calls (such as poll(2)), of non-blocking IO (fcntl(2) with O_NONBLOCK), of asynchronous IO (see aio(7)).
I recommend using some existing event-loop based library (look into libev, libevent, Glib, Poco, Qt, ... or for HTTP mostly: libonion on the server side, libcurl on the client side). Look also into 0mq.
The concepts related to callbacks, continuations, CPS could be useful and improve your thinking.
Languages like Go and its Goroutines could be helpful.
It might be hanging forever ....
That should not happen if you design your program carefully (of course having event loops using something like poll or epoll_wait - with a limited delay of less than a second and probably prefering non-blocking IO).
Probably, spending a few weeks learning more about Operating Systems concepts should be worthwhile. Also understanding most system calls (listed in syscalls(2)) after having read more about Linux programming (e.g. the old ALP book, or something newer) should be preferable. Perhaps you don't need something as sophisticated as epoll (because using just poll might be enough).
I'm trying to determine if my sqlite access to a database is thread-safe on iOS. I'm writing a non App Store app (or possibly a launch daemon), so Apple's approval isn't an issue. The database in question is the built-in sms.db, so for sure the OS is also accessing this database for reading and writing. I only want to be able to safely read it.
I've read this about reading from multiple processes with sqlite:
Multiple processes can have the same database open at the same time.
Multiple processes can be doing a SELECT at the same time. But only
one process can be making changes to the database at any moment in
time, however.
I understand that thread-safety can be compiled out of sqlite, and that sqlite3_threadsafe() can be used to test for this. Running this on iOS 5.0.1
int safe = sqlite3_threadsafe();
yields a result of 2. According to this, that means mutex locking is available. But, that doesn't necessarily mean it's in use.
I'm not entirely clear on whether thread-safety is dynamically enabled on a per connection, per database, or global basis.
I have also read this. It looks like sqlite3_config() can be used to enable safe multi-threading, but of course, I have no control, or visibility into how the OS itself may have used this call (do I?). If I were to make that call again in my app, would it make it safe to read the database, or would it only deconflict concurrent access for multiple threads in my app that used the same sqlite3 database handle?
Anyway, my question is ...
can I safely read this database that's also accessed by iOS, and if so, how?
I've never used SQLite, but I've spent a decent amount of time reading its docs because I plan on using it in the future (and the docs are interesting). I'd say that thread safety is independent of whether multiple processes can access the same database file at once. SQLite, regardless of what threading mode it is in, will lock the database file, so that multiple processes can read from the database at once but only one can write.
Thread safety only affects how your process can use SQLite. Without any thread safety, you can only call SQLite functions from one thread. But it should still, say, take an EXCLUSIVE lock before writing, so that other processes can't corrupt the database file. Thread safety just protects data in your process's memory from getting corrupted if you use multiple threads. So I don't think you ever need to worry about what another process (in this case iOS) is doing with an SQLite database.
Edit: To clarify, any time you write to the database, including a plain INSERT/UPDATE/DELETE, it will automatically take an EXCLUSIVE lock, write to the database, then release the lock. (And it actually takes a SHARED lock, then a RESERVED lock, then a PENDING lock, then an EXCLUSIVE lock before writing.) By default, if the database is already locked (say from another process), then SQLite will return SQLITE_BUSY without waiting. You can call sqlite3_busy_timeout() to tell it to wait longer.
I don't think any of this is news to you, but a few thoughts:
In terms of enabling multi-threading (either serialized or multi-threaded), the general counsel is that one can invoke sqlite3_config() (but you may have to do a shutdown first as suggested in the docs or as discussed on SO here) to enable the sort of multi-threading you want. That may be of diminished usefulness here, though, where you have no control over what sort of access iOS is requesting of sqlite and/or this database.
Thus, I would have thought that, from an academic perspective, it would not be safe to read this system database (because as you say, you have no assurance of what the OS is doing). But I wouldn't be surprised if iOS is opening the database using whatever the default mode is, so from a more pragmatic perspective, you might be fine.
Clearly, for most users concerned about multi-threaded access within a single app, the best counsel would be to bypass the sqlite3_config() silliness and just simply ensure coordinated access through your own GCD serial queue (i.e., have a dedicated queue through which all database interactions go through, gracefully eliminating the multi-thread issue altogether). Sadly, that's not an option here because you're trying to coordinate database interaction with iOS itself.
I'm working on a project and cannot find any documentation that verifies signal/thread behavior for the following case.
I need to ensure that signals generated in a thread will be delivered by the offending thread (namely, SIGSEGV). I've been told that POSIX doesn't ensure this behavior and that pthreads, for example, can generate signals in pthread 1 yet have the signal delivered in pthread 2. Therefore, I'm planning on using clone(2) to have finer control of signal/thread behavior, but still cannot find documentation in the man pages that ensures signals will be delivered by the offending thread.
Hardcore systems programmers: any documentation or insights would be very much appreciated!
The POSIX chapter on Signal Generation and Delivery states that:
At the time of generation, a
determination shall be made whether
the signal has been generated for the
process or for a specific thread
within the process. Signals which are
generated by some action attributable
to a particular thread, such as a
hardware fault, shall be generated for
the thread that caused the signal to
be generated. Signals that are
generated in association with a
process ID or process group ID or an
asynchronous event, such as terminal
activity, shall be generated for the
process.
A synchronous SIGSEGV caused by an incorrect memory access in a thread is clearly such a signal "...generated by some action attributable to a particular thread...", so they are guaranteed to be generated for the offending thread (which means handled or ignored by that thread, as appropriate).
I'm pretty sure this works, even if it isn't guaranteed. I base this observation on the fact that I used to work at a company where we routinely handled SIGSEGV in multithreaded programs and printed a stack trace to a log file (based on currently location). We did this on a variety of platforms--windows, linux, tru64 unix, aix, hpux, sunos ... and maybe one or two others that I can't recall. This (usually!) works because the location of SIGSEGV is still on the current stack (the signal handling mechanism just adds a few call frames over top of it).
It's only fair to mention that there's very little that you should do in a signal handler because there aren't many functions that are async signal safe. We ignored this and mostly got away with it, except if I recall on sunos (or aix) where we would get burned--the process would occasionally (and seemingly randomly) hard exit inside the signal handler.
I believe the recommended approach is to NOT handle SIGSEGV, and let the process exit with a core dump so you can analyze later in a debugger.
My application is basically a content based router which will route MMS events.
The logger I am using is the one that comes with the OTP framework in SASL mode "error_logger"
The issue is ::
I am using a client to generate MMS events with default values. This client (in Java) has the ability to send high load of events in multiple THREADS
I am sending 100 events in 10 threads (each thread sending 10 MMS events) to the my router written in Erlang/OTP.
The problem is, when such high load is received by my router , my Logger hangs i.e it stops updating my Log file. But the router is still able to route the events.
The conclusions that I have come up with is ::
Scheduling problem in Erlang when such high load of events is received (a separate process for each event).
A very unlikely dead-loack state.
Might be due to sending events in multiple threads rather than sending them sequentially. But I guess a router will be connected to multiple service provider boxes, so I thought of sending events in threads.
Can anybody help mw in demystifying the problem?
You already have a good answer, but I'll add to the discussion.
The error_logger is by default using cached write operations to disk. So one possibility is that you don't really notice this while under low load, but under high load your writes get stuck in the cache for a while.
On a side note: there should be no problem having multiple threads doing calls to Erlang.
Another way of testing this is to add your own logger to error_logger, and see what happens. Possibly printing to the shell or something else that is "fast".
Which version of Erlang are you using? Prior to R14A (R13B4 maybe?), there was a performance penalty when you invoked a selective receive when the message queue contained a lot of messages. This behaviour meant that in a process that receives lots of messages (error_logger being the canonical example), if it was barely keeping up with the load then a small spike in load could cause the cost of processing to spike up and stay there as the new processing cost was higher than the process could bear. This problem has been solved in R14A.
Secondly - why are you sending a high volume of events/calls/logs to a text logger? Formatting strings for output to a human readable log file is a lot more expensive than using a binary disk_log for instance. Reducing the cost of logging will help, but reducing the volume of logs will help even more. Maybe investigate exactly why you need to log these things and see if you can't record them another (less expensive) way.
Problems with error_logger are often symptoms of some other overload problem. Try looking at the message queue sizes for all your processes when this problem occurs and see if something else is backed up too. The following erlang shellcode might help:
[ { P, element(2, process_info(P, message_queue_len)) }
|| P <- erlang:processes(), is_process_alive(P) ]