I have had some issues with the visualization of the progress on my tasks.
I used to use the status page on the 8787 port, but visualizations don't always seem as smooth (refresh rates) as what I can see in blogposts.
I was quite happy when using the dask.diagnostics.progress since it ran smoothly and had a timer.
But this time when trying to persist a dataframe, my progress got blocked (stayed at 2.4 seconds and never advanced). Updating it doesn't fix the problem and on the 8787 I am not getting much info either. Any ideas why this happens?
The progress bar will stop if the computation has failed. You might want to check that users1 is producing a good result. A simple way to do this would be to trigger a cheap computation on the full dataframe, like len
>>> len(users1)
Exception ...
Other reasons for non-responsive progress bars include a poor internet connection, or a very saturated scheduler (if there are millions of tasks), but neither of these seem to be the case in your situation.
Related
Hoping someone can point me in the right direction here. My electron app needs to perform an API call every 10 minutes or so. I currently do this using a setInterval loop in the renderer process that fires every 10 minutes.
It generally works fine, for a few hours before it seems to just stop firing. I have several processes that clear and restart the setInterval to try and counteract the problem, but nothing seems to work.
The app opens new browser windows and displays content, which mean the main window may not be in focus all the time, which I suspect may have something to do with it.
I have tried adding
powerSaveBlocker.start("prevent-display-sleep");
powerSaveBlocker.start("prevent-app-suspension");
to my main electron js script, but it doesn't seem to have an effect. The problem is mostly being displayed on Windows machines. I'm not entirely sure if it happens on Mac or Linux.
So my question, is there any reason why this would be happening that intervals just die after a point? The powerSaveBlocker calls made sense to me, but they don't really seem to do anything.
Or is there a better way to have a background process running at intervals that can perform these API calls? I had looked at node-schedule but I'm not sure if it will fix this issue.
Answering my own question here with credit to #snwflk who pointed me in the right direction in a comment on the original post.
Whilst I have not been able to clarify with absolute certainty this solves the problem, I have also not seen the problem since. (It's not always 100% reproducible, and its difficult to test as it requires a machine, left alone for several hours, which may or may not display the problem).
I have however rolled the fix out to a few customers and their machines seem to still be online days later, which is a good sign.
So, the solution was to disable backgroundThrottling on the main BrowserWindow object (interval was running in the renderer processes)
Docs: https://electronjs.org/docs/api/browser-window#new-browserwindowoptions
An example
mainWindow = new BrowserWindow({
webPreferences: {
backgroundThrottling: false,
},
});
FYI be warned there have been a few bugs that prevented this setting from working, ie https://github.com/electron/electron/issues/20974 so be sure to update your electron version.
As far as I know intervals should keep running forever (the MDN page also doesn't mention anything).
If I understood correctly, your Electron app only does that API call? So it would be hard to tell the difference between the interval not triggering, and some other problem causing a freeze?
My guess would be you have an uncaught exception being thrown, or some other similar error or event. https://stackoverflow.com/a/43911292/841830 gives some suggestions of things to catch. https://github.com/sindresorhus/electron-unhandled says it can be used in both main and renderer processes to catch and report all problems.
Or is there a better way to have a background process running at intervals that can perform these API calls?
If you were on Linux, using cron to run a node script (or even just a wget or curl command) would be better than using an Electron app just for this. Task scheduler seems to be the windows equivalent of cron.
I'd like to infrequently open a Twitter streaming connection with TweetStream and listen for new statuses for about an hour.
How should I go about opening the connection, keeping it open for an hour, and then closing it gracefully?
Normally for background processes I would use Resque or Sidekiq, but from my understanding those are for completing tasks as quickly as possible, not chilling and keeping a connection open.
I thought about using a global variable like $twitter_client but that wouldn't horizontally scale.
I also thought about building a second application that runs on one box to handle this functionality, but that seems excessive if it can be integrated into the main app somehow.
To clarify, I have no trouble starting a process, capturing tweets, and using them appropriately. I'm just not sure what I should be starting. A new app? A daemon of some sort?
I've never encountered a problem like this, and am completely lost. Any direction would be much appreciated!
Although not a direct fix, this is what I would look at:
Time
You're working with time, so I'd look at what time-centric processes could be used to induce the connection for an hour
Specifically, I'd look at running a some sort of job on the server, which you could fire at specific times (programmatically if required), to open & close the connection. I only have experience with resque, but as you say, it's probably not up to the job. If I find any better solutions, I'll certainly update the answer
Storage
Once you've connected to TweetStream, you'll want to look at how you can capture the tweets for that time period. It seems a waste to create a data table just for the job, so I'd be inclined to use something like Redis to store the tweets that you need
This can then be used to output the tweets you need, allowing you to simulate storing / capturing them, but then delete them after the hour-window has passed
Delivery
I don't know what context you're using this feature in, so I'll just give you as generic process idea as possible
To display the tweets, I'd personally create some sort of record in the DB to show the time you're pinging TweetStream that day (if it changes; if it's constant, just set a constant in an initializer), and then just include some logic to try and get the tweets from Redis. If you're able to collect them, show them as you wish, else don't print anything
Hope that gives you a broader spectrum of ideas?
I have a PIC32MX340F512 board developed by another company for us, The board has a DS1338 RTCC and 24LC32A eeprom, and display unit on an I2C bus, on this bus i included a TSL2561 I2C light sensor, i wrote code in c to poll the light sensor continously , when the light sensor reaches a certain level i save the time and date and light sensor value on SD card. This all works fine but if i leave the system without exposure to light inside tunnel where incident light on one end of the tunnel is ought to be monitored the system becomes unresponsive no matter how much amount of light you apply and then if i switch power off and back on again everything starts to work normal. i am a one man development team and have been trying to find out the problem for months, i activated the watchdog timer to prevent the system from hanging but the problem still persisted. i then decided to find out if the problem is with the sensor by including a push button to activate light measurement but still when 4-5 hours elapse the PIC cant even detect a change in the the input pin. Under the impression that a hardware reset overrides anything going on i included a reset button and it also works ok for the first few hours after that the PIC doesn't seem to be responding to anything including a reset. I was getting convinced that there is nothing wrong with the firmware but also with all this happening the display unit (pic16f1933 and lcd) on the I2C shares power with the main unit and doesn't seem to be affected as it alternates between different messages constantly Does anybody have an idea what could be wrong (hardware/firmware or my sensor). I am using a 24v DC power supply purchased seperately. The PIC seems to go into a deep sleep although i dd not implement any kind of SLEEP mode in my code. Nb We use the same board for many other projects and i haven't come across such a problem . Thanks in advance.
I think you need to (if you haven't already) explore the wonderful world of in-circuit-debugging (such as with the ICD3 or PICkit 2/3). It allows you to run the processor in a special mode that lets you pause execution, see exactly which line of code is being executed, inspect variable values, and step through the code to see which parts are running and not running, or see exactly where execution takes a wrong turn. If the problem takes hours to reproduce, that's okay. You can just leave it overnight running in debug mode and hopefully it will be locked-up or 'sleeping' in the morning. At this point, you will be able to pause the processor and poke around to see if you got caught in some kind of infinite loop or something. This is often the only way to dig inside a running piece of code to see why things aren't working as you expect. But as you say, those bugs that take hours or days to manifest are the trickiest. Good luck!
It sounds like you can break up your design into two main parts, sd card interfacing, reading the rtc and reading the light sensor. If it were me I would upload a version of the code that mimics reading the light sensor but only returns fake data and see if that cures the problem. Additionally do the same with the other two modules separately and see if any of the three versions of your project not show this problem. From there just keep narrowing it down until you find the block of code thats causing problems.
if Two or more versions of your debug code show the same problem then my guess is it has to do with one of the communication protocols. I had a problem with a Pic32 silicon version blocking when using the DMA in conjunction with the SPI peripherals. So I would suggest checking the errata for your chip.
If you still cant find the problem, my only suggestion would be to check for memory leaks or arrays that are growing into reserved memory.
Hope that helps, good luck!
I have an application that can run for quite a long time scanning a database.
During this process I keep my program responsive by using processmessage.
This processmessage is triggered when my progress bar is updated and inc'ed.
This works fine is most cases, but when the databases get larger it takes longer for the progress bar to jump up 1%, the program becomes unresponsive until that time.
Is there another way to keep my program alive besides processmessages?
Multi threading is the answer. A standard Delphi application is basically a single threaded application that can do one thing at a time. Hence the gui lockup, it can't remain responsive if it's doing something else.
If you want to have a responsive gui and do heavy lifting at the same time, you need to have the heavy lifting in a separate thread or threads. This way your main thread can make sure you have a responsive program and the worker threads do the heavy lifting.
This works nice for heavy database work but also for for instance the downloading of files or situations where an answer of for instance a remote server can take a long time.
But this answer will probably give you more questions then answers because to explain HOW to use multi threading would be too big of an explanation for this question.
One other thing though: have a long and hard look at your database code. How are you retrieving records from the database, are there good indexes on the database etc. etc. etc. You can get insane speed improvements by optimizing this code before you have to start thinking about multi threading.
I've found the following resource: http://thaddy.co.uk/threads/ which you can download with pictures at: http://cc.embarcadero.com/item/14809 to be very usefull threading tutorial.
If you want to make your GUI program appear responsive, you must service the message queue in a timely fashion. There is no alternative.
When it comes to running database queries, the way to do that without freezing your UI, is to move the query to a different thread.
I've seen is posible to know what applications are running with getVisibleApplications()from "ApplicationManager" class but, is it possible to know how long were these applications used?
Thanks
If you really needed this information it would be possible.
You could have a background process start a timer and every N minutes it could call getVisibleApplications() and keep track of what has been added or removed to the list.
The difficulty is finding the right balance between accurate data and battery life as a very small value of N (seconds/minutes) will prevent the device from sleeping and will cause severe battery life degradation. A very large value of N will have minimal effect on battery life but the data won't be very accurate.
You could also combine it with events... maybe you would monitor aggressively when the backlight was on and hourly when it's off.
Not really sure what you're trying to do so my apologies if this doesn't help at all.
Simple answer is, No you cannot the duration of how long the applications were used. Out of interest, why would you want to know that?