the get_chat_history and egt_chat_members methods throw a permanent waiting error -Waiting for 20 (23,21,22,18) seconds before continuing. get_chat works fine. This error appeared a couple of days ago.
async with tg_cl:
while True:
try:
async for members in tg_cl.get_chat_members(target):
members_chat.append(members)
break
except FloodWait as Err:
print("Flood wait: {} seconds".format(Err.value))
sleep(Err.value)
continue
...............
async with tg_cl:
while True:
try:
if 'join' in chat:
info_chat = await tg_cl.join_chat(chat)
else:
info_chat = await tg_cl.get_chat(chat)
async for messages in tg_cl.get_chat_history(chat, limit=1, offset_id=-1):
count_messages = messages.id
break
except FloodWait as Err:
print("Flood wait: {} seconds".format(Err.value))
sleep(Err.value)
continue
Pyrogram already handles FloodWait errors on its own, you don't need to apply any logic yourself.
When setting up your Client instance, you can set the sleep_threshold. This is the amount of time that Pyrogram will handle a FloodWait error on its own, without any logic needed from you. You can set it to an arbitrarily high value to not get any errors anymore. Keep in mind that Pyrogram will silently handle these errors itself and only print something like "waiting x seconds" in your output.
list_of_members = []
for member in app.get_chat_members(chat_id):
list_of_members.append(member.id)
print(list_of_members)
[123, 456, 789, ...]
Please note that in Channels you can only retrieve 200 members at a time, in chats only 10 000 (ten thousand), this is a hard limit by the Server.
See Pyrogram's documentation on the available arguments, as well as some examples:
https://docs.pyrogram.org/api/methods/get_chat_members
Related
In a Rails 6.x app, I have a controller method which backgrounds queries that take longer than 2 minutes (to avoid a browser timeout), advises the user, stores the results, and sends a link that can retrieve them to generate a live page (with Highcharts charts). This works fine.
Now, I'm trying to implement the same logic with a method that backgrounds the creation of a report, via a Tempfile, and attaches the contents to an email, if the query runs too long. This code works just fine if the 2-minute timeout is NOT reached, but the Tempfile is empty at the commented line if the timeout IS reached.
I've tried wrapping the second part in another thread, and wrapping the internals of each thread with a mutex, but this is all getting above my head. I haven't done a lot of multithreading, and every time I do, I feel like I stumble around till I get it. This time, I can't even seem to stumble into it.
I don't know if the problem is with my thread(s), or a race condition with the Tempfile object. I've had trouble using Tempfiles before, because they seem to disappear quicker than I can close them. Is this one getting cleaned up before it can be sent? The file handle actually still exists on the file system at the commented point, even though it's empty, so I'm not clear on what's happening.
def report
queue = Queue.new
file = Tempfile.new('report')
thr = Thread.new do
query = %Q(blah blah blah)
#calibrations = ActiveRecord::Base.connection.exec_query query
query = %Q(blah blah blah)
#tunings = ActiveRecord::Base.connection.exec_query query
if queue.empty?
unless #tunings.empty?
CSV.open(file.path, 'wb') do |csv|
csv << ["headers...", #parameters].flatten
#calibrations.each do |c|
line = [c["h1"], c["h2"], c["h3"], c["h4"], c["h5"], c["h6"], c["h7"], c["h8"]]
t = #tunings.select { |t| t["code"] == c["code"] }.first
#parameters.each do |parameter|
line << t[parameter.downcase]
end
csv << line
end
end
send_data file.read, :type => 'text/csv; charset=iso-8859-1; header=present', :disposition => "attachment; filename=\"report.csv\""
end
else
# When "timed out", `file` is empty here
NotificationMailer.report_ready(current_user, file.read).deliver_later
end
end
give_up_at = Time.now + 120.seconds
while Time.now < give_up_at do
if !thr.alive?
break
end
sleep 1
end
if thr.alive?
queue << "Timeout"
render html: "Your report is taking longer than 2 minutes to generate. To avoid a browser timeout, it will finish in the background, and the report will be sent to you in email."
end
end
The reason the file is empty is because you are giving the query 120 seconds to complete. If after 120 seconds that has not happened you add "Timeout" to the queue. The query is still running inside the thread and has not reached the point where you check if the queue is empty or not. When the query does complete, since the queue is now not empty, you skip the part where you write the csv file and go to the Notification.report line. At that point the file is still empty because you never wrote anything into it.
In the end I think you need to rethink the overall logic of what you are trying to accomplish and there needs to be more communication between the threads and the top level.
Each thread needs to tell the top level if it has already sent the result, and the top level needs to let the thread know that its past time to directly send the result, and instead should email the result.
Here is some code that I think / hope will give some insight into how to approach this problem.
timeout_limit = 10
query_times = [5, 15, 1, 15]
timeout = []
sent_response = []
send_via_email = []
puts "time out is set to #{timeout_limit} seconds"
query_times.each_with_index do |query_time, query_id|
puts "starting query #{query_id} that will take #{query_time} seconds"
timeout[query_id] = false
sent_response[query_id] = false
send_via_email[query_id] = false
Thread.new do
## do query
sleep query_time
unless timeout[query_id]
puts "query #{query_id} has completed, displaying results now"
sent_response[query_id] = true
else
puts "query #{query_id} has completed, emailing result now"
send_via_email[query_id] = true
end
end
give_up_at = Time.now + timeout_limit
while Time.now < give_up_at
break if sent_response[query_id]
sleep 1
end
unless sent_response[query_id]
puts "query #{query_id} timed out, we will email the result of your query when it is completed"
timeout[query_id] = true
end
end
# simulate server environment
loop { }
=>
time out is set to 10 seconds
starting query 0 that will take 5 seconds
query 0 has completed, displaying results now
starting query 1 that will take 15 seconds
query 1 timed out, we will email the result of your query when it is completed
starting query 2 that will take 1 seconds
query 2 has completed, displaying results now
starting query 3 that will take 15 seconds
query 1 has completed, emailing result now
query 3 timed out, we will email the result of your query when it is completed
query 3 has completed, emailing result now
I'm making a small rails application that fetch data from some different languages at github-api.
The problem is, when i click the button that will fetch the informations, it takes a long time to redirect to the correct page. What i got from network is, the TTFB is actually 30s (!) and is getting a response with the status 302.
The controller function that is doing the logic:
Language.delete_all
search_urls = Introduction.all.map { |introduction| "https://api.github.com/search/repositories?q=#{introduction.name}&per_page=1" }
search_urls.each do |search_url|
json_file = JSON.parse(open(search_url).read)
pl = Language.new
pl.hash_response = json_file['items'].first
pl.name = pl.hash_response['language']
pl.save
end
main_languages = %w[ruby javascript python elixir java]
deletable_languages = Introduction.all.reject do |introduction|
main_languages.include?(introduction.name)
end
deletable_languages.each do |language|
language.delete
end
redirect_to languages_path
end
I believe the bottleneck is the http request in which you are doing it one by one. You could have filtered the languages that you want before generating the url and fetch them.
However, if the count of the urls after filtered is still large, say 20-50, assuming each request take 200ms, this would take at least 4s to 10s just for http request. Thats already too long for the user to wait for. In that case you should make it a background job.
If you insist to do this synchronously, you may consider fire those http requess by spawning multiple threads and join all the results after all threads are completed. You will achieve some concurrency here as the GIL will not block thread for IO wait. But this is very prone to error as you need to manage the threads on your own.
I have a sidekiq worker that waits for a change to happen to a record made by a remote client. Something like the following:
#myworker async process to wait for client to confirm status
perform(myRecordID)
sendClient(myRecordID)
didClientAcknowledge = false
while !didClientAcknowledge
didClientAcknowledge = myRecords.find(myRecordID).status == :ACK_OK
if didClientAcknowledge
break
end
# wait for client to perform an update on the record to confirm status
sleep 5.seconds
end
Rails.logger.info("client got the message")
end
my problem is that although I can see that the client has in fact performed the acknowledgement and updated the record with correct status update (ACK_OK), my sidekiq thread continues to see the old status for myRecord.
I'm sure my logic is flawed here but it seems like the sidekiq process does not "see" changes to the DB...but if I used my rails console I can see that the client has in fact updated the DB as expected...
Thanks!
Edit 1
ok so here's a thought, instead of the loop, I'll schedule another call to the worker within 5 seconds... so here's the updated code:
perform(myRecordID, retry_count)
retry_count -= 1
if retry_count < 1
return
end
sendClient(myRecordID)
didClientAcknowledge = false
if !didClientAcknowledge
didClientAcknowledge = myRecords.find(myRecordID).status == :ACK_OK
if didClientAcknowledge
Rails.logger.info("client got the message")
return
end
# wait for client to perform an update on the record to confirm status
myWorker.perform_in(5.seconds)
end
Rails.logger.info("client got the message")
end
This seems to work, but will test a bit more..one challenge is having a retry count which means I need to maintain some sort of variable between calls to the worker...
edit2 possibly this can be done by passing in the time to the first call and then checking if a timeout has been surpassed before invoking the next instance...assuming time does not stand still as well inside the async call...
edit3 Adding the retry_count argument allows us to control how many times this worker will be spawned...
I'm having trouble understanding the flow of the following code.
The code should process MERGE_SIZE lines (3 in this run), save the lines to a 'phase' file, and then process the next 3 lines and so on.
The call to savePhase has an await so I was expecting for the savePhase to complete before additional lines are processed.
As you can see in the output below every line is process and then the savePhase calls complete.
Future _sort() async {
var completer = Completer<void>();
var instance = 0;
var lineCount = MERGE_SIZE;
var phaseDirectory = Directory.systemTemp.createTempSync();
var list = <String>[];
var sentToPhase = false;
await File(filename)
.openRead()
.map(utf8.decode)
.transform(LineSplitter())
.forEach((l) async {
list.add(l);
print('$l linecount:$lineCount');
lineCount--;
if (lineCount == 0) {
lineCount = MERGE_SIZE;
instance++;
sentToPhase = true;
await savePhase(phaseDirectory, 1, instance, list, lineDelimiter);
list.clear();
print('savePhase completed');
}
});
which outputs
9 line linecount:3
8 line linecount:2
7 line linecount:1
6 line linecount:3
5 line linecount:2
4 line linecount:1
3 line linecount:3
2 line linecount:2
1 line linecount:1
savePhase completed
savePhase completed
savePhase completed
Is this something to do with the streams that openRead uses to deliver the read lines?
I thought I had await figured out, but apparently not :)
Not tested your program but I am fairly sure that your problem is that you expect the forEach() method are waiting for each Future to be completed before the next call which are not the case.
Try take a look at the following solution which are about more or less the same problem:
Sequential processing of a variable number of async functions in Dart
Flow of program
So what happens in your code is that the file your are reading seems to be small enough that the whole content can be read in one go in one of the buffers used when reading a file. This will explain why you see multiple line linecount lines before savePhase completed.
As previous mentioned, the forEach() method on Stream does not take into account that the method given as parameter does return a Future which should be awaited. You can see that in the implementation showed here:
https://api.dart.dev/stable/2.7.0/dart-async/Stream/forEach.html
So that means that the Future returned from calling forEach() does really just complete when all lines has been processed but does not wait for each Future generated for each line (remember, a async method does always return a Future regardless of it contains an await).
Since you also use shared variables between each spawned Future you will also get some funky behavior here since you e.g. share the same list but also clearing the list afterwards. So there are a potential here for errors here.
My application involves a battery-powered ESP8266 running NodeMCU for the purpose of updating sensor values periodically over MQTT.
To save battery life, I want to call dsleep() as soon as I'm done with my work. That work might involve more than 1 call to mqqt.Client.publish(). This brings us to the problem I'm facing.
I'm a Lua newbie, but as I understand, the right way to run some code after publish() has finished is to give it a PUBACK callback:
m = mqtt.Client(...)
m.publish("/my/topic", "some message", 1, 0, callback_func)
And in a simple case like above, that works great - even though the actual sending of the MQTT message is async with regard to the publish() call (see a good discussion of this here), callback_func() in the above example will only be called when publish() is done.
But when I have more than 1 call to publish() and want my callback to a) be called after they're all complete, and b) only be called once, I'm stuck.
The naive approach to this would be to put the callback (which is optional) only on the Nth publish() call:
m = mqtt.Client(...)
m.publish("/my/topic", "some message", 1, 0)
m.publish("/another/topic", "unrelated message", 1, 0, callback_func)
But this will not do what's expected. As documented:
NOTE: When calling publish() more than once, the last callback function defined will be called for ALL publish commands.
So in the above example, callback_func() ends up getting called twice (once for each successful publish().
I could combine the multiple publish() calls into a single call, but that feels like an ugly hack, and would have other adverse implications. If my two messages are conceptually distinct, this approach would push logic to separate them into the subscriber - yuck. And if they needed to go to different topics, this would be even worse. There must be a better way.
I thought perhaps mqqt.Client.close() would wait for my different publish() calls to finish, but it doesn't.
I'm out of ideas, and hoping someone with more Lua and/or NodeMCU+mqqt experience can give me a nudge in the right direction.
Here's my actual code, if it helps:
-- prior to this, we've gotten on the wifi network and acquired an IP
dofile("temp.lua") -- provides get_temp()
m = mqtt.Client("clientid", 120, "8266test", "password")
function mainloop(client)
print("connected - at top of loop")
m:publish("uptime",tmr.time(),1,0, function(client) print("sent uptime") end)
temp, humi = get_temp()
if (temp ~= nil) then
print(string.format("temp: %d", temp))
print(string.format("humi: %d", humi))
m:publish("temp",temp,1,0)
m:publish("humi",humi,1,0, function(client) -- note: this callback will be used for all publish() calls
rtctime.dsleep(SLEEP_USEC)
end)
end
end
m:on("connect", mainloop)
m:on("offline", function(client) is_connected = false print ("offline") end)
m:connect(MQQT_SVR, 1883, 0, mainloop,
function(client, reason) print("failed reason: "..reason) end)
Option 1:
publish all data at once, then go to sleep.
Option 2:
split your callback into two parts. the first part checks if you are done, the second part goes to sleep if you are done.
Of course you can solve this differently, count how many are left, count how many you have sent, send and remove items from a list until the list is empty,...
Of course there are more options but these are simple and sufficient.
Edit: example as requested
local totalItemCount = 5
function publishCallback()
itemsPublished = (itemsPublished or 0) + 1
print("item published")
if itemsPublished == totalItemCount then
print("I'm done, good night!")
end
end
for i = 1, totalItemCount do
publishCallback()
end
item published
item published
item published
item published
item published
I'm done, good night!