How to group messages, in order, based on the contents - f#

The following code is used to display message in a Telegram bot.
When the application prints text, it is put in a queue and the queue gets consolidated and is flushed at regular intervals. This is meant to minimize the number of API calls.
At the same time, there is a limit of 4kb per message sent, so the messages get grouped as long as the total does not exceed 4kb.
Additionally, messages can be declared as markdown or plain text. All messages are typically sent as markdown, but error messages that contain stack traces, etc don't work in markdown mode, Telegram just ignores them.
There messages always get marked with some 'alert' symbol. And they have to be sent on their own.
For example:
message 1
message 2
Alert message 3
message 4
message 5
message LARGE SIZE 6
message 7
would be sent as:
message 1 + message 2 (markdown)
Alert message 3 (plain)
message 4 + message 5 (markdown)
message LARGE SIZE 6 (markdown)
message 7 (markdown)
this is the code used:
let hasAlertCharacter (message: string) =
message.Contains(interruptIcon)
|| message.Contains(waitForUserIcon)
|| message.Contains(waitForEventIcon)
|| message.Contains(warningIcon)
|| message.Contains(errorIcon)
let sizeLimit = 4000
let rec takeMessages (message: string) =
let r, m = messageQueue.TryPeek()
if r && message.Length + m.Length + 1 < sizeLimit then
if hasAlertCharacter m then
if String.IsNullOrEmpty(message) then
messageQueue.TryDequeue() |> ignore
m
else
message
else
messageQueue.TryDequeue() |> ignore
takeMessages $"{message}\n{m}"
else
message
let messagesString = (takeMessages String.Empty).TrimStart('\n')
it works, but I think it's not very readable. Additionally, the code that sends the message is looking for the alert characters again:
let parseMode =
match hasAlertCharacter messagesString with
| true -> ParseMode.Default
| false -> ParseMode.Markdown
client.SendTextMessageAsync(chatId, messagesString, parseMode)
|> Async.AwaitTask
|> Async.RunSynchronously
|> ignore
so, overall, I am looking for a more elegant way to handle this.

Related

How can I split a list of strings by chunks when they have some light markdown attributes, in F#?

I have a tool that is using a Telegram chatbot to interact with its users.
Telegram limits the call rate, so I use a queue system that gets flushed at regular intervals.
So, the current code is very basic:
// flush the message queue
let flushMessageQueue() =
if not messageQueue.IsEmpty then <- messageQueue is a ConcurrentQueue
// get all the messages
let messages =
messageQueue
|> Seq.unfold(fun q ->
match q.TryDequeue () with
| true, m -> Some (m, q)
| _ -> None)
// put all the messages in a single string
let messagesString = String.Join("\n", messages)
// send the data
client.SendTextMessageAsync(chatId, messagesString, ParseMode.Markdown)
|> Async.AwaitTask
|> Async.RunSynchronously
|> ignore
this is called at regular interval, while the write is:
// broadcast message
let broadcastMessage message =
messageQueue.Enqueue(message)
printfn "%s" (message.Replace ("```", String.Empty))
But as messages became more complex, two problems came at once:
Part of the output is formatted text with simple markdown:
Some blocks of lines are wrapped between ``` sections
There are some ``` sections as well inside some lines
The text is UTF-8 and uses a bunch of symbols
Some example of text may be:
```
this is a group of lines
with one, or many many lines
```
and sometimes there are things ```like this``` as well
And... I found out that Telegram limits message size to 4kb as well
So, I thought of two things:
I can maintain a state with the open / close ``` and pull from a queue, wrap each line in triple back ticks based on the state and push into another queue that will be used to make the 4kb block.
I can keep taking messages from the re-formatted queue and aggregate them until I reach 4kb, or the end of the queue and loop around.
Is there an elegant way to do this in F#?
I remember seeing a snippet where a collection function was used to aggregate data until a certain size but it looked very inefficient as it was making a collection of line1, line1+line2, line1+line2+line3... and then picking the one with the right size.

Suave - Control when responses are 'cached' or recalculated

I want to understand how to control when responses are 'cached' versus when they are 'recalculated'.
As an example:
[<EntryPoint>]
let main [| port |] =
let config =
{ defaultConfig with
bindings = [ HttpBinding.mk HTTP IPAddress.Loopback (uint16 port) ]
listenTimeout = TimeSpan.FromMilliseconds 3000.
}
let appDemo:WebPart =
DateTime.Now.ToString()
|> sprintf "Server timestamp: %s"
|> Successful.OK
startWebServer config appDemo
If I run the above webserver and hit it several times then each time I get the same timestamp back. Which I guess makes sense; appDemo is just an expression which is calculated first time around and never again, right?
In this circumstance, I might want appDemo to be 'recalculated' for every request. How do I do that? I can't seem to find an example in the docs.
Try this - not sure how high it scores on "idiomatic Suave" scale though:
let appDemo:WebPart =
request (fun req ->
DateTime.Now.ToString()
|> sprintf "Server timestamp: %s"
|> Successful.OK)
You're right in that you're seeing the same value because it's captured at the time appDemo is evaluated. That's a property of how F# works however, and has nothing to do with Suave caching it.
Note that WebPart type is an alias for HttpContext -> Async<HttpContext option> function - so inherently it yields itself to being recalculated on each request rather than being calculated once.

Twilio queue overflow error: how large is the queue?

Twilio's Message resource has a "status" property that indicates whether a SMS message is "queued", "sending", "failed", etc. If a Message instance failed to deliver, one possible error message is "Queue overflow". In the Twilio documentation, the description for this error case is: "You tried to send too many messages too quickly and your message queue overflowed. Try sending your message again after waiting some time."
Is the queue referenced in error code 30001 an instance of this resource? https://www.twilio.com/docs/api/rest/queue
Or is the queue (in the case of a 30001 error code) something that Twilio maintains on their end? If Twilio does throttling behind the scenes (queueing SMS messages per sending phone number), what is the size of that queue? How much would we have to exceed the rate limit (per phone number) before the queue overflow referenced in error code 30001 occurs?
Emily, message queue is not related to the queue resource you linked to above and it is something maintained on our end.
Twilio can queue up to 4 hours of SMS. This means, we can send out 1 sms per second, if there are more than 14,400 messages in the queue, all the messages queued up after will fail with 30001 error queue overflow and will not be sent. This info is for Long Code numbers. The link above explains processing for other scenarios.
A few suggestions to avoid the error:
Keep messages to at most 160 characters if possible. But if not
possible, calculate how many SMS messages each message will be (if
you are not sure you can always send 1 test message and see how much
you are charged for that message).
Based on the assumption that your messages is 160 characters,
throttle the sending rate to 3600 messages per hour (1 messaage/sec *
60 sec/min * 60 min/hr).
Please let me know if you've got any other questions.
Each of the Twilio phone numbers(senders) has a separate queue in which 14400(4 hr x 60 min x 60 sec) message segments can be queued. 1 segment is sent in one second.
What is a message segment?
A message segment is not a complete message but a part of a message.
Normally SMS is sent in terms of message segments and all message
segments are combined on the user’s mobile to create the actual SMS.
Twillio message segment config:
1 character = 8 bits(1 byte)
GSM encoding = 7 bit per character
UCS-2 encoding = 16 bit per character
Data Header = 6 bytes per segment
Summary: Each character takes 8 bits, If GSM encoding is used, each
character will take 7 bits or if UCS-2 encoding is used, each char
will take 16 bits. In the case of multiple segments, 6 bytes per
segment will be used for data headers(responsible for combining all
segments of the same SMS on user mobile)
Character per Message Segment:
GSM encoding when single segment = (140 char bytes x 8 bits)/ 7 bits =
160 characters
UCS-2 encoding when single segment = (140 char bytes x 8 bits)/ 16
bits = 70 characters
GSM encoding when multiple segment = ((140 char bytes - 6 header
bytes) x 8 bits)/ 7 bits = 154 characters
UCS-2 encoding when multiple segment = ((140 char bytes - 6 header
bytes) x 8 bits)/ 16 bits = 67 characters
Based on what encoding is used(check via Twilio Admin) for your message, you can calculate how many SMS can be in the queue at a time.
References:
https://support.twilio.com/hc/en-us/articles/115002943027-Understanding-Twilio-Rate-Limits-and-Message-Queues
https://www.twilio.com/blog/2017/03/what-the-heck-is-a-segment.html

Why does order matter in this usage of Observable.merge?

I am trying to write a basic "game loop" using Observables in F#. Basically I conceptualize the fundamental input stream of events as two streams merged together: the key presses of the user (game uses just keyboard to begin with), and the regular ticks of the game (say, 60 times per second).
My problem seems to stem from the fact that one of the observed sequences, i.e. the ticks, is also the loop that calls DispatchEvents() on the Window allowing it to process its inputs and fire key pressed events, so one stream of events is actually driven by the other, if that makes sense. Here is the code:
open System;
open System.IO
open SFML.Window
open SFML.Graphics
open System.Reactive
open System.Reactive.Linq
open System.Diagnostics
type InputEvent =
| Tick of TimeSpan
| KeyPressed of Keyboard.Key
[<EntryPoint;STAThread>]
let main _ =
use window = new RenderWindow(VideoMode(640u, 480u), "GameWindow")
window.SetVerticalSyncEnabled(true)
let displayStream =
Observable.Create(
fun (observer:IObserver<TimeSpan>) ->
let sw = Stopwatch.StartNew()
while (window.IsOpen()) do
window.DispatchEvents() // this calls the KeyPressed event synchronously
window.Display() // this blocks until the next vertical sync
window.Clear()
observer.OnNext sw.Elapsed
sw.Restart()
observer.OnCompleted();
{ new IDisposable with member this.Dispose() = ()})
let onDisplay elapsedTime =
// draw game: code elided
let inputEvents = Observable.merge
(window.KeyPressed |> Observable.map (fun key -> KeyPressed(key.Code)))
(displayStream |> Observable.map (fun t -> Tick(t)))
use subscription =
inputEvents.Subscribe(fun inputEvent -> match inputEvent with
| Tick(t) -> onDisplay(t)
| KeyPressed(key) -> printfn "%A" key)
0
This works, however, if I change the order of parameters in Observable.merge:
let inputEvents = Observable.merge
(displayStream |> Observable.map (fun t -> Tick(t)))
(window.KeyPressed |> Observable.map (fun key -> KeyPressed(key.Code)))
Then the game renders (onDisplay is called), but I don't see KeyPressed events printed to the console. Why is that?
(If you're wondering what is SFML, here's the link).
In pseudo-code, what merge does is:
firstStream.Subscribe(...);
secondStream.Subscribe(...);
The subscribe function you pass to Observable.create is synchronous and never yields control back to the caller. This means that merge itself is blocked from trying to subscribe to any streams that come after displayStream. When you reorder the streams so that displayStream is first, you prevent it from ever subscribing to your KeyPressed stream. This is why you are seeing the behavior you see.
In some respects, your displayStream is behaving badly. Subscribe methods should not block.
So, either make sure displayStream is the last item in your list, or do some refactoring of your code. You could just use a Subject for displayStream. Then subscribe to everything and finally start the "display loop", where you execute the loop that is currently in your displayStream definition and each time through the loop, just call OnNext on the subject.

Erlang receive loop - variable unbound issue

So I need to write this function that receives a number from three different processes and saves them. I can't get it to work, I get the "variable unbound" error.
serverB2(Time, Client1, Client2, Client3) ->
receive
{From, TimeClient} ->
if
From == Client1 ->
TimeClient1 = TimeClient;
From == Client2 ->
TimeClient2 = TimeClient;
From == Client3 ->
TimeClient3 = TimeClient;
end,
serverB2(Time, Client1, Client2, Client3)
end,
List = [Time, TimeClient1, TimeClient2, TimeClient3],
io:format("~w \n", List).
You are getting variable unbound error because your code has three paths and in each path you will bound only one of variables TimeClient1, TimeClient2 and TimeClient3 so you have always two of them unbound. What's worse your code never stop. You will receive message, then evaluate if statement and then recursive call serverB2 code and again and again and again. There is not any code path to your statements after receive.
I would write this in this way
serverB2(Time, Client1, Client2, Client3) ->
List = [ Time
| [ receive {Client, TimeClient} -> TimeClient end
|| Client <- [Client1, Client2, Client3]]],
io:format("~w~n", [List]).
or in more conventional way
serverB2(Time, Client1, Client2, Client3) ->
List = [ Time | collect_times([Client1, Client2, Client3])],
io:format("~w~n", [List]).
collect_times([]) -> [];
collect_times([Client|Tail]) ->
TimeClient = receive {Client, Time} -> Time end,
[ TimeClient | collect_times(Tail) ].
Which is how list comprehension above would be translated by compiler.
The trick what I'm using here is that I receive messages in order what I want instead of order in which they are arrived to mailbox. I'm using selective receive there because variable Client is bound in receive pattern in both examples above. There is more complicated way how receive messages in order how they arrive which you should use only with very good reason (performance for example).

Resources