I am building a GenServer in Elixir, let's say it's a simple Queue like this
defmodule Queue do
use GenServer
def start_link(name) do
GenServer.start_link(__MODULE__, :ok, name: name)
end
def push(server, msg) do
GenServer.call(server, {:subscribe, channel, last_id})
end
def pop(server) do
GenServer.call(server, {:pop, channel, last_id})
end
# handlers here ...
end
For this queue, I wan to provide different storage backends, like
PostgreSQL
Redis
In-memory
File
So and so on. Here's the question, how can I do dependency injection with this GenServer? Ideally I want to create a Queue with different backend like this for database backend
{:ok, db_queue} = Queue.start_link(:DBQueue, db_process_pid)
and for redis may like this
{:ok, redis_queue} = Queue.start_link(:RedisQueue, redis_process_pid)
In this way, I can create the same queue server with different backend. What is the best practice for Elixir to do dependency injection for a GenServer?
If you only want to store a pid, you can use the GenServer state to store it. You can then access it from the handle_* callback functions. For example, here's how the Queue would be like:
defmodule Queue do
use GenServer
def start_link(name, pid) do
GenServer.start_link(__MODULE__, pid, name: name)
end
def init(pid), do: {:ok, pid}
...
end
You can now start it like you want to:
{:ok, db_queue} = Queue.start_link(:DBQueue, db_process_pid)
And then in your handle_* callbacks, access the state (which is just a pid in this case) from the last argument:
def handle_call({:subscribe, channel, last_id}, from, pid) do
# use the pid and the message to construct a reply
reply = ...
{:reply, reply, pid}
end
If you want to store more things, like say a module as well, just change pid everywhere to a map like %{pid: ..., module: ...}. Any Erlang term can be used as the state.
Related
I have seen many GenServer implementations, I am trying to create one with such specifications, But I am not sure its GenServer's use case.
I have a state such as
%{url: "abc.com/jpeg", name: "Camera1", id: :camera_one, frequency: 10}
I have such 100 states, with different values, my use case contains on 5 steps.
Start Each state as a Gen{?}.
Send an HTTP request to that URL.
Get results.
Send another HTTP request with the data came from the first request.
Put the Process to sleep. if the frequency is 10 then for 10 seconds and so on and after 10 seconds It will start from 1 Step again.
Now when I will start 100 such workers, there are going to be 100 * 2 HTTP requests with frequency. I am not sure about either I am going to use GenServer or GenStage or Flow or even Broadway?
I am also concerned the HTTP requests won't collapse such as one worker, with a state, will send a request and if the frequency is 1 Second before the first request comes back, the other request would have been sent, would GenServer is capable enough to handle those cases? which I think are called back pressure?
I have been asking and looking at this use case of so long, I have been guided towards RabbitMQ as well for my use case.
Any guidance would be so helpful, or any minimal example would be so grateful.
? GenServer/ GenStage / GenStateMachine
Your problem comes down to reducing the number of concurrent network requests at a given time.
A simple approach would be to have a GenServer which keeps track of the count of outgoing requests. Then, for each client (Up to 200 in your case), it can check to see if there's an open request, and then act accordingly. Here's what the server could look like:
defmodule Throttler do
use GenServer
#server
#impl true
def init(max_concurrent: max_concurrent) do
{:ok, %{count: 0, max_concurrent: max_concurrent}}
end
#impl true
def handle_call(:run, _from, %{count: count, max_concurrent: max_concurrent} = state) when count < max_concurrent, do: {:reply, :ok, %{state | count: count + 1}}
#impl true
def handle_call(:run, _from, %{count: count, max_concurrent: max_concurrent} = state) when count >= max_concurrent, do: {:reply, {:error, :too_many_requests}, state}
#impl true
def handle_call(:finished, _from, %{count: count} = state) when count > 0, do: {:reply, :ok, %{state | count: count - 1}}
end
Okay, so now we have a server where we can call handle_call(pid, :run) and it will tell us whether or not we've exceeded the count. Once the task (getting the URL) is complete, we need to call handle_call(pid, :finished) to let the server know we've completed the task.
On the client side, we can wrap that in a convenient helper function. (Note this is still within the Throttler module so __MODULE__ works)
defmodule Throttler do
#client
def start_link(max_concurrent: max_concurrent) when max_concurrent > 0 do
GenServer.start_link(__MODULE__, [max_concurrent: max_concurrent])
end
def execute_async(pid, func) do
GenServer.call(pid, :run)
|> case do
:ok ->
task = Task.async(fn ->
try do
func.()
after
GenServer.call(pid, :finished)
end
end)
{:ok, task}
{:error, reason} -> {:error, reason, func}
end
end
end
Here we pass in a function that we want to asynchronously execute on the client side, and do the work of calling :run and :finished on the server side before executing. If it succeeds, we get a task back, otherwise we get a failure.
Putting it all together, and you get code that looks like this:
{:ok, pid} = Throttler.start_link(max_concurrent: 3)
results = Enum.map(1..5, fn num ->
Throttler.execute(pid, fn ->
IO.puts("Running command #{num}")
:timer.sleep(:5000)
IO.puts("Sleep complete for #{num}")
num * 10
end)
end)
valid_tasks = Enum.filter(results, &(match?({:ok, _func}, &1))) |> Enum.map(&elem(&1, 1))
Now you have a bunch of tasks that either succeeded, or failed and you can act appropriately.
What do you do upon failure? That's the interesting part of backpressure :) The simplest thing would be to have a timeout and retry, under the assumption that you will eventually clear the pressure downstream. Otherwise you can fail out the requests entirely and keep pushing the problem upstream.
I'm running GenServer as a background job which is rescheduled each interval by Process.send_after(self(), :work, #interval).
This job is started by Supervisor when Application starts.
It's working perfectly, but now I want to test if my GenServer module is really spawning new process each interval.
How can I test it?
EDIT
I found that :sys.get_status(pid) can be use to fetch some data about process, but I would really like to use something like receive do ... end
EDIT 2
handle_info/2 function:
#impl true
def handle_info(:work, state) do
do_smt()
schedule_worker()
{:noreply, state}
end
schedule_worker/0 function:
defp schedule_worker do
Process.send_after(self(), :work, #interval)
end
There's something missing in your message. From what you have posted we can understand that every #interval milliseconds a :work message is sent. You are not telling us what the handle_info/2 is supposed to do when the message is dispatched.
Once this is defined, you can definitely write a test to assert that a message has been received by using the assert_received assertion.
I would test do_smt() by using Mock library and writing a test that makes as assertion like the following:
with_mock(MyModule, [do_stm_else: fn -> :ok]) do
do_smt()
assert_called MyModule.do_stm_else()
end
In this way, you have called the function that the task should execute, so you can assume that the task creation is being called.
If you want to let the do_stm_else function communicate with your test (in this scenario it looks a bit overengineered) you should:
get the pid of the test by calling self()
Pass the pid to the mock function to get it used
use assert_receive to verify that the communication has occurred
pid = self()
with_mock(MyModule, [do_stm_else: fn ->
Process.send(pid, :msg)
]) do
do_smt()
assert_called MyModule.do_stm_else()
end
assert_receive(:msg)
Please note that I had no time to check this, you should spend a bit to investigate.
I am trying to write a Gossip Simulator in Elixir using GenServer. I have a main() method which acts as a client creating a network topology and starts all the actors (GenServer's). It then sends a Genserver.cast() to an Actor to initiate the gossip. The Actor in its handle_cast() starts a Task.start() to start gossiping with other actors. Looks like I am not using Task.start() (line 16 in actor.ex) in the right way as the called task startGossiping() is never executed nor the statements after Task.start(). Mix just exits without giving any error. The shortened program is given below.
actor.ex -
defmodule Actor do
use GenServer
def init([nodeId, neighborList, algorithm]) do
inspect "#{nodeId}"
recCount = 1
gossipingTask = 0
{:ok, {nodeId, neighborList, algorithm, recCount, gossipingTask}}#nodeId, neighborList, algorithm, receivedCount
end
def handle_cast({:message, rumour}, state) do
{nodeId, neighborList, algorithm, recCount, gossipingTask} = state
IO.puts "nodeId - #{nodeId} recCount - #{recCount} handle_cast: #{rumour} gossipingTask - #{gossipingTask}"
nL = elem(state, 1)
IO.puts "here #{rumour}"
gossipingTask = Task.start(fn -> startGossiping(nL, rumour) end)
IO.puts "Now again - #{rumour}"
{:noreply, {nodeId, neighborList, algorithm, recCount + 1, gossipingTask}}
end
def startGossiping(nL, rumour) do
IO.puts "In startGossiping "
#{Enum.random(nL)}"
# GenServer.cast(Proj2.intToAtom(Enum.random(nL)), {:message, rumour})
end
end
proj2.ex -
defmodule Proj2 do
# Instructions to run the project
# 1) $mix escript.build
# 2) $escript proj2 100 full gossip
def main(args) do
# Receive total number of nodes, topology, algorithm, triggerNodes(optional), threshold(optional) from user.
# Read README.md for more details
numOfNodes = String.to_integer(Enum.at(args, 0))
topology = Enum.at(args, 1)
algorithm = Enum.at(args, 2)
numOfNodes = if String.contains?(topology, "2d"), do: round(:math.pow(round(:math.sqrt(numOfNodes)), 2)), else: numOfNodes
case topology do
"full" ->
Enum.each 1..numOfNodes, fn nodeId ->
neighborList = getNeighborsFull(nodeId, numOfNodes)
inspect neighborList
nodeId_atom = intToAtom(nodeId)
GenServer.start_link(Actor, [nodeId, neighborList, algorithm], name: nodeId_atom)
# IO.puts "In main, nodeId = #{nodeId}"
end
end
GenServer.cast(intToAtom(2), {:message, "This is Elixir Gossip Simulator"})
end
def getNeighborsFull(nodeId,numOfNodes) do
range = 1..numOfNodes
range
|> Enum.filter(fn(value) -> value != nodeId end)
|> Enum.map(fn(filtered_value) -> filtered_value * 1 end)
# IO.inspect Neighboringlist
end
def intToAtom(integer) do
integer |> Integer.to_string() |> String.to_atom()
end
end
UPDATE :
Haven't figured out the problem still. I am not able to start any concurrent process actually. spawn, start_link, Task none of them are starting an asynchronous task.
Neither GenServer.cast/3 nor Task.start/1 would prevent your application from exiting.
I am unsure why would you build an escript to test the code. You have the following options to wait until the execution finishes before exiting (including but not limited to):
• use mix run --no-halt
• create an Application
• start the task linked and use Task.await/2 somewhere in your code to wait while the task is finished.
I want to listening SerialPort and when message occurs then get or create Log model with id received from my device.
How to load once automatically SerialPort and keep established connection and if key_detected? in listener deal with Log model?
This is my autoloaded module in lib:
module Serialport
class Connection
def initialize(port = "/dev/tty0")
port_str = port
baud_rate = 9600
data_bits = 8
stop_bits = 1
parity = SerialPort::NONE
#sp = SerialPort.new(port_str, baud_rate, data_bits, stop_bits, parity)
#key_parts = []
#key_limit = 16 # number of slots in the RFID card.
while true do
listener
end
#sp.close
end
def key_detected?
#key_parts << #sp.getc
if #key_parts.size >= #key_limit
self.key = #key_parts.join()
#key_parts = []
true
else
false
end
end
def listener
if key_detected?
puts self.key
# log = Log.find(rfid: self.key).first_or_create(rfid: self.key)
end
end
end
end
Model:
class Log < ActiveRecord::Base
end
I would have written this in a comment, but it's a bit long... But I wonder if you could clarify your question, and I will update my answer as we go:
With all due respect to the Rails ability to "autoload", why not initialize a connection in an initialization file or while setting up the environment?
i.e., within a file in you_app/config/initializers called serial_port.rb:
SERIAL_PORT_CONNECTION = Serialport::Connection.new
Implementing an infinite loop within your Rails application will, in all probability, hang the Rails app and prevent it from being used as a web service.
What are you trying to accomplish?
If you just want to use active_record or active_support, why not just include these two gems in a separate script?
Alternatively, consider creating a separate thread for the infinite loop (or better yet, use a reactor (They are not that difficult to write, but there are plenty pre-written in the wild, such as Iodine which I wrote for implementing web services)...
Here's an example for an updated listener method, using a separate thread so you call it only once:
def listener
Thread.new do
loop { self.key while key_detected? }
# this will never be called - same as in your code.
#sp.close
end
end
I want to implement an action that calls remote service with RabbitMQ and presents returned data. I implemented this (more as a proof of concept so far) in similar way to example taken from here: https://github.com/baowen/RailsRabbit and it looks like this:
controller:
def rpc
text = params[:text]
c = RpcClient.new('RPC server route key')
response = c.call text
render text: response
end
RabbitMQ RPC client:
class RpcClient < MQ
attr_reader :reply_queue
attr_accessor :response, :call_id
attr_reader :lock, :condition
def initialize()
# initialize exchange:
conn = Bunny.new(:automatically_recover => false)
conn.start
ch = conn.create_channel
#x = ch.default_exchange
#reply_queue = ch.queue("", :exclusive => true)
#server_queue = 'rpc_queue'
#lock = Mutex.new
#condition = ConditionVariable.new
that = self
#reply_queue.subscribe do |_delivery_info, properties, payload|
if properties[:correlation_id] == that.call_id
that.response = payload.to_s
that.lock.synchronize { that.condition.signal }
end
end
end
def call(message)
self.call_id = generate_uuid
#x.publish(message.to_s,
routing_key: #server_queue,
correlation_id: call_id,
reply_to: #reply_queue.name)
lock.synchronize { condition.wait(lock) }
response
end
private
def generate_uuid
# very naive but good enough for code
# examples
"#{rand}#{rand}#{rand}"
end
end
A few tests indicate that this approach works. On the other hand, this approach assumes creating a client (and subscribing to the queue) for every request on this action, which is inefficient according to the RabbitMQ tutorial. So I've got two questions:
Is it possible to avoid creating a queue for every Rails request?
How will this approach (with threads and mutex) interfere with my whole Rails environment? Is it safe to implement things this way in Rails? I'm using Puma as my web server, if it's relevant.
Is it possible to avoid creating a queue for every Rails request?
Yes - there is no need for every single request to have it's own reply queue.
You can use the built-in direct-reply queue. See the documentation here.
If you don't want to use the direct-reply feature, you can create a single reply queue per rails instance. You can use a single reply queue, and have the correlation id help you figure out where the reply needs to go within that rails instance.
How will this approach (with threads and mutex) interfere with my whole Rails environment? Is it safe to implement things this way in Rails?
what's the purpose of the lock / mutex in this code? doesn't seem necessary to me, but i'm probably missing something since i haven't done ruby in about 5 years :)