When trying to start my process from a DynamicSupervisor im getting the following error:
{:error,
{:EXIT,
{:badarg,
[
{:erlang, :apply,
[
BfgEngine.MarketService,
:start_link,
{{BfgEngine.MarketService, :start_link, ["1111"]}, :permanent, 5000,
:worker, [BfgEngine.MarketService]}
], []},
{:supervisor, :do_start_child_i, 3, [file: 'supervisor.erl', line: 379]},
{:supervisor, :handle_call, 3, [file: 'supervisor.erl', line: 404]},
{:gen_server, :try_handle_call, 4, [file: 'gen_server.erl', line: 661]},
{:gen_server, :handle_msg, 6, [file: 'gen_server.erl', line: 690]},
{:proc_lib, :init_p_do_apply, 3, [file: 'proc_lib.erl', line: 249]}
]}}}
The code im using is:
def start_market(market_id) do
spec = {MarketService, market_id}
DynamicSupervisor.start_child(__MODULE__, spec)
end
However it is not clear to me what is going wrong. What argument to which function is it thats not correct? How do I break down and read the given error message?
Update:
This is the init method of my supervisor:
#impl true
def init(initial_arg) do
DynamicSupervisor.init(
strategy: :one_for_one,
extra_arguments: [initial_arg]
)
end
Update 2:
This is the start_link of market_service:
def start_link(market_id) when is_bitstring(market_id) do
GenServer.start_link(__MODULE__, market_id, name: via_tuple(market_id))
end
Im using the default child_spec im getting from GenServer
Update 3:
Changing to:
def start_market(market_id) do
spec = {MarketService, market_id: market_id}
DynamicSupervisor.start_child(__MODULE__, spec)
end
Gives:
{:error,
{:undef,
[
{BfgEngine.MarketService, :start_link, [[], [market_id: "222"]], []},
{DynamicSupervisor, :start_child, 3,
[file: 'lib/dynamic_supervisor.ex', line: 654]},
{DynamicSupervisor, :handle_start_child, 2,
[file: 'lib/dynamic_supervisor.ex', line: 640]},
{:gen_server, :try_handle_call, 4, [file: 'gen_server.erl', line: 661]},
{:gen_server, :handle_msg, 6, [file: 'gen_server.erl', line: 690]},
{:proc_lib, :init_p_do_apply, 3, [file: 'proc_lib.erl', line: 249]}
]}}
Changing to:
def start_market(market_id) do
spec = {MarketService, :market_id, market_id}
DynamicSupervisor.start_child(__MODULE__, spec)
end
Gives:
** (ArgumentError) supervisors expect each child to be one of:
* a module
* a {module, arg} tuple
* a child specification as a map with at least the :id and :start fields
* or a tuple with 6 elements generated by Supervisor.Spec (deprecated)
Got: {BfgEngine.MarketService, :market_id, "222"}
(elixir) lib/supervisor.ex:657: Supervisor.init_child/1
(elixir) lib/supervisor.ex:744: Supervisor.child_spec/2
(elixir) lib/dynamic_supervisor.ex:304: DynamicSupervisor.start_child/2
You got badarg exception to function erlang:apply/3 when there are three arguments BfgEngine.MarketService, :start_link and {{BfgEngine.MarketService, :start_link, ["1111"]}, :permanent, 5000, :worker, [BfgEngine.MarketService]} and it happen in function supervisor:do_start_child_i/3.
The arguments to the function erlang:apply/3 should be MFA a.k.a Module, Function, Arguments. {{BfgEngine.MarketService, :start_link, ["1111"]}, :permanent, 5000, :worker, [BfgEngine.MarketService]} is not Arguments because it obviously is not a list of arguments. From your code, I can guess the error is the content of variable spec. You should provide some proplist or map. I don't know, you should read the documentation of DynamicSupervisor more carefully.
The error message
As of understanding the error message thrown by Elixir, you can refer to the official Erlang documentation. The section about errors and exceptions from Learn You Some Erlang For Great Good can help. The answer of #Hynek -Pichi Vychodil is also accurate.
Your specific problem
As #Milan Jaric mentioned, your error would come from:
def start_market(market_id) do
spec = {MarketService, market_id}
DynamicSupervisor.start_child(__MODULE__, spec)
end
But not only! DynamicSupervisor.start_child(__MODULE__, spec) is calling MarketService.start_link/1!
Your problem lies in the combination of this function from the DynamicSupervisor module, and the way you parse values in MarketService.start_link/1:
def start_link(market_id) when is_bitstring(market_id) do
GenServer.start_link(__MODULE__, market_id, name: via_tuple(market_id))
end
Indeed, this code should work, if you also have implemented MarketService.init/1 correctly. I am not able to reproduce the error. Are you sure market_id is really a bitstring?
Personally, I had based my code on the official documentation:
defmodule MySupervisor do
use DynamicSupervisor
def start_link(init_arg) do
DynamicSupervisor.start_link(__MODULE__, init_arg, name: __MODULE__)
end
def start_child(foo, bar, baz) do
# If MyWorker is not using the new child specs, we need to pass a map:
# spec = %{id: MyWorker, start: {MyWorker, :start_link, [foo, bar, baz]}}
spec = {MyWorker, foo: foo, bar: bar, baz: baz}
DynamicSupervisor.start_child(__MODULE__, spec)
end
#impl true
def init(init_arg) do
DynamicSupervisor.init(
strategy: :one_for_one,
extra_arguments: [init_arg]
)
end
end
As you can see, they suggest to use a Keyword list here:
spec = {MyWorker, foo: foo, bar: bar, baz: baz}
It works only if you've implemented MyWorker.start_link/1 as follows:
def start_link(args) do
foo = Keyword.fetch!(args, :foo)
bar = Keyword.fetch!(args, :bar)
baz = Keyword.fetch!(args, :baz)
Genserver.start_link(__MODULE__, {foo, bar, baz}, [])
def init({foo, bar, baz}) do
# do something...
state = {foo, bar, baz}
{:ok, state}
In your case, if you change start_market/1 to:
def start_market(market_id) do
spec = {MarketService, market_id: market_id}
DynamicSupervisor.start_child(__MODULE__, spec)
end
It will not work because this MarketService.start_link/1 will fail:
def start_link(market_id) when is_bitstring(market_id) do
GenServer.start_link(__MODULE__, market_id, name: via_tuple(market_id))
end
market_id here is not a bitstring but a Keyword list. So you got to modify MarketService.start_link/1 function to:
def start_link(args) when is_list(args) do
market_id = Keyword.fetch!(args, :market_id)
GenServer.start_link(__MODULE__, market_id, name: via_tuple(market_id))
end
And write a MarketService.init/1 as follows:
def init(market_id) do
# do something... Let's keep it simple for the example:
state = market_id
{:ok, state}
end
Helpful resources
José Valim's answer on a similar issue posted on Elixir Forum
Official module-based DynamicSupervisor documentation
A well-written series of articles about GenServer and Supervisors in Elixir...
...and its corresponding source code
It is hard to tell from posted code, but could you try changing start_market to:
def start_market(market_id) do
spec = {MarketService, :market_id, market_id}
DynamicSupervisor.start_child(__MODULE__, spec)
end
UPDATE (Below are two options):
def start_market(market_id) do
spec = &{
id: MarketService,
start: {MarketService, start_link, [market_id]},
type: :worker
}
DynamicSupervisor.start_child(__MODULE__, spec)
end
OR
def start_market(market_id) do
spec = {MarketService, [market_id]}
DynamicSupervisor.start_child(__MODULE__, spec)
end
Related
EDIT: same things happens when I fork a process manually...
I'm getting some weird behavior with a Rails Job that calls a module of mine called RedisService.
I've added lib/modules to my autoload_paths but the TextService module that calls the RedisService one loses reference to it, sometimes immediately, sometimes 3 or 4 job calls in...
I've even required the module in my TextService to no avail, even added some puts to check that always show the module is defined and responds to the method I'm calling...!
Something escapes me...
Here's a gist to the backtrace
Repo: https://gitlab.com/thomasbromehead/snmp-simulator-ruby-manager.
ruby --version: 2.6.5
rails version: 6.1.3.1
My "service" objects:
Module that calls RedisService
require_relative 'redis_service'
module TextService
def self.write_to_file(dataObject, redis, path: "./")
begin
file_with_path = path + dataObject.filename
# Store all lines prior to the one being modified, File.read closes the file
f = File.read(file_with_path)
new_content = f.gsub(dataObject.old_set_value, dataObject.new_set_value)
# File.open closes the file when passed a block
File.open(file_with_path, "w") { |file| file.puts new_content }
puts "Redis is: #{redis}" ======> RedisService
puts "Redis responds to multi: #{redis.respond_to?(:multi)}" ======> true
redis.multi do
redis.zrem("#{dataObject.name}-sorted-set", dataObject.old_set_value)
redis.hset("#{dataObject.name}-offsets", "#{dataObject.start_index}:#{dataObject.oid}:#{dataObject.end_index}", dataObject.new_set_value)
redis.zadd("#{dataObject.name}-sorted-set", dataObject.start_index, dataObject.new_set_value)
end
rescue EOFError
end
end
Variation class called from VariateJob
require_relative '../../../lib/modules/redis_service'
module Snmp
class Variation
include ActiveModel::Model
attr_reader :oid, :type, :duration, :to, :from, :filename, :redis
def initialize(oid:nil, duration:nil, type:nil, to:nil, filename: nil, from:nil)
#to = to
#from = from
#oid = oid
#type = type
#filename = filename
#redis = RedisService
end
def run(data)
current_value, new_set_value, start_index, end_index = prepare_values(JSON.parse(data))
transferData = Snmp::TransferData.new({
filename: #filename,
old_set_value: current_value,
new_set_value: new_set_value,
start_index: start_index,
end_index: end_index,
name: #name,
oid: oid
})
TextService.write_to_file(transferData, #redis)
end
VariateJob
class VariateJob < ApplicationJob
queue_as :default
def perform(dumped_variation, data)
Marshal.load(dumped_variation).run(Marshal.load(data))
end
end
VariationsController
class VariationsController < ApplicationController
before_action :set_file_name, only: :start
def start
if params["linear"]
type = :linear
elsif params["random"]
type = :random
end
data = redis.hscan_each("##name-offsets", match: "*:#{params["snmp_variation"]["oid"]}*")
# data is an Enumerator, transform it to an array and dump to JSON
variation = Snmp::Variation.new(params_to_keywords(params["snmp_variation"]).merge({type: type}))
VariateJob.perform_later(Marshal.dump(variation), Marshal.dump(JSON.generate(data.to_a.first)))
end
RedisService
require 'redis'
module RedisService
include GlobalID::Identification
[...]
def self.multi(&block)
#redis.multi { block.call() }
end
[...]
end
You are not losing the reference to the RedisService, but to Redis in your RedisService. Probably because you use a server or worker that forks new processes and you don't initialize a new connection after the fork.
To fix this issue I would replace this method
def self.start(host,port)
#redis ||= Redis.new(host: host, port: port)
self
end
with
def self.redis
#redis ||= Redis.new(host: ::Snmpapp.redis_config[:host], port: ::Snmpapp.redis_config[:port])
end
And then I would replace all call to the #redis with a redis call to the new method.
I want to communicate between Elixir and Python. I don't want to use NIFs and stuff - I prefer loosely coupled using zeroMQ as this will allow me to use other languages than Python later. I am using the chumak library which is a native implementation of zeromq in Erlang, and seems well maintained. I have used it successfully in the past for pub sub.
Apart from pub-sub, I'm finding that req-rep and req-router sockets work fine. However dealer-router does not. This is really important because only dealer and router give you true async in zeromq.
Here is the python code for the router side:
import zmq
context = zmq.Context()
rout = context.socket(zmq.ROUTER)
rout.bind("tcp://192.168.1.192:8760")
Here is the Elixir req code which works fine...
iex(1)> {ok, sock1} = :chumak.socket(:req, 'reqid')
{:ok, #PID<0.162.0>}
iex(2)> {ok, _peer} = :chumak.connect(sock1, :tcp, '192.168.1.192', 8760)
{:ok, #PID<0.164.0>}
iex(3)> :chumak.send(sock1, 'hello from req socket')
:ok
.... because I get it on the Python side:
In [5]: xx = rout.recv_multipart()
In [6]: xx
Out[6]: ['reqid', '', 'hello from req socket']
However, here is what I get if I try a dealer socket on the Elixir side:
iex(4)> {ok, sock2} = :chumak.socket(:dealer, 'dealid')
{:ok, #PID<0.170.0>}
iex(5)> {ok, _peer} = :chumak.connect(sock2, :tcp, '192.168.1.192', 8760)
{:ok, #PID<0.172.0>}
iex(6)> :chumak.send(sock2, 'hello from dealer socket')
{:error, :not_implemented_yet}
iex(7)> :chumak.send_multipart(sock2, ['a', 'b', 'hello from dealer socket'])
22:13:38.705 [error] GenServer #PID<0.172.0> terminating
** (FunctionClauseError) no function clause matching in :chumak_protocol.encode_more_message/3
(chumak) /home/tbrowne/code/elixir/chutest/deps/chumak/src/chumak_protocol.erl:676: :chumak_protocol.encode_more_message('a', :null, %{})
(stdlib) lists.erl:1354: :lists.mapfoldl/3
(chumak) /home/tbrowne/code/elixir/chutest/deps/chumak/src/chumak_protocol.erl:664: :chumak_protocol.encode_message_multipart/3
(chumak) /home/tbrowne/code/elixir/chutest/deps/chumak/src/chumak_peer.erl:159: :chumak_peer.handle_cast/2
(stdlib) gen_server.erl:616: :gen_server.try_dispatch/4
(stdlib) gen_server.erl:686: :gen_server.handle_msg/6
(stdlib) proc_lib.erl:247: :proc_lib.init_p_do_apply/3
Last message: {:"$gen_cast", {:send, ['a', 'b', 'hello from dealer socket'], {#PID<0.160.0>, #Reference<0.79795089.2401763329.172383>}}}
State: {:state, :ready, '192.168.1.192', 8760, :client, [], :dealer, 'dealid', [], {3, 0}, #Port<0.4968>, {:decoder, :ready, 0, nil, nil, {:some, 3}, {:some, 0}, %{}, :null, false}, #PID<0.170.0>, {[], []}, [], false, false, false, :null, %{}}
22:13:38.710 [info] [:unhandled_handle_info, {:module, :chumak_socket}, {:msg, {:EXIT, #PID<0.172.0>, {:function_clause, [{:chumak_protocol, :encode_more_message, ['a', :null, %{}], [file: '/home/tbrowne/code/elixir/chutest/deps/chumak/src/chumak_protocol.erl', line: 676]}, {:lists, :mapfoldl, 3, [file: 'lists.erl', line: 1354]}, {:chumak_protocol, :encode_message_multipart, 3, [file: '/home/tbrowne/code/elixir/chutest/deps/chumak/src/chumak_protocol.erl', line: 664]}, {:chumak_peer, :handle_cast, 2, [file: '/home/tbrowne/code/elixir/chutest/deps/chumak/src/chumak_peer.erl', line: 159]}, {:gen_server, :try_dispatch, 4, [file: 'gen_server.erl', line: 616]}, {:gen_server, :handle_msg, 6, [file: 'gen_server.erl', line: 686]}, {:proc_lib, :init_p_do_apply, 3, [file: 'proc_lib.erl', line: 247]}]}}}]
As you can see I get this huge error on the :chumak.send_multipart, while :chumak.send doesn't even work. What's going on here?
The dealer socket works fine by the way from the Python side:
import zmq
context = zmq.Context()
deal = context.socket(zmq.DEALER)
deal.setsockopt_string(zmq.IDENTITY, u"Thomas")
deal.connect("tcp://192.168.1.192:8760")
deal.send("hello from python deal")
Now on router side:
In [5]: xx = rout.recv_multipart()
In [6]: xx
Out[6]: ['reqid', '', 'hello from req socket']
In [7]: dd = rout.recv_multipart()
In [8]: dd
Out[8]: ['Thomas', 'hello from python deal']
So I'm wondering if I have a syntax, or type error, in my Elixir chumak dealer socket, or if it's simply a bug. I have tried this on both amd64 and armv7l architectures and the problem is identical.
All the elixir code is based on the Erlang version in the chumak example for dealer-router.
My mix.exs deps looks like this:
[
{:chumak, "~> 1.2"},
{:msgpack, "~> 0.7.0"}
]
the only obvious thing I see is your use of send_multipart. Its signature in the source:
-spec send_multipart(SocketPid::pid(), [Data::binary()]) -> ok.
you are doing this:
:chumak.send_multipart(sock2, ['a', 'b', 'hello from dealer socket'])
------------
iex(2)> is_binary('a')
false
iex(3)> is_binary('hello from dealer socket')
false
Otherwise, I can not see much of a difference between your code and the example code that is in chumak's repo.
I'm starting a new project, my first with Rails 5.1.0. I have a pb with my first request spec.
describe 'Users', type: :request do
it 'are created from external data' do
json_string = File.read('path/to/test_data/user_data.json')
params = { user: JSON.parse(json_string) }
headers = { "CONTENT_TYPE" => "application/json" }
expect do
post '/api/v1/users', params.to_s, headers
end.to change {
User.count
}.by(1)
expect(response.status).to eq 200
end
end
this spec return the error ArgumentError: wrong number of arguments (given 3, expected 1). The official documentation don't say much.
If I take out the .to_s, and send a hash, like this:
post '/api/v1/users', params, headers
I got another error:
ArgumentError: unknown keyword: user
Any thought?
I think they changed the syntax recently. Now it should use keyword args. So, something like this:
post '/api/v1/users', params: params, headers: headers
Here's a little addendum to Sergio's answer. If you are upgrading from Rails 4 to Rails 5, have lots of tests, and aren't too keen on changing them all – at least not until you've finished upgrading – I've found a way to make them work with the old method signature.
In my spec_helper I added
module FixLegacyTestRequests
def get(path, par = {}, hdr = {})
process(:get, path, params: par, headers: hdr)
end
def post(path, par = {}, hdr = {})
process(:post, path, params: par, headers: hdr)
end
def put(path, par = {}, hdr = {})
process(:put, path, params: par, headers: hdr)
end
def delete(path, par = {}, hdr = {})
process(:delete, path, params: par, headers: hdr)
end
end
and then I added this configuration for each test:
RSpec.configure do |config|
config.before :each do |example|
extend(FixLegacyTestRequests) # to be removed at some point!
end
end
My tests went back to working, and I think it should be safe because it's only applied to the currently running test and shouldn't pollute any gem's code such as with a monkey patch.
I have a problem with a long-time consuming migration, which I desired to run in the parallel (it can be runned in the parallel). Actually migration is about taking all records in the database and implement time- and resource- consuming operations on each of them.
Sometimes individual record migration hanging out, so I give 10 minutes to finish. If migration isn't finished, I want it to gracefully shut down without any exception (see below)
I'm also using poolboy erlang package to parallelize implementation since migration consumes not only the time, but resources too. The problem is that I don't know how to handle error when timeout happened and code is going to break. My supervision tree is:
defmodule MyReelty.Repo.Migrations.MoveVideosFromVimeoToB2 do
use Ecto.Migration
alias MyReelty.Repo
alias MyReelty.Repo.Migrations.MoveVideosFromVimeoToB2.Migrator
# parallel nature of migration force us to disable transaction
#disable_ddl_transaction true
#migrator_waiting_time 10 * 60 * 1000 # timeout
#poolboy_waiting_time #migrator_waiting_time + 10 * 1000 # give a time for graceful shutdown
#pool_name :migrator
#pool_size 3
#pool_config [
{ :name, { :local, #pool_name }},
{ :worker_module, Migrator },
{ :size, #pool_size },
{ :max_overflow, 0 },
{ :strategy, :fifo }
]
def up do
children = [
:poolboy.child_spec(#pool_name, #pool_config)
]
opts = [strategy: :one_for_one, name: MyReelty.Supervisor]
Supervisor.start_link(children, opts)
rows = Review |> Repo.all
IO.puts "Total amount of reviews is: #{length(rows)}"
parallel_migrations(rows)
end
def parallel_migrations(rows) do
Enum.map(rows, fn(row) ->
pooled_migration(#pool_name, row)
end)
end
def pooled_migration(pool, x) do
:poolboy.transaction(
pool,
(fn(pid) -> Migrator.move(pid, { x, #migrator_waiting_time }) end),
#poolboy_waiting_time
)
end
defmodule Migrator do
alias MyReelty.Repo
alias MyReelty.Review
use GenServer
def start_link(_) do
GenServer.start_link(__MODULE__, nil, [])
end
def move(server, { params, waiting_time }) do
GenServer.call(server, { :move, params }, waiting_time)
end
def handle_call({ :move, result }, _from, state) do
big_time_and_resource_consuming_task_here
{:reply, %{}, state}
end
end
end
The problem if migration of some record in the database takes more than 10 mins I have this kind of exception:
20:18:16.917 [error] Task #PID<0.282.0> started from #PID<0.70.0> terminating
** (stop) exited in: GenServer.call(#PID<0.278.0>, {:move, [2, "/videos/164064419", "w 35th st Springfield United States Illinois 60020"]}, 60000)
** (EXIT) time out
(elixir) lib/gen_server.ex:604: GenServer.call/3
(poolboy) src/poolboy.erl:76: :poolboy.transaction/3
(elixir) lib/task/supervised.ex:94: Task.Supervised.do_apply/2
(elixir) lib/task/supervised.ex:45: Task.Supervised.reply/5
(stdlib) proc_lib.erl:247: :proc_lib.init_p_do_apply/3
Function: #Function<5.53617785/0 in MyReelty.Repo.Migrations.MoveVideosFromVimeoToB2.parallel_migrations/1>
Args: []
20:18:16.918 [error] GenServer MyReelty.Repo terminating
** (stop) exited in: GenServer.call(#PID<0.278.0>, {:move, [2, "/videos/164064419", "w 35th st Springfield United States Illinois 60020"]}, 60000)
** (EXIT) time out
Last message: {:EXIT, #PID<0.70.0>, {:timeout, {GenServer, :call, [#PID<0.278.0>, {:move, [2, "/videos/164064419", "w 35th st Springfield United States Illinois 60020"]}, 60000]}}}
State: {:state, {:local, MyReelty.Repo}, :one_for_one, [{:child, #PID<0.231.0>, DBConnection.Poolboy, {:poolboy, :start_link, [[name: {:local, MyReelty.Repo.Pool}, strategy: :fifo, size: 1, max_overflow: 0, worker_module: DBConnection.Poolboy.Worker], {Postgrex.Protocol, [types: true, username: "adik", types: true, name: MyReelty.Repo.Pool, otp_app: :my_reelty, repo: MyReelty.Repo, adapter: Ecto.Adapters.Postgres, database: "my_reelty_dev", hostname: "localhost", extensions: [{Geo.PostGIS.Extension, [library: Geo]}, {Ecto.Adapters.Postgres.DateTime, []}, {Postgrex.Extensions.JSON, [library: Poison]}], pool_size: 1, pool_timeout: 5000, timeout: 15000, adapter: Ecto.Adapters.Postgres, database: "my_dev", hostname: "localhost", pool_size: 10, pool: DBConnection.Poolboy, port: 5432]}]}, :permanent, 5000, :worker, [:poolboy]}], :undefined, 3, 5, [], 0, Ecto.Repo.Supervisor, {MyReelty.Repo, :my_reelty, Ecto.Adapters.Postgres, [otp_app: :my_reelty, repo: MyReelty.Repo, adapter: Ecto.Adapters.Postgres, database: "my_reelty_dev", hostname: "localhost", extensions: [{Geo.PostGIS.Extension, [library: Geo]}], pool_size: 1]}}
I tried to insert terminate/2 or handle_info/2 to Migrator and play with it, but I even haven't reached this functions to be invoked. How can I handle timeouts and prevent them to break my migration?
UPDATED
I used #johlo's hint, but I still getting time out. My function is:
def init(_) do
Process.flag(:trap_exit, true)
{:ok, %{}}
end
When the Migrator.move/2 (i.e. the GenServer.call) function times out it will crash the entire MoveVideosFromVimeoToB2 process since that's the actual process that makes the GenServer call.
The solution here is to catch the timeout in the anonymous function in pooled_migration, something like (I'm not very familiar with Elixir syntax, so it might not compile, but you should get the idea) :
def pooled_migration(pool, x) do
:poolboy.transaction(
pool,
(fn(pid) ->
try do
Migrator.move(pid, { x, #migrator_waiting_time })
catch
:exit, reason ->
# Ignore error, log it or something else
:ok
end
end),
#poolboy_waiting_time
)
end
It's not the Migrator process that times out, it's the GenServer call to the Migrator that does and we need to try-catch that.
Also note that the Migrator process isn't killed it is still running, see the timeouts section in the GenServer call documentation.
UPDATE:
As #asiniy mentions in the comments the #poolboy_waiting_time should be set to :infinity so the poolboy.transaction function doesn't throw a timeout error when waiting for a free Migrator worker process. Since the Migrator will exit eventually this is safe.
I would like my JSON output in Ruby on Rails to be "pretty" or nicely formatted.
Right now, I call to_json and my JSON is all on one line. At times this can be difficult to see if there is a problem in the JSON output stream.
Is there way to configure to make my JSON "pretty" or nicely formatted in Rails?
Use the pretty_generate() function, built into later versions of JSON. For example:
require 'json'
my_object = { :array => [1, 2, 3, { :sample => "hash"} ], :foo => "bar" }
puts JSON.pretty_generate(my_object)
Which gets you:
{
"array": [
1,
2,
3,
{
"sample": "hash"
}
],
"foo": "bar"
}
The <pre> tag in HTML, used with JSON.pretty_generate, will render the JSON pretty in your view. I was so happy when my illustrious boss showed me this:
<% if #data.present? %>
<pre><%= JSON.pretty_generate(#data) %></pre>
<% end %>
Thanks to Rack Middleware and Rails 3 you can output pretty JSON for every request without changing any controller of your app. I have written such middleware snippet and I get nicely printed JSON in browser and curl output.
class PrettyJsonResponse
def initialize(app)
#app = app
end
def call(env)
status, headers, response = #app.call(env)
if headers["Content-Type"] =~ /^application\/json/
obj = JSON.parse(response.body)
pretty_str = JSON.pretty_unparse(obj)
response = [pretty_str]
headers["Content-Length"] = pretty_str.bytesize.to_s
end
[status, headers, response]
end
end
The above code should be placed in app/middleware/pretty_json_response.rb of your Rails project.
And the final step is to register the middleware in config/environments/development.rb:
config.middleware.use PrettyJsonResponse
I don't recommend to use it in production.rb. The JSON reparsing may degrade response time and throughput of your production app. Eventually extra logic such as 'X-Pretty-Json: true' header may be introduced to trigger formatting for manual curl requests on demand.
(Tested with Rails 3.2.8-5.0.0, Ruby 1.9.3-2.2.0, Linux)
If you want to:
Prettify all outgoing JSON responses from your app automatically.
Avoid polluting Object#to_json/#as_json
Avoid parsing/re-rendering JSON using middleware (YUCK!)
Do it the RAILS WAY!
Then ... replace the ActionController::Renderer for JSON! Just add the following code to your ApplicationController:
ActionController::Renderers.add :json do |json, options|
unless json.kind_of?(String)
json = json.as_json(options) if json.respond_to?(:as_json)
json = JSON.pretty_generate(json, options)
end
if options[:callback].present?
self.content_type ||= Mime::JS
"#{options[:callback]}(#{json})"
else
self.content_type ||= Mime::JSON
json
end
end
Check out Awesome Print. Parse the JSON string into a Ruby Hash, then display it with ap like so:
require "awesome_print"
require "json"
json = '{"holy": ["nested", "json"], "batman!": {"a": 1, "b": 2}}'
ap(JSON.parse(json))
With the above, you'll see:
{
"holy" => [
[0] "nested",
[1] "json"
],
"batman!" => {
"a" => 1,
"b" => 2
}
}
Awesome Print will also add some color that Stack Overflow won't show you.
If you find that the pretty_generate option built into Ruby's JSON library is not "pretty" enough, I recommend my own NeatJSON gem for your formatting.
To use it:
gem install neatjson
and then use
JSON.neat_generate
instead of
JSON.pretty_generate
Like Ruby's pp it will keep objects and arrays on one line when they fit, but wrap to multiple as needed. For example:
{
"navigation.createroute.poi":[
{"text":"Lay in a course to the Hilton","params":{"poi":"Hilton"}},
{"text":"Take me to the airport","params":{"poi":"airport"}},
{"text":"Let's go to IHOP","params":{"poi":"IHOP"}},
{"text":"Show me how to get to The Med","params":{"poi":"The Med"}},
{"text":"Create a route to Arby's","params":{"poi":"Arby's"}},
{
"text":"Go to the Hilton by the Airport",
"params":{"poi":"Hilton","location":"Airport"}
},
{
"text":"Take me to the Fry's in Fresno",
"params":{"poi":"Fry's","location":"Fresno"}
}
],
"navigation.eta":[
{"text":"When will we get there?"},
{"text":"When will I arrive?"},
{"text":"What time will I get to the destination?"},
{"text":"What time will I reach the destination?"},
{"text":"What time will it be when I arrive?"}
]
}
It also supports a variety of formatting options to further customize your output. For example, how many spaces before/after colons? Before/after commas? Inside the brackets of arrays and objects? Do you want to sort the keys of your object? Do you want the colons to all be lined up?
Dumping an ActiveRecord object to JSON (in the Rails console):
pp User.first.as_json
# => {
"id" => 1,
"first_name" => "Polar",
"last_name" => "Bear"
}
Using <pre> HTML code and pretty_generate is good trick:
<%
require 'json'
hash = JSON[{hey: "test", num: [{one: 1, two: 2, threes: [{three: 3, tthree: 33}]}]}.to_json]
%>
<pre>
<%= JSON.pretty_generate(hash) %>
</pre>
Here is a middleware solution modified from this excellent answer by #gertas. This solution is not Rails specific--it should work with any Rack application.
The middleware technique used here, using #each, is explained at ASCIIcasts 151: Rack Middleware by Eifion Bedford.
This code goes in app/middleware/pretty_json_response.rb:
class PrettyJsonResponse
def initialize(app)
#app = app
end
def call(env)
#status, #headers, #response = #app.call(env)
[#status, #headers, self]
end
def each(&block)
#response.each do |body|
if #headers["Content-Type"] =~ /^application\/json/
body = pretty_print(body)
end
block.call(body)
end
end
private
def pretty_print(json)
obj = JSON.parse(json)
JSON.pretty_unparse(obj)
end
end
To turn it on, add this to config/environments/test.rb and config/environments/development.rb:
config.middleware.use "PrettyJsonResponse"
As #gertas warns in his version of this solution, avoid using it in production. It's somewhat slow.
Tested with Rails 4.1.6.
#At Controller
def branch
#data = Model.all
render json: JSON.pretty_generate(#data.as_json)
end
If you're looking to quickly implement this in a Rails controller action to send a JSON response:
def index
my_json = '{ "key": "value" }'
render json: JSON.pretty_generate( JSON.parse my_json )
end
Here's my solution which I derived from other posts during my own search.
This allows you to send the pp and jj output to a file as needed.
require "pp"
require "json"
class File
def pp(*objs)
objs.each {|obj|
PP.pp(obj, self)
}
objs.size <= 1 ? objs.first : objs
end
def jj(*objs)
objs.each {|obj|
obj = JSON.parse(obj.to_json)
self.puts JSON.pretty_generate(obj)
}
objs.size <= 1 ? objs.first : objs
end
end
test_object = { :name => { first: "Christopher", last: "Mullins" }, :grades => [ "English" => "B+", "Algebra" => "A+" ] }
test_json_object = JSON.parse(test_object.to_json)
File.open("log/object_dump.txt", "w") do |file|
file.pp(test_object)
end
File.open("log/json_dump.txt", "w") do |file|
file.jj(test_json_object)
end
I have used the gem CodeRay and it works pretty well. The format includes colors and it recognises a lot of different formats.
I have used it on a gem that can be used for debugging rails APIs and it works pretty well.
By the way, the gem is named 'api_explorer' (http://www.github.com/toptierlabs/api_explorer)
if you want to handle active_record object, puts is enough.
for example:
without puts
2.6.0 (main):0 > User.first.to_json
User Load (0.4ms) SELECT "users".* FROM "users" ORDER BY "users"."id" ASC LIMIT $1 [["LIMIT", 1]]
=> "{\"id\":1,\"admin\":true,\"email\":\"admin#gmail.com\",\"password_digest\":\"$2a$10$TQy3P7NT8KrdCzliNUsZzuhmo40LGKoth2hwD3OI.kD0lYiIEwB1y\",\"created_at\":\"2021-07-20T08:34:19.350Z\",\"updated_at\":\"2021-07-20T08:34:19.350Z\",\"name\":\"Arden Stark\"}"
with puts
2.6.0 (main):0 > puts User.first.to_json
User Load (0.3ms) SELECT "users".* FROM "users" ORDER BY "users"."id" ASC LIMIT $1 [["LIMIT", 1]]
{"id":1,"admin":true,"email":"admin#gmail.com","password_digest":"$2a$10$TQy3P7NT8KrdCzliNUsZzuhmo40LGKoth2hwD3OI.kD0lYiIEwB1y","created_at":"2021-07-20T08:34:19.350Z","updated_at":"2021-07-20T08:34:19.350Z","name":"Arden Stark"}
=> nil
if you are handle the json data, JSON.pretty_generate is a good alternative
Example:
obj = {foo: [:bar, :baz], bat: {bam: 0, bad: 1}}
json = JSON.pretty_generate(obj)
puts json
Output:
{
"foo": [
"bar",
"baz"
],
"bat": {
"bam": 0,
"bad": 1
}
}
if it's in the ROR project, I always prefer to use gem pry-rails to format my codes in the rails console rather than awesome_print which is too verbose.
Example of pry-rails:
it also has syntax highlight.
# example of use:
a_hash = {user_info: {type: "query_service", e_mail: "my#email.com", phone: "+79876543322"}, cars_makers: ["bmw", "mitsubishi"], car_models: [bmw: {model: "1er", year_mfc: 2006}, mitsubishi: {model: "pajero", year_mfc: 1997}]}
pretty_html = a_hash.pretty_html
# include this module to your libs:
module MyPrettyPrint
def pretty_html indent = 0
result = ""
if self.class == Hash
self.each do |key, value|
result += "#{key}: #{[Array, Hash].include?(value.class) ? value.pretty_html(indent+1) : value}"
end
elsif self.class == Array
result = "[#{self.join(', ')}]"
end
"#{result}"
end
end
class Hash
include MyPrettyPrint
end
class Array
include MyPrettyPrint
end
Simplest example, I could think of:
my_json = '{ "name":"John", "age":30, "car":null }'
puts JSON.pretty_generate(JSON.parse(my_json))
Rails console example:
core dev 1555:0> my_json = '{ "name":"John", "age":30, "car":null }'
=> "{ \"name\":\"John\", \"age\":30, \"car\":null }"
core dev 1556:0> puts JSON.pretty_generate(JSON.parse(my_json))
{
"name": "John",
"age": 30,
"car": null
}
=> nil
Pretty print variant (Rails):
my_obj = {
'array' => [1, 2, 3, { "sample" => "hash"}, 44455, 677778, nil ],
foo: "bar", rrr: {"pid": 63, "state with nil and \"nil\"": false},
wwww: 'w' * 74
}
require 'pp'
puts my_obj.as_json.pretty_inspect.
gsub('=>', ': ').
gsub(/"(?:[^"\\]|\\.)*"|\bnil\b/) {|m| m == 'nil' ? 'null' : m }.
gsub(/\s+$/, "")
Result:
{"array": [1, 2, 3, {"sample": "hash"}, 44455, 677778, null],
"foo": "bar",
"rrr": {"pid": 63, "state with nil and \"nil\"": false},
"wwww":
"wwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwww"}
If you are using RABL you can configure it as described here to use JSON.pretty_generate:
class PrettyJson
def self.dump(object)
JSON.pretty_generate(object, {:indent => " "})
end
end
Rabl.configure do |config|
...
config.json_engine = PrettyJson if Rails.env.development?
...
end
A problem with using JSON.pretty_generate is that JSON schema validators will no longer be happy with your datetime strings. You can fix those in your config/initializers/rabl_config.rb with:
ActiveSupport::TimeWithZone.class_eval do
alias_method :orig_to_s, :to_s
def to_s(format = :default)
format == :default ? iso8601 : orig_to_s(format)
end
end
I use the following as I find the headers, status and JSON output useful as
a set. The call routine is broken out on recommendation from a railscasts presentation at: http://railscasts.com/episodes/151-rack-middleware?autoplay=true
class LogJson
def initialize(app)
#app = app
end
def call(env)
dup._call(env)
end
def _call(env)
#status, #headers, #response = #app.call(env)
[#status, #headers, self]
end
def each(&block)
if #headers["Content-Type"] =~ /^application\/json/
obj = JSON.parse(#response.body)
pretty_str = JSON.pretty_unparse(obj)
#headers["Content-Length"] = Rack::Utils.bytesize(pretty_str).to_s
Rails.logger.info ("HTTP Headers: #{ #headers } ")
Rails.logger.info ("HTTP Status: #{ #status } ")
Rails.logger.info ("JSON Response: #{ pretty_str} ")
end
#response.each(&block)
end
end
I had a JSON object in the rails console, and wanted to display it nicely in the console (as opposed to displaying like a massive concatenated string), it was as simple as:
data.as_json