Is there a uWSGI option/hook/subsystem to fill a cache before an app is ready to process requests?
The static load-file-in-cache is not an option. I need to call a function or program which populates a uWSGI cache.
A prerequisite is that uWSGI actually has caching enabled, something like:
[uwsgi]
cache2 = name=mycache,items=100
There are many options for startup hooks. Take a look at this page in the uWSGI doc. You can choose at what point in the uWSGI startup sequence you want to hook up (as-root, as-user, pre-app, accepting1, ...) and you have the choice between shell exec (exec) and function calls (call).
The config directive may look something like:
[uwsgi]
call-as-root = cache_clear("mycache")
call-as-user = cache_set("key", "value")
See also Cache functions.
I believe that load-file-in-cache directive may also be able to handle your problem:
[uwsgi]
exec-as-user=/usr/local/bin/populate_my_cache_folder.sh
for-glob = /my/cache/folder/*.*
load-file-in-cache = files %(_)
endfor =
Related
Our project has a few GRPC servers defined as go_binary targets. We develop client SDKs for Java and Python applications and we would like to use java_test and py_test. Is any way to start a specific go_binary target before java_test or py_test?
You can create a test harness that starts the gRPC server before running the tests. For example, you could add the binary to the data attribute of the test, and then started it beforehand:
go_binary(
name = "my_grpc_server",
[...]
)
py_test(
name = "my_test",
[...]
data = [":my_grpc_server"],
)
and then inside the test file:
class ClientTestCase(unittest.TestCase):
def setUp(self):
r = runfiles.Create()
self.server = subprocess.Popen([r.Rlocation("path/to/my_grpc_server")])
def tearDown(self):
self.server.terminate()
self.server.wait()
This example is very simple, you'll probably run into issues regarding the availability of the port the server listens on, or waiting for the server to start up. You could add flags to your gRPC server to allow communication over a domain socket, or make it listen on an unused port and have the test parse the port number from the server's log output.
For details on finding the server with runfiles: https://github.com/bazelbuild/bazel/blob/a7a0d48fbeb059ee60e77580e5d05baeefdd5699/tools/python/runfiles/runfiles.py#L16-L58
If you find yourself copy-pasting this pattern a lot, or having to implement it in multiple languages, you could try using an sh_test() rule to wrap the underlying py_test or java_test, and to start the server, then start the test with an environment variable telling it how to reach the server (eg MY_GRPC_SERVER_ADDRESS=localhost:${test_port}.
I am trying to use the Golang SDK of Docker in order to maintain a slice variable with currently running containers on the local Docker instance. This slice is exported from a package and I want to use it to feed a web page.
I am not really used to goroutines and channels and that's why I am wondering if I have spotted a good solution for my problem.
I have a docker package as follows.
https://play.golang.org/p/eMmqkMezXZn
It has a Running variable containing the current state of running containers.
var Running []types.Container
I use a reload function to load the running containers in the Running variable.
// Reload the list of running containers
func reload() error {
...
Running, err = cli.ContainerList(context.Background(), types.ContainerListOptions{
All: false,
})
...
}
And then I start a goroutine from the init function to listen to Docker events and trigger the reload function accordingly.
func init() {
...
// Listen for docker events
go listen()
...
}
// Listen for docker events
func listen() {
filter := filters.NewArgs()
filter.Add("type", "container")
filter.Add("event", "start")
filter.Add("event", "die")
msg, errChan := cli.Events(context.Background(), types.EventsOptions{
Filters: filter,
})
for {
select {
case err := <-errChan:
panic(err)
case <-msg:
fmt.Println("reloading")
reload()
}
}
}
My question is, is it proper to update a variable from inside a goroutine (in terms of sync)? Maybe there is a cleaner way to achieve what I am trying to build?
Update
My concern here is not really about caching. It is more about hiding the "complexity" of the process of listening and update from the Docker SDK. I wanted to provide something like an index to easily let the end user loop and display currently running containers.
I was aware of data-races problems in threaded programs but I did not realize I was as actually in a context of concurrence here (I never wrote concurrent programs in Go before).
I effectively need to re-think the solution to be more idiomatic. As far as I can see, I have two options here: either protecting the variable with a mutex or re-thinking the design to integrate channels.
What means the most to me is to hide or encapsulate the method of synchronization used so the package users need not concern of how the shared state is protected.
Would you have any recommendations?
Thanks a lot for your help,
Loric
No, it is not idiomatic Go to share the Running variable between two goroutines. You do this by sharing it between the routine that runs your main function, and the listen function which is started with go—which spawns another goroutine.
Why, is because it breaks with
Do not communicate by sharing memory; instead, share memory by
communicating. ¹
So the design of the API needs to change in order to be idiomatic; you need to remove the Running variable and replace it with what? It depends on what you are trying to achieve. If you are trying to cache the cli.ContainerList because you need to call it often, and it might be expensive, you should implement a cache which is invalidated on each cli.Events.
What is your motivation?
I need to watch or listen to a folder on an SFTP server for any changes. At some given time in the future (I don't know when), the folder will be updated with a file. Instead of pinging every minute, how would I setup a listener or watcher on that folder so I know when it has that file? Does Erlang's ssh_sftp module provide a function for this?
Neither SFTP nor FTP protocol have any mechanism to notify a client about changes in a remote folder. The only solution to detect changes is to periodically enumerate remote directory tree and find differences.
Ref : https://winscp.net/eng/docs/library_example_watch_for_changes
I don't know whether you still need it or not but I have already implemented what you need. I won't be able to share the code due to legal agreements however I can hint you the way to implement it.
Requirement: Run a function whenever a file lands to a directory instead of polling the directory at a defined interval to make processing efficient.
Step 1: Start the ssh daemon process with sftp as a subsystem
{ok, Pid} = ssh:daemon(Port, [{user_passwords, [{User, Pass}]},
{system_dir, SystemDir},
{user_dir, UserDir},
{key_cb, cac_auth},
{shell, {cacsftpd_server, display_info, []}},
{subsystems, [ssh_sftpd:subsystem_spec([{cwd,
Home}, {file_handler, sftpd_file_handler}])]}])
Step 2: In sftpd_file_handler you can apply the trigger to call ur desired handler once the file is completely transferred ;)).
While not in Erlang, but rather Elixir, take a look at: https://github.com/Codenaut/exsftpd/blob/master/lib/sftpd_file_handler.ex
The essential bit is the write function:
defmodule Exsftpd.SftpFileHandler do
...
def write(io_device, data, state) do
{:file.write(io_device, data), state}
my_custom_function(state)
{:ok, state}
end
...
end
Here you can call whatever function you desire after (or instead of) writing the content to the filesystem.
Initialise the server like so: (see https://github.com/Codenaut/exsftpd/blob/master/lib/server.ex for a more elaborate example):
:ssh.daemon(2222,
system_dir: '/tmp/ssh',
subsystems: [
Exsftpd.SftpdChannel.subsystem_spec(
file_handler: {Exsftpd.SftpFileHandler, []}
)
],
user_dir: '/tmp/ssh/users'
)
I am trying out Node.js Express framework, and looking for plugin that allows me to interact with my models via console, similar to Rails console. Is there such a thing in NodeJS world?
If not, how can I interact with my Node.js models and data, such as manually add/remove objects, test methods on data etc.?
Create your own REPL by making a js file (ie: console.js) with the following lines/components:
Require node's built-in repl: var repl = require("repl");
Load in all your key variables like db, any libraries you swear by, etc.
Load the repl by using var replServer = repl.start({});
Attach the repl to your key variables with replServer.context.<your_variable_names_here> = <your_variable_names_here>. This makes the variable available/usable in the REPL (node console).
For example: If you have the following line in your node app:
var db = require('./models/db')
Add the following lines to your console.js
var db = require('./models/db');
replServer.context.db = db;
Run your console with the command node console.js
Your console.js file should look something like this:
var repl = require("repl");
var epa = require("epa");
var db = require("db");
// connect to database
db.connect(epa.mongo, function(err){
if (err){ throw err; }
// open the repl session
var replServer = repl.start({});
// attach modules to the repl context
replServer.context.epa = epa;
replServer.context.db = db;
});
You can even customize your prompt like this:
var replServer = repl.start({
prompt: "Node Console > ",
});
For the full setup and more details, check out:
http://derickbailey.com/2014/07/02/build-your-own-app-specific-repl-for-your-nodejs-app/
For the full list of options you can pass the repl like prompt, color, etc: https://nodejs.org/api/repl.html#repl_repl_start_options
Thank you to Derick Bailey for this info.
UPDATE:
GavinBelson has a great recommendation for running with sequelize ORM (or anything that requires promise handling in the repl).
I am now running sequelize as well, and for my node console I'm adding the --experimental-repl-await flag.
It's a lot to type in every time, so I highly suggest adding:
"console": "node --experimental-repl-await ./console.js"
to the scripts section in your package.json so you can just run:
npm run console
and not have to type the whole thing out.
Then you can handle promises without getting errors, like this:
const product = await Product.findOne({ where: { id: 1 });
I am not very experienced in using node, but you can enter node in the command line to get to the node console. I then used to require the models manually
Here is the way to do it, with SQL databases:
Install and use Sequelize, it is Node's ORM answer to Active Record in Rails. It even has a CLI for scaffolding models and migrations.
node --experimental-repl-await
> models = require('./models');
> User = models.User; //however you load the model in your actual app this may vary
> await User.findAll(); //use await, then any sequelize calls here
TLDR
This gives you access to all of the models just as you would in Rails active record. Sequelize takes a bit of getting used to, but in many ways it is actually more flexible than Active Record while still having the same features.
Sequelize uses promises, so to run these properly in REPL you will want to use the --experimental-repl-await flag when running node. Otherwise, you can get bluebird promise errors
If you don't want to type out the require('./models') step, you can use console.js - a setup file for REPL at the root of your directory - to preload this. However, I find it easier to just type this one line out in REPL
It's simple: add a REPL to your program
This may not fully answer your question, but to clarify, node.js is much lower-level than Rails, and as such doesn't prescribe tools and data models like Rails. It's more of a platform than a framework.
If you are looking for a more Rails-like experience, you may want to look at a more 'full-featured' framework built on top of node.js, such as Meteor, etc.
(nodeA#foo.hyd.com)8> spawn(nodeA#bar.del.com, tut, test, [hello, 5]).
I want to spawn a process on bar.del.com which has no file system access to foo.hyd.com (from where I am spawning the process), running subroutine "test" of module "tut".
Is there a way to do so, w/o providing the nodeA#bar.del.com with the compiled "tut" module file?
You can use the following function to load a module at remote node without providing the file itself:
load_module(Node, Module) ->
{_Module, Bin, Filename} = code:get_object_code(Module),
rpc:call(Node, code, load_binary, [Module, Filename, Bin]).
As noted in code:load_binary/3 Filename argument is used only to track the path to module and the file it points to is not used by local node_server.
I'm interpreting your question as a desire to not copy the *.beams from your filesystem to the remote file system.
If you are just testing things you can use the nl(Mod) call in the erl shell to load the module on all (currently) known nodes. That is, those that show up in nodes().
This will send the code over and load it from the memory copy, it will not store it in the remote filesystem.
You can also start the remote node using the slave module. A slave accesses its master's filesystem and code server. Ordinary auto-loading will then make sure the module exists in the slave when you call your test:tut/2 function.
You can send the local code to the remote node:
> {Mod, Bin, File} = code:get_object_code(Module).
> rpc:call(RemoteNode, code, load_binary, [Mod, File, Bin]).