I'd like to perform an atomic GET in Redis, and if the value returned is equal to some expected value, I'd like to do a SET, but I want to chain all of this together as one atomic operation. (I'm trying to set a flag that indicates whether any process is writing data to disk, as only one process may be permitted to do so.)
Is it possible to accomplish this with Redis?
I have seen documentation on MULTI operations but I haven't seen conditional operations i MULTI operations. Any suggestions others can offer with this would be greatly appreciated!
You can do both the GET and set operations on the redis server itself using Lua scripts. They're atomic and allow you to add logic too.
I ended up using redlock-py, an implementation of the redlock algorithm that the Redis docs recommend for creating write locks: https://redis.io/topics/distlock. The linked article is fantastic reading for anyone looking to create similar write locks in Redis.
redis-if - lua script for "conditional transactions". More convenient than WATCH + MULTY.
You can pass any combination of conditions & followed commands as json object:
const Redis = require('ioredis')
const redis = new Redis()
redis.defineCommand('transaction', { lua: require('redis-if').script, numberOfKeys: 0 })
await redis.set('custom-state', 'initialized')
await redis.set('custom-counter', 0)
// this call will change state and do another unrelated operation (increment) atomically
let success = await redis.transaction(JSON.stringify({
if: [
// apply changes only if this process has acquired a lock
[ 'initialized', '==', [ 'sget', 'custom-state' ] ]
],
exec: [
[ 'set', 'custom-state', 'finished' ],
[ 'incr', 'custom-counter' ]
]
}))
With this script we removed all custom scripting from our projects.
I came across this post looking for a similar type of function, but I didn't see any options that appealed to me. I opted instead to write a small module in Rust that provides this exact type of operation:
https://github.com/KennethWilke/redis-setif
With this module you would do this via:
SETIF <key> <expected> <new>
HSETIF <key> <field> <expected> <new>
You can do this by SET command, with these 2 arguments, which according to the docs here:
GET - return the old string stored at key, or nil if key did not
exist.
NX - Only set the key if it does not already exist.
Since Redis doesn't execute any command while another command is running - you have the 2 operations in an atomic manner.
Related
Using a TpFIBTransaction component, I'm trying to start a READ UNCOMMITTED transaction.
First of all, the TPBMode property has 3 possible values:
tpbDefault
tpbReadCommitted
tpbRepeatableRead
In TpFIBTransaction.StartTransaction I saw that setting tpbReadCommitted forces the following parameters:
write
isc_tpb_nowait
read_committed
rec_version
Using tpbRepeatableRead forces the following parameters instead:
write
isc_tpb_nowait
concurrency
So, it seems the only way to have "custom" transaction parameters is to set the tpbDefault value.
The values allowed for the TrParams property are the following (from fib.pas unit)
TPBConstantNames: array[1..isc_tpb_last_tpb_constant] of String = (
'consistency',
'concurrency',
'shared',
'protected',
'exclusive',
'wait',
'nowait',
'read',
'write',
'lock_read',
'lock_write',
'verb_time',
'commit_time',
'ignore_limbo',
'read_committed',
'autocommit',
'rec_version',
'no_rec_version',
'restart_requests',
'no_auto_undo',
'no_savepoint'
);
I've tried adding the 'read' value only, but it seems it's still unable to read uncommitted data, even if there's not 'read_committed' in TrParams property.
MyTransaction.TrParams.Clear();
MyTransaction.TrParams.Add('read');
Is there some missing value in TPBConstantNames (Something like 'read_uncommitted', if it exists...), or is there another way to setup a Firebird "read uncommitted" transaction?
It is not possible because Firebird does not support read uncomitted isolation level.
You can find the following information in the documentation documentation:
Note
The READ UNCOMMITTED isolation level is a synonym for READ
COMMITTED, and provided only for syntax compatibility. It provides the
exact same semantics as READ COMMITTED, and does not allow you to view
uncommitted changes of other transactions.
and:
The three isolation levels supported in Firebird are:
SNAPSHOT
SNAPSHOT TABLE STABILITY
READ COMMITTED with two specifications (NO RECORD_VERSION and
RECORD_VERSION)
So I'm using puppet3 and I have X.yaml and Y.yaml. X.yaml has profiles::resolv_conf::nameservers: [ '1.1.1.1', '8.8.8.8', '2.2.2.2' ]in it. I want to add that [ '1.1.1.1', '8.8.8.8', '2.2.2.2' ] as a value to the servers: which is in Y.yaml:
'dns_test':
plugin_type: 'dns_query'
options:
'servers': \['1.1.1.1', '8.8.8.8', "2.2.2.2"\]
'domains': \['google.com'\]
'record_type': 'A'
'timeout': 5
tags:
'input_source': 'dns_query'
By doing this I want to make sure that when someone change values in profiles::resolv_conf::nameservers: that value is changed in this telegraf plugin too.
I tried multiple solution but the one that was the closest was:
'dns_test':
plugin_type: 'dns_query'
options:
'servers': "%{hiera('profiles::resolv_conf::nameservers')}"
'domains': ['google.com']
'record_type': 'A'
'timeout': 5
tags: 'input_source': 'dns_query'
but problem is that puppet was adding extra " " to the value and final value in plugin conf was:
"["1.1.1.1", "2.2.2.2", "8.8.8.8"]" instead of ["1.1.1.1", "2.2.2.2", "8.8.8.8"]
TL;DR: You can't.
From the current docs and the Puppet documentation archive, I confirm that no version of the %{hiera} interpolation function or its replacement, %{lookup}, ever supported interpolating values other than strings. That's expressed in the current docs like so:
The lookup and hiera interpolation functions look up a key and return
the resulting value. The result of the lookup must be a string; any
other result causes an error.
(Emphasis added)
What you're looking for would be supported by Hiera 5's %{alias} function, provided that the data are available somewhere else in the same hierarchy (which is also a requirement for %{hiera}). Since you're stuck on Puppet 3, however, you're probably on Hiera 2, and certainly not later than Hiera 3.
"But wait!" You may say. "I'm getting a successful interpolation, but the data are just munged". Specifically, you wrote:
problem is that puppet was adding extra " " to the value and final value
Since %{hiera()} interpolates only strings, it is not surprising that you got a string value, given that you got a value at all. I do find it a bit surprising that Puppet did not throw an error, but I'm not prepared to comment further on that without a minimum reproducible example that demonstrates the behavior.
In writing my tests, I'd like to be able to inject a connection into the request so that I can wrap the entire test case in a transaction (even if there is more than one request in the test case).
I've attempted to do this using a BeforeMiddleware which I can link in my test cases to insert a connection, as such:
pub type DatabaseConnection = PooledConnection<ConnectionManager<PgConnection>>;
pub struct DatabaseOverride {
conn: DatabaseConnection,
}
impl BeforeMiddleware for DatabaseOverride {
fn before(&self, req: &mut Request) -> IronResult<()> {
req.extensions_mut().entry::<DatabaseOverride>().or_insert(self.conn);
Ok(())
}
}
However, I'm encountering a compile error in trying to do this:
error: the trait bound `std::rc::Rc<diesel::pg::connection::raw::RawConnection>: std::marker::Sync` is not satisfied [E0277]
impl BeforeMiddleware for DatabaseOverride {
^~~~~~~~~~~~~~~~
help: run `rustc --explain E0277` to see a detailed explanation
note: `std::rc::Rc<diesel::pg::connection::raw::RawConnection>` cannot be shared between threads safely
note: required because it appears within the type `diesel::pg::PgConnection`
note: required because it appears within the type `r2d2::Conn<diesel::pg::PgConnection>`
note: required because it appears within the type `std::option::Option<r2d2::Conn<diesel::pg::PgConnection>>`
note: required because it appears within the type `r2d2::PooledConnection<r2d2_diesel::ConnectionManager<diesel::pg::PgConnection>>`
note: required because it appears within the type `utility::db::DatabaseOverride`
note: required by `iron::BeforeMiddleware`
error: the trait bound `std::cell::Cell<i32>: std::marker::Sync` is not satisfied [E0277]
impl BeforeMiddleware for DatabaseOverride {
^~~~~~~~~~~~~~~~
help: run `rustc --explain E0277` to see a detailed explanation
note: `std::cell::Cell<i32>` cannot be shared between threads safely
note: required because it appears within the type `diesel::pg::PgConnection`
note: required because it appears within the type `r2d2::Conn<diesel::pg::PgConnection>`
note: required because it appears within the type `std::option::Option<r2d2::Conn<diesel::pg::PgConnection>>`
note: required because it appears within the type `r2d2::PooledConnection<r2d2_diesel::ConnectionManager<diesel::pg::PgConnection>>`
note: required because it appears within the type `utility::db::DatabaseOverride`
note: required by `iron::BeforeMiddleware`
Is there a way around this with diesel's connections? I've found several examples on Github to do this using the pg crate, but I'd like to keep using diesel.
This answer will certainly solve the problem, but it's not optimal. As mentioned, you can't share a single connection as it's not thread safe. However, while wrapping it in a Mutex makes it thread-safe, it would force all the server threads to use a single connection. Instead, you want to use a connection pool.
You can accomplish this with the r2d2 and r2d2-diesel crates. This will establish multiple connections as needed, and reuse them when possible in a thread safe manner.
Since there isn't enough code provided for me to reproduce your issue, I've made this:
use std::cell::Cell;
trait Middleware: Sync {}
struct Unsharable(Cell<bool>);
impl Middleware for Unsharable {}
fn main() {}
which has the same error:
error: the trait bound `std::cell::Cell<bool>: std::marker::Sync` is not satisfied [E0277]
impl Middleware for Unsharable {}
^~~~~~~~~~
help: run `rustc --explain E0277` to see a detailed explanation
note: `std::cell::Cell<bool>` cannot be shared between threads safely
note: required because it appears within the type `Unsharable`
note: required by `Middleware`
You can solve the problem by changing the type to make it cross-thread compatible:
use std::sync::Mutex;
struct Sharable(Mutex<Unsharable>);
impl Middleware for Sharable {}
Note that Rust has done a very good thing for you: it prevented you from using a type that is unsafe to be called in multiple threads.
In writing my tests, I'd like to be able to inject a connection into the request so that I can wrap the entire test case in a transaction (even if there is more than one request in the test case).
I'd suggest that it's possible an architectural change would be even better. Separate the domains of "web framework" from your "database". The authors of Growing Object-Oriented Software, Guided by Tests (a highly recommended book) advocate for this style.
Pull apart your code such that there is a method that simply accepts some type that can start / end a transaction, write the interesting stuff there, and test it thoroughly. Then have just enough glue code in the web layer to create a transaction object, then call the next layer down.
In the Tinkerpop or Titan documentation, all operations are based on a sample graph. How to creat a new empty graph to work on?
I am programming in erlang connecting to Tinkergraph, planned to use Titan later in production. There is no erlang driver for both so I am connecting by REST. It is easy to read from graph, but if I want to read from user's input then write into the graph, for example, to create a person named teddy:
screenshot 1
I got those errors. What is the correct way?
Thank you.
Update: For following situation:
23> Newperson=terry.
terry
24> Newperson.
terry
If I want to add this terry, below two will not work. What's the correct way to do it?
screenshot 2
1
TitanGraph titanGraph = TitanFactory.open(config); will open a titan graph without the sample data.
If you have already commited the sample data to your keyspace then you can just change the keyspace defined in your config file.
For example if you are using a cassandra backend you would change storage.cassandra.keyspace=xxxxxx .
You can also clear any keyspace using TitanCleanup.clear(graph);
2
As for the error you are seeing. It looks like you are trying to label your vertex incorrectly. I posted the following and it worked:
{
"gremlin" : "g.addV(label, x).property(y,z)",
"bindings" :
{
"x" : "person",
"y" : "name",
"z" : "Teddy"
}
}
A final note, when you start using Titan 1.0.0 make sure you checkout this section of the tinkerpop docs. Especially make sure to change the channel in the gremlin-server.yaml config to:
channelizer: com.tinkerpop.gremlin.server.channel.HttpChannelizer
Answer to my own question: construct a Body by lists:concat() or ++, then post
I am trying out Node.js Express framework, and looking for plugin that allows me to interact with my models via console, similar to Rails console. Is there such a thing in NodeJS world?
If not, how can I interact with my Node.js models and data, such as manually add/remove objects, test methods on data etc.?
Create your own REPL by making a js file (ie: console.js) with the following lines/components:
Require node's built-in repl: var repl = require("repl");
Load in all your key variables like db, any libraries you swear by, etc.
Load the repl by using var replServer = repl.start({});
Attach the repl to your key variables with replServer.context.<your_variable_names_here> = <your_variable_names_here>. This makes the variable available/usable in the REPL (node console).
For example: If you have the following line in your node app:
var db = require('./models/db')
Add the following lines to your console.js
var db = require('./models/db');
replServer.context.db = db;
Run your console with the command node console.js
Your console.js file should look something like this:
var repl = require("repl");
var epa = require("epa");
var db = require("db");
// connect to database
db.connect(epa.mongo, function(err){
if (err){ throw err; }
// open the repl session
var replServer = repl.start({});
// attach modules to the repl context
replServer.context.epa = epa;
replServer.context.db = db;
});
You can even customize your prompt like this:
var replServer = repl.start({
prompt: "Node Console > ",
});
For the full setup and more details, check out:
http://derickbailey.com/2014/07/02/build-your-own-app-specific-repl-for-your-nodejs-app/
For the full list of options you can pass the repl like prompt, color, etc: https://nodejs.org/api/repl.html#repl_repl_start_options
Thank you to Derick Bailey for this info.
UPDATE:
GavinBelson has a great recommendation for running with sequelize ORM (or anything that requires promise handling in the repl).
I am now running sequelize as well, and for my node console I'm adding the --experimental-repl-await flag.
It's a lot to type in every time, so I highly suggest adding:
"console": "node --experimental-repl-await ./console.js"
to the scripts section in your package.json so you can just run:
npm run console
and not have to type the whole thing out.
Then you can handle promises without getting errors, like this:
const product = await Product.findOne({ where: { id: 1 });
I am not very experienced in using node, but you can enter node in the command line to get to the node console. I then used to require the models manually
Here is the way to do it, with SQL databases:
Install and use Sequelize, it is Node's ORM answer to Active Record in Rails. It even has a CLI for scaffolding models and migrations.
node --experimental-repl-await
> models = require('./models');
> User = models.User; //however you load the model in your actual app this may vary
> await User.findAll(); //use await, then any sequelize calls here
TLDR
This gives you access to all of the models just as you would in Rails active record. Sequelize takes a bit of getting used to, but in many ways it is actually more flexible than Active Record while still having the same features.
Sequelize uses promises, so to run these properly in REPL you will want to use the --experimental-repl-await flag when running node. Otherwise, you can get bluebird promise errors
If you don't want to type out the require('./models') step, you can use console.js - a setup file for REPL at the root of your directory - to preload this. However, I find it easier to just type this one line out in REPL
It's simple: add a REPL to your program
This may not fully answer your question, but to clarify, node.js is much lower-level than Rails, and as such doesn't prescribe tools and data models like Rails. It's more of a platform than a framework.
If you are looking for a more Rails-like experience, you may want to look at a more 'full-featured' framework built on top of node.js, such as Meteor, etc.