Configure composite filter for specific logger in Log4j2 via properties file - log4j2

Log4j2 provides various kind of filters [1] and these can be configured at four levels:
Context-wide
Logger
Appender
Appender Reference
In my use-case, I would like to setup a CompositeFilter at a specified Logger (level 2 from above) using the property file syntax.
For the sake of the example let's assume that we want to setup a composite filter composed from two RegexFilter matching .*SQL.* and .*JPQL.* respectively. Assuming that we have the following events:
logger.info("Messase with no keyword");
logger.info("Messase with SQL keyword");
logger.info("Messase with JPQL keyword");
logger.info("Messase with both SQL and JPQL keywords");
Only the last event should pass from the composite filter.
I made several attempts [2] (also outlined below) to get the desired output but I wasn't able to figure out how to properly setup a CompositeFilter using the property syntax so any help would be greatly appreciated.
Attempt 0
logger.t1.filter.f1.type = RegexFilter
logger.t1.filter.f1.regex = .*SQL.*
logger.t1.filter.f1.onMatch = NEUTRAL
logger.t1.filter.f1.onMismatch = DENY
logger.t1.filter.f2.type = RegexFilter
logger.t1.filter.f2.regex = .*JPQL.*
logger.t1.filter.f2.onMatch = ACCEPT
logger.t1.filter.f2.onMismatch = DENY
logger.t1.appenderRefs = a
logger.t1.appenderRef.a.ref = L0
Attempt 1
logger.t1.filters = f1, f2
logger.t1.filter.f1.type = RegexFilter
logger.t1.filter.f1.regex = .*SQL.*
logger.t1.filter.f1.onMatch = NEUTRAL
logger.t1.filter.f1.onMismatch = DENY
logger.t1.filter.f2.type = RegexFilter
logger.t1.filter.f2.regex = .*JPQL.*
logger.t1.filter.f2.onMatch = ACCEPT
logger.t1.filter.f2.onMismatch = DENY
Attempt 2
logger.t1.filters.f1.type = RegexFilter
logger.t1.filters.f1.regex = .*SQL.*
logger.t1.filters.f1.onMatch = NEUTRAL
logger.t1.filters.f1.onMismatch = DENY
logger.t1.filters.f2.type = RegexFilter
logger.t1.filters.f2.regex = .*JPQL.*
logger.t1.filters.f2.onMatch = ACCEPT
logger.t1.filters.f2.onMismatch = DENY
logger.t1.appenderRefs = a
logger.t1.appenderRef.a.ref = L0
Attempt 3
logger.t1.filter.M.type = Filters
logger.t1.filter.M.f1.type = RegexFilter
logger.t1.filter.M.f1.regex = .*SQL.*
logger.t1.filter.M.f1.onMatch = NEUTRAL
logger.t1.filter.M.f1.onMismatch = DENY
logger.t1.filter.M.f2.type = RegexFilter
logger.t1.filter.M.f2.regex = .*JPQL.*
logger.t1.filter.M.f2.onMatch = ACCEPT
logger.t1.filter.M.f2.onMismatch = DENY
logger.t1.appenderRefs = a
logger.t1.appenderRef.a.ref = L0
Some further clarifications:
I am strictly interested for setting up a CompositeFilter, not
write a custom filter, not use a single combined RegexFilter etc.
I am interested to see how it is done using the property syntax, not
XML, JSON, or other.
[1] https://logging.apache.org/log4j/2.x/manual/filters.html
[2] https://github.com/zabetak/logging-log4j2/commit/a9944885100db42f7e3ba1b3ff81d59b743d0ab7

There are two problems with your configuration:
A LoggerConfig accepts only a single filter. If you want to use multiple filters, configure a CompositeFilter explicitly (which has a plugin name "Filters),
The order of filters in a properties configuration file is basically random, since Properties is a Hashtable and does not have a predefined order.
Considering the two facts above a variation of your attempt 3 should work:
logger.t1.filter.M.type = Filters
logger.t1.filter.M.f1.type = RegexFilter
logger.t1.filter.M.f1.regex = .*SQL.*
logger.t1.filter.M.f1.onMatch = NEUTRAL
logger.t1.filter.M.f1.onMismatch = DENY
logger.t1.filter.M.f2.type = RegexFilter
logger.t1.filter.M.f2.regex = .*JPQL.*
logger.t1.filter.M.f2.onMatch = NEUTRAL
logger.t1.filter.M.f2.onMismatch = DENY
Both filters return NEUTRAL, when the pattern matches, which guarantees that an inversion of the order of filters does not break the logic.

Related

Why does all my emission mu of HMM in pyro converge to the same number?

I'm trying to create a Gaussian HMM model in pyro to infer the parameters of a very simple Markov sequence. However, my model fails to infer the parameters and something wired happened during the training process. Using the same sequence, hmmlearn has successfully infer the true parameters.
Full code can be accessed in here:
https://colab.research.google.com/drive/1u_4J-dg9Y1CDLwByJ6FL4oMWMFUVnVNd#scrollTo=ZJ4PzdTUBgJi
My model is modified from the example in here:
https://github.com/pyro-ppl/pyro/blob/dev/examples/hmm.py
I manually created a first order Markov sequence where there are 3 states, the true means are [-10, 0, 10], sigmas are [1,2,1].
Here is my model
def model(observations, num_state):
assert not torch._C._get_tracing_state()
with poutine.mask(mask = True):
p_transition = pyro.sample("p_transition",
dist.Dirichlet((1 / num_state) * torch.ones(num_state, num_state)).to_event(1))
p_init = pyro.sample("p_init",
dist.Dirichlet((1 / num_state) * torch.ones(num_state)))
p_mu = pyro.param(name = "p_mu",
init_tensor = torch.randn(num_state),
constraint = constraints.real)
p_tau = pyro.param(name = "p_tau",
init_tensor = torch.ones(num_state),
constraint = constraints.positive)
current_state = pyro.sample("x_0",
dist.Categorical(p_init),
infer = {"enumerate" : "parallel"})
for t in pyro.markov(range(1, len(observations))):
current_state = pyro.sample("x_{}".format(t),
dist.Categorical(Vindex(p_transition)[current_state, :]),
infer = {"enumerate" : "parallel"})
pyro.sample("y_{}".format(t),
dist.Normal(Vindex(p_mu)[current_state], Vindex(p_tau)[current_state]),
obs = observations[t])
My model is compiled as
device = torch.device("cuda:0")
obs = torch.tensor(obs)
obs = obs.to(device)
torch.set_default_tensor_type("torch.cuda.FloatTensor")
guide = AutoDelta(poutine.block(model, expose_fn = lambda msg : msg["name"].startswith("p_")))
Elbo = Trace_ELBO
elbo = Elbo(max_plate_nesting = 1)
optim = Adam({"lr": 0.001})
svi = SVI(model, guide, optim, elbo)
As the training goes, the ELBO has decreased steadily as shown. However, the three means of the states converges.
I have tried to put the for loop of my model into a pyro.plate and switch pyro.param to pyro.sample and vice versa, but nothing worked for my model.
I have not tried this model, but I think it should be possible to solve the problem by modifying the model in the following way:
def model(observations, num_state):
assert not torch._C._get_tracing_state()
with poutine.mask(mask = True):
p_transition = pyro.sample("p_transition",
dist.Dirichlet((1 / num_state) * torch.ones(num_state, num_state)).to_event(1))
p_init = pyro.sample("p_init",
dist.Dirichlet((1 / num_state) * torch.ones(num_state)))
p_mu = pyro.sample("p_mu",
dist.Normal(torch.zeros(num_state), torch.ones(num_state)).to_event(1))
p_tau = pyro.sample("p_tau",
dist.HalfCauchy(torch.zeros(num_state)).to_event(1))
current_state = pyro.sample("x_0",
dist.Categorical(p_init),
infer = {"enumerate" : "parallel"})
for t in pyro.markov(range(1, len(observations))):
current_state = pyro.sample("x_{}".format(t),
dist.Categorical(Vindex(p_transition)[current_state, :]),
infer = {"enumerate" : "parallel"})
pyro.sample("y_{}".format(t),
dist.Normal(Vindex(p_mu)[current_state], Vindex(p_tau)[current_state]),
obs = observations[t])
The model would then be trained using MCMC:
# MCMC
hmc_kernel = NUTS(model, target_accept_prob = 0.9, max_tree_depth = 7)
mcmc = MCMC(hmc_kernel, num_samples = 1000, warmup_steps = 100, num_chains = 1)
mcmc.run(obs)
The results could then be analysed using:
mcmc.get_samples()

How to load Seurat Object into WGCNA Tutorial Format

As far as I can find, there is only one tutorial about loading Seurat objects into WGCNA (https://ucdavis-bioinformatics-training.github.io/2019-single-cell-RNA-sequencing-Workshop-UCD_UCSF/scrnaseq_analysis/scRNA_Workshop-PART6.html). I am really new to programming so it's probably just my inexperience, but I am not sure how to load my Seurat object into a format that works with WGCNA's tutorials (https://horvath.genetics.ucla.edu/html/CoexpressionNetwork/Rpackages/WGCNA/Tutorials/).
Here is what I have tried thus far:
This tries to replicate datExpr and datTraits from part I.1:
library(WGCNA)
library(Seurat)
#example Seurat object -----------------------------------------------
ERlist <- list(c("CPB1", "RP11-53O19.1", "TFF1", "MB", "ANKRD30B",
"LINC00173", "DSCAM-AS1", "IGHG1", "SERPINA5", "ESR1",
"ILRP2", "IGLC3", "CA12", "RP11-64B16.2", "SLC7A2",
"AFF3", "IGFBP4", "GSTM3", "ANKRD30A", "GSTT1", "GSTM1",
"AC026806.2", "C19ORF33", "STC2", "HSPB8", "RPL29P11",
"FBP1", "AGR3", "TCEAL1", "CYP4B1", "SYT1", "COX6C",
"MT1E", "SYTL2", "THSD4", "IFI6", "K1AA1467", "SLC39A6",
"ABCD3", "SERPINA3", "DEGS2", "ERLIN2", "HEBP1", "BCL2",
"TCEAL3", "PPT1", "SLC7A8", "RP11-96D1.10", "H4C8",
"PI15", "PLPP5", "PLAAT4", "GALNT6", "IL6ST", "MYC",
"BST2", "RP11-658F2.8", "MRPS30", "MAPT", "AMFR", "TCEAL4",
"MED13L", "ISG15", "NDUFC2", "TIMP3", "RP13-39P12.3", "PARD68"))
tnbclist <- list(c("FABP7", "TSPAN8", "CYP4Z1", "HOXA10", "CLDN1",
"TMSB15A", "C10ORF10", "TRPV6", "HOXA9", "ATP13A4",
"GLYATL2", "RP11-48O20.4", "DYRK3", "MUCL1", "ID4", "FGFR2",
"SHOX2", "Z83851.1", "CD82", "COL6A1", "KRT23", "GCHFR",
"PRICKLE1", "GCNT2", "KHDRBS3", "SIPA1L2", "LMO4", "TFAP2B",
"SLC43A3", "FURIN", "ELF5", "C1ORF116", "ADD3", "EFNA3",
"EFCAB4A", "LTF", "LRRC31", "ARL4C", "GPNMB", "VIM",
"SDR16C5", "RHOV", "PXDC1", "MALL", "YAP1", "A2ML1",
"RP1-257A7.5", "RP11-353N4.6", "ZBTB18", "CTD-2314B22.3", "GALNT3",
"BCL11A", "CXADR", "SSFA2", "ADM", "GUCY1A3", "GSTP1",
"ADCK3", "SLC25A37", "SFRP1", "PRNP", "DEGS1", "RP11-110G21.2",
"AL589743.1", "ATF3", "SIVA1", "TACSTD2", "HEBP2"))
genes = c(unlist(c(ERlist,tnbclist)))
mat = matrix(rnbinom(500*length(genes),mu=500,size=1),ncol=500)
rownames(mat) = genes
colnames(mat) = paste0("cell",1:500)
sobj = CreateSeuratObject(mat)
sobj = NormalizeData(sobj)
sobj$ClusterName = factor(sample(0:1,ncol(sobj),replace=TRUE))
sobj$Patient = paste0("Patient", 1:500)
sobj = AddModuleScore(object = sobj, features = tnbclist,
name = "TNBC_List",ctrl=5)
sobj = AddModuleScore(object = sobj, features = ERlist,
name = "ER_List",ctrl=5)
#WGCNA -----------------------------------------------------------------
sobjwgcna <- sobj
sobjwgcna <- FindVariableFeatures(sobjwgcna, selection.method = "vst", nfeatures = 2000,
verbose = FALSE, assay = "RNA")
options(stringsAsFactors = F)
sobjwgcnamat <- GetAssayData(sobjwgcna)
datExpr <- t(sobjwgcnamat)[,VariableFeatures(sobjwgcna)]
datTraits <- sobjwgcna#meta.data
datTraits = subset(datTraits, select = -c(nCount_RNA, nFeature_RNA))
I then copy-paste the code as written in the WGCNA I.2a tutorial (https://horvath.genetics.ucla.edu/html/CoexpressionNetwork/Rpackages/WGCNA/Tutorials/FemaleLiver-02-networkConstr-auto.pdf), and that all works until I get to this line in the I.3 tutorial (https://horvath.genetics.ucla.edu/html/CoexpressionNetwork/Rpackages/WGCNA/Tutorials/FemaleLiver-03-relateModsToExt.pdf):
MEList = moduleEigengenes(datExpr, colors = moduleColors)
Error in t.default(expr[, restrict1]) : argument is not a matrix
I tried converting both moduleColors and datExpr into a matrix with as.matrix(), but the error still persists.
Hopefully this makes sense, and thanks for reading!
So doing as.matrix(datExpr) right after datExpr <- t(sobjwgcnamat)[,VariableFeatures(sobjwgcna)] worked. I had been trying it right before MEList = moduleEigengenes(datExpr, colors = moduleColors)
and that didn't work. Seems simple but order matters I guess.

Lua: Concise expression of table scope

I'm working on a game where a bunch of characters will be generated on the fly, based on some constraints defined either in the project or externally via mod files. I am using MoonSharp Lua (5.2) interpreter for interfacing with my C# code, and Lua tables to store the constraint presets. As an example:
require "Defaults"
AgePresets = {}
-- Single value
AgePresets.Newborn = 0
-- Simple ranges
AgePresets.Default = defaultAgeRange --referring to the Defaults require
AgePresets.Child = {1, 12}
AgePresets.Teenager = {13, 19}
AgePresets.YoungAdult = {20, 29}
AgePresets.Adult = {30, 40}
AgePresets.MiddleAge = {40, 60}
AgePresets.Senior = {61, 80}
AgePresets.Elder = {81, 99}
AgePresets.Methuselah = {100, 150}
AgePresets.Methuselah2 = {150, 200}
-- Weighted ranges // again referring to previously defined elements to keep things concise
AgePresets.Tween = {
{weight = 1, minmax = AgePresets.Teenager },
{weight = 1, minmax = AgePresets.YoungAdult }
}
This works fine, but from an end-user point of view, there's a lot of unnecessary typing involved. We are clearly working on AgePresets here but it is still mentioned as a prefix before every member name.
I could of course define AgePresets as an array, like AgePresets = { Child = {}, Teenager = {} } but the problem with that is then I cannot refer to previously defined elements in the array.
This doesn't work:
AgePresets = {
Child = {1,12},
RefToChild = Child, //attempt to index a nil value exception
Teen = {13,19}
}
What I ideally want to achieve is a clean, concise way for users to enter this data in, like in the first example but without having to put AgePresets. prefix before everything. How do I go about declaring a scope in a file such that all succeeding members defined in the file will be within that scope, while maintaining the ability to refer to other members defined previously in the scope?
AgePresets = setmetatable({}, {__index = _G})
do
local _ENV = AgePresets
Newborn = 0
Child = {1,12}
RefToChild = Child -- this ref is Ok
Teen = {13,19}
YoungAdult = {20,29}
Tween = {
{weight = 1, minmax = Teen },
{weight = 1, minmax = YoungAdult }
}
rnd = math.random(10) -- global functions are available here
end
setmetatable(AgePresets, nil)
You can mix the two styles: table constructor for fields that don't need to reference variables that aren't in scope yet, followed by assignment statements for the rest.
I would do that unless the order of the fields in the code significantly enhanced comprehension.

Flume Multiplexing not working

I have configured my flume agent like below. Somehow, the flume agent doesn't run properly. It keeps hanging without any errors. Is there any problem with the below configuration.
FYI: I have a file named "country" with hard-coded header as state
#Define sources, sink and channels
foo.sources = s1
foo.channels = chn-az chn-oth
foo.sinks = sink-az sink-oth
#
### # # Define a source on agent and connect to channel memory-channel.
foo.sources.s1.type = exec
foo.sources.s1.command = cat /home/hadoop/flume/country.txt
foo.sources.s1.batchSize = 1
foo.sources.s1.channels = chn-ca chn-oth
#selector configuration
foo.sources.s1.selector.type = multiplexing
foo.sources.s1.selector.header = state
foo.sources.s1.selector.mapping.AZ = chn-az
foo.sources.s1.selector.default = chn-oth
#
#
### Define a memory channel on agent called memory-channel.
foo.channels.chn-az.type = memory
foo.channels.chn-oth.type = memory
#
#
##Define sinks that outputs to hdfs.
foo.sinks.sink-az.channel = chn-az
foo.sinks.sink-az.type = hdfs
foo.sinks.sink-az.hdfs.path = hdfs://master:9099/user/hadoop/flume
foo.sinks.sink-az.hdfs.filePrefix = statefilter
foo.sinks.sink-az.hdfs.fileType = DataStream
foo.sinks.sink-az.hdfs.writeFormat = Text
foo.sinks.sink-az.batchSize = 1
foo.sinks.sink-az.rollInterval = 0
#
foo.sinks.sink-oth.channel = chn-oth
foo.sinks.sink-oth.type = hdfs
foo.sinks.sink-oth.hdfs.path = hdfs://master:9099/user/hadoop/flume
foo.sinks.sink-oth.hdfs.filePrefix = statefilter
foo.sinks.sink-oth.hdfs.fileType = DataStream
foo.sinks.sink-oth.batchSize = 1
foo.sinks.sink-oth.rollInterval = 0
Thanks,
Vinoth
Regarding the channels list configured at the source:
foo.sources.s1.channels = chn-ca chn-oth
I think chn-ca should be chn-az.
Nevertheless, I think such a configuration will never work since the "state" header used by the selector is not created by any Flume component. You must introduce an interceptor for that, typically the Regex Extractor Interceptor.

TYPO3 - Unique content elements in translations not displaying

I'm setting up a Typo3 site, with a default language of English, and a German translation (id=1).
However, on the German translation, I need to be able to create additional content elements in the 'default' column that do not exist in the default.
However, whenever I try to create new content elements, they are showing up in BE, but on the frontend it only renders ones that were created with the 'Copy Default' button.
This is my config:
config.linkVars = L
config.uniqueLinkVars = 1
config.sys_language_overlay = default
config.sys_language_mode = content_overlay
config.language = en
config.locale_all = en_EN
config.htmlTag_langKey = en-EN
config.sys_language_uid = 0
[browser = msie]
config.htmlTag_setParams = xmlns="http://www.w3.org/1999/xhtml" xmlns:v=”urn:schemas-microsoft-com:vml” xml:lang="en"
[globalVar = GP:L = 1]
config.language = de
config.locale_all = de_DE
config.htmlTag_langKey = de-DE
config.sys_language_uid = 1
[globalVar = GP:L = 1] && [browser = msie]
config.htmlTag_setParams = xmlns="http://www.w3.org/1999/xhtml" xmlns:v=”urn:schemas-microsoft-com:vml” xml:lang="de"
[global]
I've copied over the 2 default elements, then tried to add additional elements that are not rendering.
I've not worked with TYPO3 before, but I'm pretty sure those additional content elements should be rendered? Do I need to include any additional markup in the templates to enable it?
Try this:
config.sys_language_overlay = hideNonTranslated
config.sys_language_mode = strict
[globalVar = GP:L = 1]
config.sys_language_overlay = 0
[end]
By setting config.sys_language_overlay = 0, TYPO3 should display your german records even if there is no record in the default language.
Also i corrected your default values for config.sys_language_overlay and config.sys_language_mode since they are not valid.
The documentation can be found at TSREF.
Answering my own question for future visitors...
When you set config.sys_language_overlay it tells Typo3 to actually get all the records from the default language, and then just overlay the matches over the top - that way, it will only show translated elements that have been descended from the default language.
Taking that out completly, it then allows you to use as many content elements in a translation as you want, without paying attention to the default language.
As #Shufla mentioned, using config.sys_language_mode = strict then means that any translations that have less elements than the default won't then inherit the default language ones.

Resources