How to combine different mono and use the combined result with error handling? - project-reactor

I have a scenario where i need to use different mono which could return me errors and set map values to null if error is returned.
Ex:
Mono<A> a=Some api call;
Mono<A> b=Some api giving error;
Mono<A> c=Some api call;
Now i want to set the resulting response to map
Map<String,A> m=new HashMap<>();
m.put("a",a);
m.put("b",null);
m.put("c",c);
Can anyone help on how to do all this in reactive non blocking way.
I tried zip but it will not execute if any of the api return error or if i use onErrorReturn(null).
Thanks in advance

To solve your problems, you will have to use some tricks. The problem is that :
Giving an empty mono or mono that ends in error cancel zip operation (source: Mono#zip javadoc)
Reactive streams do not allow null values (source: Reactive stream spec, table 2: Subscribers, bullet 13)
Also, note that putting a null value in a hash map is the same as cancelling any previous value associated with the key (it's important in case you're updating an existing map).
Now, to bypass your problem, you can add an abstraction layer, and wrap your values in domain objects.
You can have an object that represents a query, another a valid result, and the last one will mirror an error.
With that, you can design publishers that will always succeed with non null values.
That's a technic used a lot in functional programming : common errors are part of the (one possible) result value.
Now, let's see the example that create a new Map from multiple Monos:
import reactor.core.publisher.Flux;
import reactor.core.publisher.Mono;
import java.time.Duration;
import java.util.Map;
public class BypassMonoError {
/**
* An object identified by a key. It serves to track which key to associate to computed values
* #param <K> Type of the key
*/
static class Identified<K> {
protected final K id;
Identified(K id) {
this.id = id;
}
public K getId() {
return id;
}
}
/**
* Describe the result value of an operation, along with the key associated to it.
*
* #param <K> Type of the identifier of the result
* #param <V> Value type
*/
static abstract class Result<K, V> extends Identified<K> {
Result(K id) {
super(id);
}
/**
*
* #return Computed value on success, or null if the operation has failed. Note that here, we cannot tell from
* a success returning a null value or an error
*/
abstract V getOrNull();
}
static final class Success<K, V> extends Result<K, V> {
private final V value;
Success(K id, V value) {
super(id);
this.value = value;
}
#Override
V getOrNull() {
return value;
}
}
static final class Error<K, V> extends Result<K, V> {
private final Exception error;
Error(K id, Exception error) {
super(id);
this.error = error;
}
#Override
V getOrNull() {
return null;
}
public Exception getError() {
return error;
}
}
/**
* A request that can asynchronously generate a result for the associated identifier.
*/
static class Query<K, V> extends Identified<K> {
private final Mono<V> worker;
Query(K id, Mono<V> worker) {
super(id);
this.worker = worker;
}
/**
* #return The operator that computes the result value. Note that any error is silently wrapped in an
* {#link Error empty result with error metadata}.
*/
public Mono<Result<K, V>> runCatching() {
return worker.<Result<K, V>>map(success -> new Success<>(id, success))
.onErrorResume(Exception.class, error -> Mono.just(new Error<K, V>(id, error)));
}
}
public static void main(String[] args) {
final Flux<Query<String, String>> queries = Flux.just(
new Query("a", Mono.just("A")),
new Query("b", Mono.error(new Exception("B"))),
new Query("c", Mono.delay(Duration.ofSeconds(1)).map(v -> "C"))
);
final Flux<Result<String, String>> results = queries.flatMap(query -> query.runCatching());
final Map<String, String> myMap = results.collectMap(Result::getId, Result::getOrNull)
.block();
for (Map.Entry<String, String> entry : myMap.entrySet()) {
System.out.printf("%s -> %s%n", entry.getKey(), entry.getValue());
}
}
}
Note : In the above example, we silently ignore any occurred error. However, when using the flux, you can test if a result is an error, and if it is, you are free to design your own error management (log, fail-first, send in another flux, etc.).
This outputs:
a -> A
b -> null
c -> C

Related

Save a list in a property in an entity in xodus

I have not found how to save a general list of primitive types, e.g. ints, or strings, in a property of an entity. I might have missed something obvious...
https://github.com/JetBrains/xodus/wiki/Entity-Stores described that "only Java primitive types, Strings, and ComparableSet values can be used by default".
It seems not hard to convert an Iterable into a ComparableSet. However it is a Set.
I will take a look into PersistentEntityStore.registerCustomPropertyType() to see if that helps. I just feel wrong to do that to just save a list of integers.
Links seemed to be able to serve as a way of saving a list of Entitys. But it seems there is no addProperty() counterpart to addLink().
Appreciated if some one can share a way or a workaround for this, or maybe why this is not supported.
Thanks
As mentioned in the comments, one workaround I came up with was to create a ComparableList, by adopting code from ComparableSet.
The idea was to make a list that is able to convert to and from an ArrayByteIterable, and register it with .registerCustomPropertyType(). To do that, 2 classes are needed, ComparableList and ComparableListBinding. I'm sharing one iteration I used at the bottom. By the way, I made them immutable, comparing to the mutable ComparableSet. The newly implemented types should be registered once in a transaction of the store before using them.
That should allow you to store and retrieve a list. However the items in ComparableList would not get indexed as they would when saved in a ComparableSet -- there are some special treatment for ComparableSet in the entity store implementation. So without modifying the library, indexing would work only with hacks like creating another property to just index the values.
I was considering to implement a different entity store that could better support lists, on top of the xodus key-value store, bypassing the xodus entity store entirely. That might be a better solution to the list issue we are talking about here.
ComparableList:
#SuppressWarnings("unchecked")
public class ComparableList<T extends Comparable<T>> implements Comparable<ComparableList<T>>,
Iterable<T> {
#Nonnull
private final ImmutableList<T> list;
public ComparableList(#Nonnull final Iterable<T> iterable) {
list = ImmutableList.copyOf(iterable);
}
#Override
public int compareTo(#Nonnull final ComparableList<T> other) {
final Iterator<T> thisIt = list.iterator();
final Iterator<T> otherIt = other.list.iterator();
while (thisIt.hasNext() && otherIt.hasNext()) {
final int cmp = thisIt.next().compareTo(otherIt.next());
if (cmp != 0) {
return cmp;
}
}
if (thisIt.hasNext()) {
return 1;
}
if (otherIt.hasNext()) {
return -1;
}
return 0;
}
#NotNull
#Override
public Iterator<T> iterator() {
return list.iterator();
}
#Nullable
public Class<T> getItemClass() {
final Iterator<T> it = list.iterator();
return it.hasNext() ? (Class<T>) it.next().getClass() : null;
}
#Override
public String toString() {
return list.toString();
}
}
ComparableListBinding:
#SuppressWarnings({"unchecked", "rawtypes"})
public class ComparableListBinding extends ComparableBinding {
public static final ComparableListBinding INSTANCE = new ComparableListBinding();
private ComparableListBinding() {}
#Override
public ComparableList readObject(#NotNull final ByteArrayInputStream stream) {
final int valueTypeId = stream.read() ^ 0x80;
final ComparableBinding itemBinding = ComparableValueType.getPredefinedBinding(valueTypeId);
final ImmutableList.Builder<Comparable> builder = ImmutableList.builder();
while (stream.available() > 0) {
builder.add(itemBinding.readObject(stream));
}
return new ComparableList(builder.build());
}
#Override
public void writeObject(#NotNull final LightOutputStream output,
#NotNull final Comparable object) {
final ComparableList<? extends Comparable> list = (ComparableList) object;
final Class itemClass = list.getItemClass();
if (itemClass == null) {
throw new ExodusException("Attempt to write empty ComparableList");
}
final ComparableValueType type = ComparableValueType.getPredefinedType(itemClass);
output.writeByte(type.getTypeId());
final ComparableBinding itemBinding = type.getBinding();
list.forEach(o -> itemBinding.writeObject(output, o));
}
/**
* De-serializes {#linkplain ByteIterable} entry to a {#code ComparableList} value.
*
* #param entry {#linkplain ByteIterable} instance
* #return de-serialized value
*/
public static ComparableList entryToComparableList(#NotNull final ByteIterable entry) {
return (ComparableList) INSTANCE.entryToObject(entry);
}
/**
* Serializes {#code ComparableList} value to the {#linkplain ArrayByteIterable} entry.
*
* #param object value to serialize
* #return {#linkplain ArrayByteIterable} entry
*/
public static ArrayByteIterable comparableSetToEntry(#NotNull final ComparableList object) {
return INSTANCE.objectToEntry(object);
}
}

Does Apache Beam support custom file names for its output?

While in a distributed processing environment it is common to use "part" file names such as "part-000", is it possible to write an extension of some sort to rename the individual output file names (such as a per window file name) of Apache Beam?
To do this, one might have to be able to assign a name for a window or infer a file name based on the window's content. I would like to know if such an approach is possible.
As to whether the solution should be streaming or batch, a streaming mode example is preferable
Yes as suggested by jkff you can achieve this using TextIO.write.to(FilenamePolicy).
Examples are below:
If you want to write output to particular local file you can use:
lines.apply(TextIO.write().to("/path/to/file.txt"));
Below is the simple way to write the output using the prefix, link. This example is for google storage, instead of this you can use local/s3 paths.
public class MinimalWordCountJava8 {
public static void main(String[] args) {
PipelineOptions options = PipelineOptionsFactory.create();
// In order to run your pipeline, you need to make following runner specific changes:
//
// CHANGE 1/3: Select a Beam runner, such as BlockingDataflowRunner
// or FlinkRunner.
// CHANGE 2/3: Specify runner-required options.
// For BlockingDataflowRunner, set project and temp location as follows:
// DataflowPipelineOptions dataflowOptions = options.as(DataflowPipelineOptions.class);
// dataflowOptions.setRunner(BlockingDataflowRunner.class);
// dataflowOptions.setProject("SET_YOUR_PROJECT_ID_HERE");
// dataflowOptions.setTempLocation("gs://SET_YOUR_BUCKET_NAME_HERE/AND_TEMP_DIRECTORY");
// For FlinkRunner, set the runner as follows. See {#code FlinkPipelineOptions}
// for more details.
// options.as(FlinkPipelineOptions.class)
// .setRunner(FlinkRunner.class);
Pipeline p = Pipeline.create(options);
p.apply(TextIO.read().from("gs://apache-beam-samples/shakespeare/*"))
.apply(FlatMapElements
.into(TypeDescriptors.strings())
.via((String word) -> Arrays.asList(word.split("[^\\p{L}]+"))))
.apply(Filter.by((String word) -> !word.isEmpty()))
.apply(Count.<String>perElement())
.apply(MapElements
.into(TypeDescriptors.strings())
.via((KV<String, Long> wordCount) -> wordCount.getKey() + ": " + wordCount.getValue()))
// CHANGE 3/3: The Google Cloud Storage path is required for outputting the results to.
.apply(TextIO.write().to("gs://YOUR_OUTPUT_BUCKET/AND_OUTPUT_PREFIX"));
p.run().waitUntilFinish();
}
}
This example code will give you more control on writing the output:
/**
* A {#link FilenamePolicy} produces a base file name for a write based on metadata about the data
* being written. This always includes the shard number and the total number of shards. For
* windowed writes, it also includes the window and pane index (a sequence number assigned to each
* trigger firing).
*/
protected static class PerWindowFiles extends FilenamePolicy {
private final ResourceId prefix;
public PerWindowFiles(ResourceId prefix) {
this.prefix = prefix;
}
public String filenamePrefixForWindow(IntervalWindow window) {
String filePrefix = prefix.isDirectory() ? "" : prefix.getFilename();
return String.format(
"%s-%s-%s", filePrefix, formatter.print(window.start()), formatter.print(window.end()));
}
#Override
public ResourceId windowedFilename(int shardNumber,
int numShards,
BoundedWindow window,
PaneInfo paneInfo,
OutputFileHints outputFileHints) {
IntervalWindow intervalWindow = (IntervalWindow) window;
String filename =
String.format(
"%s-%s-of-%s%s",
filenamePrefixForWindow(intervalWindow),
shardNumber,
numShards,
outputFileHints.getSuggestedFilenameSuffix());
return prefix.getCurrentDirectory().resolve(filename, StandardResolveOptions.RESOLVE_FILE);
}
#Override
public ResourceId unwindowedFilename(
int shardNumber, int numShards, OutputFileHints outputFileHints) {
throw new UnsupportedOperationException("Unsupported.");
}
}
#Override
public PDone expand(PCollection<InputT> teamAndScore) {
if (windowed) {
teamAndScore
.apply("ConvertToRow", ParDo.of(new BuildRowFn()))
.apply(new WriteToText.WriteOneFilePerWindow(filenamePrefix));
} else {
teamAndScore
.apply("ConvertToRow", ParDo.of(new BuildRowFn()))
.apply(TextIO.write().to(filenamePrefix));
}
return PDone.in(teamAndScore.getPipeline());
}
Yes. Per documentation of TextIO:
If you want better control over how filenames are generated than the default policy allows, a custom FilenamePolicy can also be set using TextIO.Write.to(FilenamePolicy)
This is perfectly valid example with beam 2.1.0. You can call on your data (PCollection e.g)
import org.apache.beam.sdk.io.FileBasedSink.FilenamePolicy;
import org.apache.beam.sdk.io.TextIO;
import org.apache.beam.sdk.io.fs.ResolveOptions.StandardResolveOptions;
import org.apache.beam.sdk.io.fs.ResourceId;
import org.apache.beam.sdk.transforms.display.DisplayData;
#SuppressWarnings("serial")
public class FilePolicyExample {
public static void main(String[] args) {
FilenamePolicy policy = new WindowedFilenamePolicy("somePrefix");
//data
data.apply(TextIO.write().to("your_DIRECTORY")
.withFilenamePolicy(policy)
.withWindowedWrites()
.withNumShards(4));
}
private static class WindowedFilenamePolicy extends FilenamePolicy {
final String outputFilePrefix;
WindowedFilenamePolicy(String outputFilePrefix) {
this.outputFilePrefix = outputFilePrefix;
}
#Override
public ResourceId windowedFilename(
ResourceId outputDirectory, WindowedContext input, String extension) {
String filename = String.format(
"%s-%s-%s-of-%s-pane-%s%s%s",
outputFilePrefix,
input.getWindow(),
input.getShardNumber(),
input.getNumShards() - 1,
input.getPaneInfo().getIndex(),
input.getPaneInfo().isLast() ? "-final" : "",
extension);
return outputDirectory.resolve(filename, StandardResolveOptions.RESOLVE_FILE);
}
#Override
public ResourceId unwindowedFilename(
ResourceId outputDirectory, Context input, String extension) {
throw new UnsupportedOperationException("Expecting windowed outputs only");
}
#Override
public void populateDisplayData(DisplayData.Builder builder) {
builder.add(DisplayData.item("fileNamePrefix", outputFilePrefix)
.withLabel("File Name Prefix"));
}
}
}
You can check https://beam.apache.org/releases/javadoc/2.3.0/org/apache/beam/sdk/io/FileIO.html for more information, you should search "File naming" in "Writing files".
.apply(
FileIO.<RootElement>write()
.via(XmlIO
.sink(RootElement.class)
.withRootElement(ROOT_XML_ELEMENT)
.withCharset(StandardCharsets.UTF_8))
.to(FILE_PATH)
.withNaming((window, pane, numShards, shardIndex, compression) -> NEW_FILE_NAME)

I want to remove the secret information of a PrivateKey (sun.security.rsa.RSAPrivateCrtKeyImpl) from memory?

I have some Java code that generates an instance of a class that implements the interface java.security.PrivateKey (the actual object is an instance of sun.security.rsa.RSAPrivateCrtKeyImpl). The purpose of this code is to generate a PrivateKey, then split it into shares (divisions of the key) and then store the shares on smartcards. The shares part is working fine, the question I have is how to handle the PrivateKey once it is no longer needed.
There is a requirement that the PrivateKey information (which should be kept secret) should be removed from memory as soon as possible after it has been split into shares.
The interface PrivateKey extends the interface Destroyable, but I dont see the methods of Destroyable being implemented anywhere (it must be implemented somewhere in this hierarchy). But if I call destroy on the PrivateKey that I have (which is a RSAPrivateCrtKeyImpl) then it throws a javax.security.auth.DestroyFailedException.
The object that I have seems totally immutable, is there any way to overwrite the fields within the object? such as setting them to zero? or what approach should be followed for removing such secret immutable objects from memory? Thanks!
Yes you can, by using reflection.
I created a simple code you can use to destroy the key informations :
public static void destroyRSAKey(sun.security.rsa.RSAPrivateCrtKeyImpl key) throws NoSuchFieldException, SecurityException {
destroyBigInteger(key.getModulus());
destroyBigInteger(key.getPublicExponent());
destroyBigInteger(key.getPrivateExponent());
destroyBigInteger(key.getPrimeP());
destroyBigInteger(key.getPrimeQ());
destroyBigInteger(key.getPrimeExponentP());
destroyBigInteger(key.getPrimeExponentQ());
destroyBigInteger(key.getCrtCoefficient());
}
public static final void destroyBigInteger(java.math.BigInteger destroyThis) {
try {
java.lang.reflect.Field f = java.math.BigInteger.class.getDeclaredField("mag");
f.setAccessible(true);
f.set(destroyThis, new int[] { 0 });
f.setAccessible(false);
f = java.math.BigInteger.class.getDeclaredField("signum");
f.setAccessible(true);
f.setInt(destroyThis, 0);
f.setAccessible(false);
} catch (Throwable e) {
e.printStackTrace();
}
}
No you cannot. The Java software implementation of key creation uses BigInteger which is indeed immutable. As you may have found out the Destroyable.destroy method is only implemented in few classes. As of Java 7 these were the directly implementing classes:
KerberosKey, KerberosTicket, KeyStore.PasswordProtection, X500PrivateCredential
I would guess that these classes do implement Destroyable.destroy but that all the others do not.
So the only thing you can do is to rely on the garbage collector and possibly the operating system to clean up and / or overwrite the BigInteger instances. Note that you would have the same issue when using the private key.
If you already have a SmartCard I would suggest using the smart cards to generate and split the keys. Generate the key internally, then encrypt the key with a symmetric AES key, split the AES key and transport the wrapped RSA key and the (2 remaining) key parts out of the smart card. Usually XOR is used instead of splitting the key literally.
EDIT
As documented above, java.security.PrivateKey.destroy() is not implemented in Java 8, which is a Serious Security Gap of the Sun Java Runtime Environment when deploying RSA algorithms.
Based on Doomny 58's solution, I got an actually running method destroyPrivateRSAKey() with proper content destruction; any error is mapped to DestroyFailedException, so when the call succeeds you may be sure, the private RSA parameters are destroyed.
/**
* Destroy data of a BigInteger
*
* #param destroyThis [in] BigIntegeer onject to destroy
* #throws DestroyFailedException on any error
*/
public static final void destroyBigInteger(java.math.BigInteger destroyThis) throws DestroyFailedException
{
if (destroyThis == null)
{
return;
}
try
{
// Retrieve buffer field 'mag', overwrite with 0-s and set to {0}
java.lang.reflect.Field f = java.math.BigInteger.class.getDeclaredField("mag");
f.setAccessible(true);
int[] mag = (int[]) f.get(destroyThis);
if (mag != null)
{
java.util.Arrays.fill(mag, 0);
}
f.set(destroyThis, new int[] { 0 });
f.setAccessible(false);
// set field signum = 0
f = java.math.BigInteger.class.getDeclaredField("signum");
f.setAccessible(true);
f.setInt(destroyThis, 0);
f.setAccessible(false);
}
catch (Throwable e)
{
e.printStackTrace();
throw new DestroyFailedException();
}
}
/**
* Get BigInteger member of an object with spec'd accesor method
* #param o [in] The object
* #param accessor [in] Name of the accessor method
* #return The BigInteger
* #throws DestroyFailedException on any error
*/
private static BigInteger accessBIF(Object o, String accessor) throws DestroyFailedException
{
try
{
Class<?> cls = o.getClass();
Method m = cls.getDeclaredMethod(accessor);
m.setAccessible(true);
BigInteger res = (BigInteger) m.invoke(o);
m.setAccessible(false);
return res;
}
catch (Throwable x)
{
x.printStackTrace();
throw new DestroyFailedException();
}
}
/**
* Destroy BigInteger member
*
* #param o [in] Object with BigInteger
* #param accessor [in] Member Accessor method
* #throws DestroyFailedException
*/
private static void destroyBIF(Object o, String accessor) throws DestroyFailedException
{
destroyBigInteger(accessBIF(o, accessor));
}
/**
* Destroy Private RSA Key
*
* #param key [in] A sun.security.rsa.RSAPrivateCrtKeyImpl
* #throws DestroyFailedException
*/
public static void destroyPrivateRSAKey(Object key) throws DestroyFailedException
{
destroyBIF(key, "getModulus");
destroyBIF(key, "getPublicExponent");
destroyBIF(key, "getPrivateExponent");
destroyBIF(key, "getPrimeP");
destroyBIF(key, "getPrimeQ");
destroyBIF(key, "getPrimeExponentP");
destroyBIF(key, "getPrimeExponentQ");
destroyBIF(key, "getCrtCoefficient");
}

Displaying Many Jira issues using Issue Navigator

My question is similar to "Displaying Jira issues using Issue Navigator" But my concern is that sometimes my list of issues is quite long and providing the user with a link to the Issue Navigator only works if my link is shorter than the max length of URLs.
Is there another way? Cookies? POST data? Perhaps programmatically creating and sharing a filter on the fly, and returning a link to the Issue Navigator that uses this filter? (But at some point I'd want to delete these filters so I don't have so many lying around.)
I do have the JQL query for getting the same list of issues, but it takes a very (very) long time to run and my code has already done the work of finding out what the result is -- I don't want the user to wait twice for the same thing (once while I'm generating my snazzy graphical servlet view and a second time when they want to see the same results in the Issue Navigator).
The easiest way to implement this is to ensure that the list of issues is cached somewhere within one of your application components. Each separate list of issues should be identified by its own unique ID (the magicKey below) that you define yourself.
You would then write your own JQL function that looks up your pre-calculated list of issues by that magic key, which then converts the list of issues into the format that the Issue Navigator requires.
The buildUriEncodedJqlQuery() method can be used to create such an example JQL query dynamically. For example, if the magic key is 1234, it would yield this JQL query: issue in myJqlFunction(1234).
You'd then feed the user a URL that looks like this:
String url = "/secure/IssueNavigator.jspa?mode=hide&reset=true&jqlQuery=" + MyJqlFunction.buildUriEncodedJqlQuery(magicKey);
The end result is that the user will be placed in the Issue Navigator looking at exactly the list of issues your code has provided.
This code also specifically prevents the user from saving your JQL function as part of a saved filter, since it's assumed that the issue cache in your application will not be permanent. If that's not correct, you will want to empty out the sanitiseOperand part.
In atlassian-plugins.xml:
<jql-function key="myJqlFunction" name="My JQL Function"
class="com.mycompany.MyJqlFunction">
</jql-function>
JQL function:
import com.atlassian.crowd.embedded.api.User;
import com.atlassian.jira.JiraDataType;
import com.atlassian.jira.JiraDataTypes;
import com.atlassian.jira.jql.operand.QueryLiteral;
import com.atlassian.jira.jql.query.QueryCreationContext;
import com.atlassian.jira.plugin.jql.function.ClauseSanitisingJqlFunction;
import com.atlassian.jira.plugin.jql.function.JqlFunction;
import com.atlassian.jira.plugin.jql.function.JqlFunctionModuleDescriptor;
import com.atlassian.jira.util.MessageSet;
import com.atlassian.jira.util.MessageSetImpl;
import com.atlassian.jira.util.NotNull;
import com.atlassian.query.clause.TerminalClause;
import com.atlassian.query.operand.FunctionOperand;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;
public class MyJqlFunction implements JqlFunction, ClauseSanitisingJqlFunction
{
private static final String JQL_FUNCTION_NAME = "myJqlFunctionName";
private static final int JQL_FUNCTION_MIN_ARG_COUNT = 1;
private static final int JQL_FUNCTION_MAGIC_KEY_INDEX = 0;
public MyJqlFunction()
{
// inject your app's other components here
}
#Override
public void init(#NotNull JqlFunctionModuleDescriptor moduleDescriptor)
{
}
#Override
public JiraDataType getDataType()
{
return JiraDataTypes.ISSUE;
}
#Override
public String getFunctionName()
{
return JQL_FUNCTION_NAME;
}
#Override
public int getMinimumNumberOfExpectedArguments()
{
return JQL_FUNCTION_MIN_ARG_COUNT;
}
#Override
public boolean isList()
{
return true;
}
/**
* This function generates a URL-escaped JQL query that corresponds to the supplied magic key.
*
* #param magicKey
* #return
*/
public static String buildUriEncodedJqlQuery(String magicKey)
{
return "issue%20in%20" + JQL_FUNCTION_NAME + "(%22"
+ magicKey + "%22%)";
}
#Override
public List<QueryLiteral> getValues(#NotNull QueryCreationContext queryCreationContext,
#NotNull FunctionOperand operand,
#NotNull TerminalClause terminalClause)
{
User searchUser = queryCreationContext.getUser();
MessageSet messages = new MessageSetImpl();
List<QueryLiteral> values = internalGetValues(searchUser,
operand,
terminalClause,
messages,
!queryCreationContext.isSecurityOverriden());
return values;
}
private List<QueryLiteral> internalGetValues(#NotNull User searchUser,
#NotNull FunctionOperand operand,
#NotNull TerminalClause terminalClause,
#NotNull MessageSet messages,
#NotNull Boolean checkSecurity)
{
List<QueryLiteral> result = new ArrayList<QueryLiteral>();
if (searchUser==null)
{
// handle anon user
}
List<String> args = operand.getArgs();
if (wasSanitised(args))
{
messages.addErrorMessage("this function can't be used as part of a saved filter etc");
return result;
}
if (args.size() < getMinimumNumberOfExpectedArguments())
{
messages.addErrorMessage("too few arguments, etc");
return result;
}
final String magicKey = args.get(JQL_FUNCTION_MAGIC_KEY_INDEX);
// You need to implement this part yourself! This is where you use the supplied
// magicKey to fetch a list of issues from your own internal data source.
List<String> myIssueKeys = myCache.get(magicKey);
for (String id : myIssueKeys)
{
result.add(new QueryLiteral(operand, id));
}
return result;
}
#Override
public MessageSet validate(User searcher, #NotNull FunctionOperand operand, #NotNull TerminalClause terminalClause)
{
MessageSet messages = new MessageSetImpl();
internalGetValues(searcher, operand, terminalClause, messages, true);
return messages;
}
#Override
public FunctionOperand sanitiseOperand(User paramUser, #NotNull FunctionOperand operand)
{
// prevent the user from saving this as a filter, since the results are presumed to be dynamic
return new FunctionOperand(operand.getName(), Arrays.asList(""));
}
private boolean wasSanitised(List<String> args)
{
return (args.size() == 0 || args.get(JQL_FUNCTION_MAGIC_KEY_INDEX).isEmpty());
}
}

Method to create and store method chain at runtime

The problem I have is that I need to do about 40+ conversions to convert loosely typed info into strongly typed info stored in db, xml file, etc.
I'm plan to tag each type with a tuple i.e. a transformational form like this:
host.name.string:host.dotquad.string
which will offer a conversion from the input to an output form. For example, the name stored in the host field of type string, the input is converted into a dotquad notation of type string and stored back into host field. More complex conversions may need several steps, with each step being accomplished by a method call, hence method chaining.
Examining further the example above, the tuple 'host.name.string' with the field host of name www.domain.com. A DNS lookup is done to covert domain name to IP address. Another method is applied to change the type returned by the DNS lookup into the internal type of dotquad of type string. For this transformation, there is 4 seperate methods called to convert from one tuple into another. Some other conversions may require more steps.
Ideally I would like an small example of how method chains are constructed at runtime. Development time method chaining is relatively trivial, but would require pages and pages of code to cover all possibilites, with 40+ conversions.
One way I thought of doing is, is parsing the tuples at startup, and writing the chains out to an assembly, compiling it, then using reflection to load/access. Its would be really ugly and negate the performance increases i'm hoping to gain.
I'm using Mono, so no C# 4.0
Any help would be appreciated.
Bob.
Here is a quick and dirty solution using LINQ Expressions. You have indicated that you want C# 2.0, this is 3.5, but it does run on Mono 2.6. The method chaining is a bit hacky as i didn't exactly know how your version works, so you might need to tweak the expression code to suit.
The real magic really happens in the Chainer class, which takes a collection of strings, which represent the MethodChain subclass. Take a collection like this:
{
"string",
"string",
"int"
}
This will generate a chain like this:
new StringChain(new StringChain(new IntChain()));
Chainer.CreateChain will return a lambda that calls MethodChain.Execute(). Because Chainer.CreateChain uses a bit of reflection, it's slow, but it only needs to run once for each expression chain. The execution of the lambda is nearly as fast as calling actual code.
Hope you can fit this into your architecture.
public abstract class MethodChain {
private MethodChain[] m_methods;
private object m_Result;
public MethodChain(params MethodChain[] methods) {
m_methods = methods;
}
public MethodChain Execute(object expression) {
if(m_methods != null) {
foreach(var method in m_methods) {
expression = method.Execute(expression).GetResult<object>();
}
}
m_Result = ExecuteInternal(expression);
return this;
}
protected abstract object ExecuteInternal(object expression);
public T GetResult<T>() {
return (T)m_Result;
}
}
public class IntChain : MethodChain {
public IntChain(params MethodChain[] methods)
: base(methods) {
}
protected override object ExecuteInternal(object expression) {
return int.Parse(expression as string);
}
}
public class StringChain : MethodChain {
public StringChain(params MethodChain[] methods):base(methods) {
}
protected override object ExecuteInternal(object expression) {
return (expression as string).Trim();
}
}
public class Chainer {
/// <summary>
/// methods are executed from back to front, so methods[1] will call method[0].Execute before executing itself
/// </summary>
/// <param name="methods"></param>
/// <returns></returns>
public Func<object, MethodChain> CreateChain(IEnumerable<string> methods) {
Expression expr = null;
foreach(var methodName in methods.Reverse()) {
ConstructorInfo cInfo= null;
switch(methodName.ToLower()) {
case "string":
cInfo = typeof(StringChain).GetConstructor(new []{typeof(MethodChain[])});
break;
case "int":
cInfo = typeof(IntChain).GetConstructor(new[] { typeof(MethodChain[]) });
break;
}
if(cInfo == null)
continue;
if(expr != null)
expr = Expression.New(cInfo, Expression.NewArrayInit( typeof(MethodChain), Expression.Convert(expr, typeof(MethodChain))));
else
expr = Expression.New(cInfo, Expression.Constant(null, typeof(MethodChain[])));
}
var objParam = Expression.Parameter(typeof(object));
var methodExpr = Expression.Call(expr, typeof(MethodChain).GetMethod("Execute"), objParam);
Func<object, MethodChain> lambda = Expression.Lambda<Func<object, MethodChain>>(methodExpr, objParam).Compile();
return lambda;
}
[TestMethod]
public void ExprTest() {
Chainer chainer = new Chainer();
var lambda = chainer.CreateChain(new[] { "int", "string" });
var result = lambda(" 34 ").GetResult<int>();
Assert.AreEqual(34, result);
}
}
The command pattern would fit here. What you could do is queue up commands as you need different operations performed on the different data types. Those messages could then all be processed and call the appropriate methods when you're ready later on.
This pattern can be implemented in .NET 2.0.
Do you really need to do this at execution time? Can't you create the combination of operations using code generation?
Let me elaborate:
Assuming you have a class called Conversions which contains all the 40+ convertions you mentioned like this:
//just pseudo code..
class conversions{
string host_name(string input){}
string host_dotquad(string input){}
int type_convert(string input){}
float type_convert(string input){}
float increment_float(float input){}
}
Write a simple console app or something similar which uses reflection to generate code for methods like this:
execute_host_name(string input, Queue<string> conversionQueue)
{
string ouput = conversions.host_name(input);
if(conversionQueue.Count == 0)
return output;
switch(conversionQueue.dequeue())
{
// generate case statements only for methods that take in
// a string as parameter because the host_name method returns a string.
case "host.dotquad": return execute_host_dotquad(output,conversionQueue);
case "type.convert": return execute_type_convert(output, conversionQueue);
default: // exception...
}
}
Wrap all this in a Nice little execute method like this:
object execute(string input, string [] conversions)
{
Queue<string> conversionQueue = //create the queue..
case(conversionQueue.dequeue())
{
case "host.name": return execute_host_name(output,conversionQueue);
case "host.dotquad": return execute_host_dotquad(output,conversionQueue);
case "type.convert": return execute_type_convert(output, conversionQueue);
default: // exception...
}
}
This code generation application need to be executed only when your method signatures changes or when you decide to add new transformations.
Main advantages:
No runtime overhead
Easy to add/delete/change the conversions (code generator will take care of the code changes :) )
What do you think?
I apologize for the long code dump and the fact that it is in Java, rather than C#, but I found your problem quite interesting and I do not have much C# experience. Hopefully you will be able to adapt this solution without difficulty.
One approach to solving your problem is to create a cost for each conversion -- usually this is related to the accuracy of the conversion -- and then perform a search to find the best possible conversion sequence to get from one type to another.
The reason for needing a cost function is to choose among multiple conversion paths. For example, converting from an integer to a string is lossless, but there is no guarantee that every string can be represented by an integer. So, if you had two conversion chains
string -> integer -> float -> decimal
string -> float -> decimal
You would want to select the second one because it will reduce the chance of a conversion failure.
The Java code below implements such a scheme and performs a best-first search to find an optimal conversion sequence. I hope you find it useful. Running the code produces the following output:
> No conversion possible from string to integer
> The optimal conversion sequence from string to host.dotquad.string is:
> string to host.name.string, cost = -1.609438
> host.name.string to host.dns, cost = -1.609438 *PERFECT*
> host.dns to host.dotquad, cost = -1.832581
> host.dotquad to host.dotquad.string, cost = -1.832581 *PERFECT*
Here is the Java code.
/**
* Use best-first search to find an optimal sequence of operations for
* performing a type conversion with maximum fidelity.
*/
import java.util.*;
public class TypeConversion {
/**
* Define a type-conversion interface. It converts between to
* user-defined types and provides a measure of fidelity (accuracy)
* of the conversion.
*/
interface ITypeConverter<T, F> {
public T convert(F from);
public double fidelity();
// Could use reflection instead of handling this explicitly
public String getSourceType();
public String getTargetType();
}
/**
* Create a set of user-defined types.
*/
class HostName {
public String hostName;
public HostName(String hostName) {
this.hostName = hostName;
}
}
class DnsLookup {
public String ipAddress;
public DnsLookup(HostName hostName) {
this.ipAddress = doDNSLookup(hostName);
}
private String doDNSLookup(HostName hostName) {
return "127.0.0.1";
}
}
class DottedQuad {
public int[] quad = new int[4];
public DottedQuad(DnsLookup lookup) {
String[] split = lookup.ipAddress.split(".");
for ( int i = 0; i < 4; i++ )
quad[i] = Integer.parseInt( split[i] );
}
}
/**
* Define a set of conversion operations between the types. We only
* implement a minimal number for brevity, but this could be expanded.
*
* We start by creating some broad classes to differentiate among
* perfect, good and bad conversions.
*/
abstract class PerfectTypeConversion<T, F> implements ITypeConverter<T, F> {
public abstract T convert(F from);
public double fidelity() { return 1.0; }
}
abstract class GoodTypeConversion<T, F> implements ITypeConverter<T, F> {
public abstract T convert(F from);
public double fidelity() { return 0.8; }
}
abstract class BadTypeConversion<T, F> implements ITypeConverter<T, F> {
public abstract T convert(F from);
public double fidelity() { return 0.2; }
}
/**
* Concrete classes that do the actual conversions.
*/
class StringToHostName extends BadTypeConversion<HostName, String> {
public HostName convert(String from) { return new HostName(from); }
public String getSourceType() { return "string"; }
public String getTargetType() { return "host.name.string"; }
}
class HostNameToDnsLookup extends PerfectTypeConversion<DnsLookup, HostName> {
public DnsLookup convert(HostName from) { return new DnsLookup(from); }
public String getSourceType() { return "host.name.string"; }
public String getTargetType() { return "host.dns"; }
}
class DnsLookupToDottedQuad extends GoodTypeConversion<DottedQuad, DnsLookup> {
public DottedQuad convert(DnsLookup from) { return new DottedQuad(from); }
public String getSourceType() { return "host.dns"; }
public String getTargetType() { return "host.dotquad"; }
}
class DottedQuadToString extends PerfectTypeConversion<String, DottedQuad> {
public String convert(DottedQuad f) {
return f.quad[0] + "." + f.quad[1] + "." + f.quad[2] + "." + f.quad[3];
}
public String getSourceType() { return "host.dotquad"; }
public String getTargetType() { return "host.dotquad.string"; }
}
/**
* To find the best conversion sequence, we need to instantiate
* a list of converters.
*/
ITypeConverter<?,?> converters[] =
{
new StringToHostName(),
new HostNameToDnsLookup(),
new DnsLookupToDottedQuad(),
new DottedQuadToString()
};
Map<String, List<ITypeConverter<?,?>>> fromMap =
new HashMap<String, List<ITypeConverter<?,?>>>();
public void buildConversionMap()
{
for ( ITypeConverter<?,?> converter : converters )
{
String type = converter.getSourceType();
if ( !fromMap.containsKey( type )) {
fromMap.put( type, new ArrayList<ITypeConverter<?,?>>());
}
fromMap.get(type).add(converter);
}
}
public class Tuple implements Comparable<Tuple>
{
public String type;
public double cost;
public Tuple parent;
public Tuple(String type, double cost, Tuple parent) {
this.type = type;
this.cost = cost;
this.parent = parent;
}
public int compareTo(Tuple o) {
return Double.compare( cost, o.cost );
}
}
public Tuple findOptimalConversionSequence(String from, String target)
{
PriorityQueue<Tuple> queue = new PriorityQueue<Tuple>();
// Add a dummy start node to the queue
queue.add( new Tuple( from, 0.0, null ));
// Perform the search
while ( !queue.isEmpty() )
{
// Pop the most promising candidate from the list
Tuple tuple = queue.remove();
// If the type matches the target type, return
if ( tuple.type == target )
return tuple;
// If we have reached a dead-end, backtrack
if ( !fromMap.containsKey( tuple.type ))
continue;
// Otherwise get all of the possible conversions to
// perform next and add their costs
for ( ITypeConverter<?,?> converter : fromMap.get( tuple.type ))
{
String type = converter.getTargetType();
double cost = tuple.cost + Math.log( converter.fidelity() );
queue.add( new Tuple( type, cost, tuple ));
}
}
// No solution
return null;
}
public static void convert(String from, String target)
{
TypeConversion tc = new TypeConversion();
// Build a conversion lookup table
tc.buildConversionMap();
// Find the tail of the optimal conversion chain.
Tuple tail = tc.findOptimalConversionSequence( from, target );
if ( tail == null ) {
System.out.println( "No conversion possible from " + from + " to " + target );
return;
}
// Reconstruct the conversion path (skip dummy node)
List<Tuple> solution = new ArrayList<Tuple>();
for ( ; tail.parent != null ; tail = tail.parent )
solution.add( tail );
Collections.reverse( solution );
StringBuilder sb = new StringBuilder();
Formatter formatter = new Formatter(sb);
sb.append( "The optimal conversion sequence from " + from + " to " + target + " is:\n" );
for ( Tuple tuple : solution ) {
formatter.format( "%20s to %20s, cost = %f", tuple.parent.type, tuple.type, tuple.cost );
if ( tuple.cost == tuple.parent.cost )
sb.append( " *PERFECT*");
sb.append( "\n" );
}
System.out.println( sb.toString() );
}
public static void main(String[] args)
{
// Run two tests
convert( "string", "integer" );
convert( "string", "host.dotquad.string" );
}
}

Resources