I am trying to implement the following simple imperative logic:
boolean saveImperative(final List list) {
final String existingList = readByNameImperative(list.getName());
if (Objects.isNull(existingList)) {
templateSaveImperative(existingList);
return true;
} else {
templateSaveImperative(existingList);
return false;
}
}
using Project Reactor in declarative way and this is what I was able to achieve:
#Test
public void testDeclarative() {
final Mono<List> list = createList("foo");
final Boolean result = save(list).block();
System.out.println(result);
}
private Mono<Boolean> save(final Mono<List> listMono) {
final Mono<List> existingListMono = listMono.cache()
.flatMap(list -> readByName(list.getName()));
// if
final Mono<List> savedListMono = existingListMono
.flatMap(existingList -> templateSave(Mono.just(existingList)));
final Mono<Boolean> trueResult = savedListMono.map(x -> true);
// else
return trueResult.switchIfEmpty(templateSave(existingListMono).map(x -> false));
}
private Mono<List> templateSave(final Mono<List> listMono) {
return listMono.map(list -> {
System.out.println("templateSave has been called");
return list;
});
}
private Mono<List> readByName(final String listName) {
if (listName != "list001") {
return Mono.empty();
}
return createList(listName);
}
private Mono<List> createList(final String name) {
final List list = List.builder().name(name).build();
return Mono.just(list);
}
#Value
#Builder
private static class List {
private final String name;
}
If I execute the test with list001, it will print:
templateSave has been called
true
as expected, but if I call it with foo, then I got
null
What I would be missing? I would expect an output like:
templateSave has been called
false
in that case.
final Mono<List> existingListMono = listMono.cache()
.flatMap(list -> readByName(list.getName()));
...in your save method, will take your existing list and flat map it using readByName().
Your readByName() method is the following:
private Mono<List> readByName(final String listName) {
if (listName != "list001") {
return Mono.empty();
}
return createList(listName);
}
(I don't believe it's related to this problem, but don't use == or != for comparing strings.)
Since your listName is foo, not list001, it returns an empty Mono - thus existingListMono becomes an empty Mono, and by implication so do savedListMono and trueResult.
When you call your switchIfEmpty() statement however, you pass in templateSave(existingListMono) - and since existingListMono is an empty Mono as above, the save() method returns an empty Mono.
...and when you block on an empty Mono you'll get null - hence the result.
As such, you may wish to use listMono instead of existingListMono in your return statement on the save() method, which will give you the result you're after:
trueResult.switchIfEmpty(templateSave(listMono).map(x -> false))
Related
I am trying to develop custom source for parallel GCS content scanning. The naive approach would be to loop through listObjects function calls:
while (...) {
Objects objects = gcsUtil.listObjects(bucket, prefix, pageToken);
pageToken = objects.getNextPageToken();
...
}
The problem is performance for the tens of millions objects.
Instead of the single thread code we can add delimiter / and submit parallel processed for each prefix found:
...
Objects objects = gcsUtil.listObjects(bucket, prefix, pageToken, "/");
for (String subPrefix : object.getPrefixes()) {
scanAsync(bucket, subPrefix);
}
...
Next idea was to try to wrap this logic in Splittable DoFn.
Choice of RestrictionTracker: I don't see how can be used any of exiting RestrictionTracker. So decided to write own. Restriction itself is basically list of prefixes to scan. tryClaim checks if there is more prefixes left and receive newly scanned to append them to current restriction. trySplit splits list of prefixes.
The problem that I faced that trySplit can be called before all subPrefixes are found. In this case current restriction may receive more work to do after it was splitted. And it seems that trySplit is being called until it returns a not null value for a given RestrictionTracker: after certain number of elements goes to the output or after 1 second via scheduler or when ProcessContext.resume() returned. This doesn't work in my case as new prefixes can be found at any time. And I can't checkpoint via return ProcessContext.resume() because if split was already done, possible work that left in current restriction will cause checkDone() function to fail.
Another problem that I suspect that I couldn't achieve parallel execution in DirectRunner. As trySplit was always called with fractionOfRemainder=0 and new RestrictionTracker was started only after current one completed its piece of work.
It would be also great to read more detailed explanation about Splittable DoFn components lifecycle. How parallel execution per element is achieved. And how and when state of RestrictionTracker can be modified.
UPD: Adding simplified code that should show intended implementation
#DoFn.BoundedPerElement
private static class ScannerDoFn extends DoFn<String, String> {
private transient GcsUtil gcsUtil;
#GetInitialRestriction
public ScannerRestriction getInitialRestriction(#Element String bucket) {
return ScannerRestriction.init(bucket);
}
#ProcessElement
public ProcessContinuation processElement(
ProcessContext c,
#Element String bucket,
RestrictionTracker<ScannerRestriction, ScannerPosition> tracker,
OutputReceiver<String> outputReceiver) {
if (gcsUtil == null) {
gcsUtil = c.getPipelineOptions().as(GcsOptions.class).getGcsUtil();
}
ScannerRestriction currentRestriction = tracker.currentRestriction();
ScannerPosition position = new ScannerPosition();
while (true) {
if (tracker.tryClaim(position)) {
if (position.completedCurrent) {
// position.clear();
// ideally I would to get checkpoint here before starting new work
return ProcessContinuation.resume();
}
try {
Objects objects = gcsUtil.listObjects(
currentRestriction.bucket,
position.currentPrefix,
position.currentPageToken,
"/");
if (objects.getItems() != null) {
for (StorageObject o : objects.getItems()) {
outputReceiver.output(o.getName());
}
}
if (objects.getPrefixes() != null) {
position.newPrefixes.addAll(objects.getPrefixes());
}
position.currentPageToken = objects.getNextPageToken();
if (position.currentPageToken == null) {
position.completedCurrent = true;
}
} catch (Throwable throwable) {
logger.error("Error during scan", throwable);
}
} else {
return ProcessContinuation.stop();
}
}
}
#NewTracker
public RestrictionTracker<ScannerRestriction, ScannerPosition> restrictionTracker(#Restriction ScannerRestriction restriction) {
return new ScannerRestrictionTracker(restriction);
}
#GetRestrictionCoder
public Coder<ScannerRestriction> getRestrictionCoder() {
return ScannerRestriction.getCoder();
}
}
public static class ScannerPosition {
private String currentPrefix;
private String currentPageToken;
private final List<String> newPrefixes;
private boolean completedCurrent;
public ScannerPosition() {
this.currentPrefix = null;
this.currentPageToken = null;
this.newPrefixes = Lists.newArrayList();
this.completedCurrent = false;
}
public void clear() {
this.currentPageToken = null;
this.currentPrefix = null;
this.completedCurrent = false;
}
}
private static class ScannerRestriction {
private final String bucket;
private final LinkedList<String> prefixes;
private ScannerRestriction(String bucket) {
this.bucket = bucket;
this.prefixes = Lists.newLinkedList();
}
public static ScannerRestriction init(String bucket) {
ScannerRestriction res = new ScannerRestriction(bucket);
res.prefixes.add("");
return res;
}
public ScannerRestriction empty() {
return new ScannerRestriction(bucket);
}
public boolean isEmpty() {
return prefixes.isEmpty();
}
public static Coder<ScannerRestriction> getCoder() {
return ScannerRestrictionCoder.INSTANCE;
}
private static class ScannerRestrictionCoder extends AtomicCoder<ScannerRestriction> {
private static final ScannerRestrictionCoder INSTANCE = new ScannerRestrictionCoder();
private final static Coder<List<String>> listCoder = ListCoder.of(StringUtf8Coder.of());
#Override
public void encode(ScannerRestriction value, OutputStream outStream) throws IOException {
NullableCoder.of(StringUtf8Coder.of()).encode(value.bucket, outStream);
listCoder.encode(value.prefixes, outStream);
}
#Override
public ScannerRestriction decode(InputStream inStream) throws IOException {
String bucket = NullableCoder.of(StringUtf8Coder.of()).decode(inStream);
List<String> prefixes = listCoder.decode(inStream);
ScannerRestriction res = new ScannerRestriction(bucket);
res.prefixes.addAll(prefixes);
return res;
}
}
}
private static class ScannerRestrictionTracker extends RestrictionTracker<ScannerRestriction, ScannerPosition> {
private ScannerRestriction restriction;
private ScannerPosition lastPosition = null;
ScannerRestrictionTracker(ScannerRestriction restriction) {
this.restriction = restriction;
}
#Override
public boolean tryClaim(ScannerPosition position) {
restriction.prefixes.addAll(position.newPrefixes);
position.newPrefixes.clear();
if (position.completedCurrent) {
// completed work for current prefix
assert lastPosition != null && lastPosition.currentPrefix.equals(position.currentPrefix);
lastPosition = null;
return true; // return true but we would need to claim again if we need to get next prefix
} else if (lastPosition != null && lastPosition.currentPrefix.equals(position.currentPrefix)) {
// proceed work for current prefix
lastPosition = position;
return true;
}
// looking for next prefix
assert position.currentPrefix == null;
assert lastPosition == null;
if (restriction.isEmpty()) {
// no work to do
return false;
}
position.currentPrefix = restriction.prefixes.poll();
lastPosition = position;
return true;
}
#Override
public ScannerRestriction currentRestriction() {
return restriction;
}
#Override
public SplitResult<ScannerRestriction> trySplit(double fractionOfRemainder) {
if (lastPosition == null && restriction.isEmpty()) {
// no work at all
return null;
}
if (lastPosition != null && restriction.isEmpty()) {
// work at the moment only at currently scanned prefix
return SplitResult.of(restriction, restriction.empty());
}
int size = restriction.prefixes.size();
int newSize = new Double(Math.round(fractionOfRemainder * size)).intValue();
if (newSize == 0) {
ScannerRestriction residual = restriction;
restriction = restriction.empty();
return SplitResult.of(restriction, residual);
}
ScannerRestriction residual = restriction.empty();
for (int i=newSize; i<=size; i++) {
residual.prefixes.add(restriction.prefixes.removeLast());
}
return SplitResult.of(restriction, residual);
}
#Override
public void checkDone() throws IllegalStateException {
if (lastPosition != null) {
throw new IllegalStateException("Called checkDone on not completed job");
}
}
#Override
public IsBounded isBounded() {
return IsBounded.UNBOUNDED;
}
}
This is my code:
#Path("/hello")
#AllArgsConstructor
public class GreetingResource {
private final Driver driver;
#GET
#Produces(MediaType.TEXT_PLAIN)
public Uni<String> hello() {
return Multi.createFrom().resource(
driver::rxSession,
session -> session.readTransaction(tx -> {
RxResult result = tx.run("MATCH (f:Fruit) RETURN f.name as name ORDER BY f.name");
return Multi.createFrom().publisher(result.records()).map(record -> record.get("name").asString());
})
).withFinalizer(session -> {
return Multi.createFrom().publisher(session.close());
});
}
}
I'm getting those two compilation messages:
Type mismatch: cannot convert from Multi<Object> to Uni<String>
Type mismatch: cannot convert from Multi<Object> to Uni<Void>
I don't quite figure since record.get("name").asString returns me an String...
Any ideas?
The finalizer function must return a Uni<Void>. In your code, it returns a Publisher<Object>. Also, your method is going to return a Multi and not a Uni.
Try the following approach:
#GET
#Produces(MediaType.TEXT_PLAIN)
public Multi<String> hello() {
return Multi.createFrom().resource(
driver::rxSession,
session -> session.readTransaction(tx -> {
RxResult result = tx.run(
"MATCH (f:Fruit) RETURN f.name as name ORDER BY f.name");
return Multi.createFrom().publisher(result.records())
.map(record -> record.get("name").asString());
})
).withFinalizer(session -> {
return Uni.createFrom().publisher(session.close());
});
}
Following code gives me below error -
'System.Threading.Tasks.TaskFactory.StartNew(System.Func)' has some
invalid arguments
public Task<string[]> SayHelloAsync()
{
Task<string> t1 = Task.Factory.StartNew<string>(GreetUser());
Task<string> t2 = Task.Factory.StartNew<string>(GreetCustomer());
Task.WhenAll(t1, t2);
}
private string GreetUser()
{
return "Hello";
}
private string GreetCustomer()
{
return "Namaste";
}
Whats wrong with this.
I am using ASP.Net MVC
'System.Threading.Tasks.TaskFactory.StartNew(System.Func)'
has some invalid arguments
You should state a new task in following way -
Task.Factory.StartNew<string>(() => GreetUser());
overall the method should be like -
public Task<string[]> SayHelloAsync()
{
Task<string> t1 = Task.Factory.StartNew<string>(() => GreetUser());
Task<string> t2 = Task.Factory.StartNew<string>(() => GreetCustomer());
return Task.WhenAll(t1, t2);
}
private string GreetUser()
{
return "Hi";
}
private string GreetCustomer()
{
return "Hi";
}
To expand on ramiramilu's answers, if your DoWork method is parameterless you can just pass it and it will also work (note that you just need to pass the method, not CALL the method).
void StartWork()
{
var task = System.Threading.Tasks.Task.Factory.StartNew<string>(DoWork); //note, you're using "DoWork", not "DoWork()"!!!
}
string DoWork()
{
//do stuff
return "stuff";
}
Using the lambda expressions is only needed when your DoWork methods require input parameters which you want to pass from the calling method. In this case you NEED to use the lambda expression.
void StartWork()
{
var task = System.Threading.Tasks.Task.Factory.StartNew<string>(() => DoWork("my param"));
}
string DoWork(string myParams)
{
//do stuff
return myParams;
}
Otherwise, you can just pass the name of the method and call it a day.
I need to allow my content pipeline extension to use a pattern similar to a factory. I start with a dictionary type:
public delegate T Mapper<T>(MapFactory<T> mf, XElement d);
public class MapFactory<T>
{
Dictionary<string, Mapper<T>> map = new Dictionary<string, Mapper<T>>();
public void Add(string s, Mapper<T> m)
{
map.Add(s, m);
}
public T Get(XElement xe)
{
if (xe == null) throw new ArgumentNullException(
"Invalid document");
var key = xe.Name.ToString();
if (!map.ContainsKey(key)) throw new ArgumentException(
key + " is not a valid key.");
return map[key](this, xe);
}
public IEnumerable<T> GetAll(XElement xe)
{
if (xe == null) throw new ArgumentNullException(
"Invalid document");
foreach (var e in xe.Elements())
{
var val = e.Name.ToString();
if (map.ContainsKey(val))
yield return map[val](this, e);
}
}
}
Here is one type of object I want to store:
public partial class TestContent
{
// Test type
public string title;
// Once test if true
public bool once;
// Parameters
public Dictionary<string, object> args;
public TestContent()
{
title = string.Empty;
args = new Dictionary<string, object>();
}
public TestContent(XElement xe)
{
title = xe.Name.ToString();
args = new Dictionary<string, object>();
xe.ParseAttribute("once", once);
}
}
XElement.ParseAttribute is an extension method that works as one might expect. It returns a boolean that is true if successful.
The issue is that I have many different types of tests, each of which populates the object in a way unique to the specific test. The element name is the key to MapFactory's dictionary. This type of test, while atypical, illustrates my problem.
public class LogicTest : TestBase
{
string opkey;
List<TestBase> items;
public override bool Test(BehaviorArgs args)
{
if (items == null) return false;
if (items.Count == 0) return false;
bool result = items[0].Test(args);
for (int i = 1; i < items.Count; i++)
{
bool other = items[i].Test(args);
switch (opkey)
{
case "And":
result &= other;
if (!result) return false;
break;
case "Or":
result |= other;
if (result) return true;
break;
case "Xor":
result ^= other;
break;
case "Nand":
result = !(result & other);
break;
case "Nor":
result = !(result | other);
break;
default:
result = false;
break;
}
}
return result;
}
public static TestContent Build(MapFactory<TestContent> mf, XElement xe)
{
var result = new TestContent(xe);
string key = "Or";
xe.GetAttribute("op", key);
result.args.Add("key", key);
var names = mf.GetAll(xe).ToList();
if (names.Count() < 2) throw new ArgumentException(
"LogicTest requires at least two entries.");
result.args.Add("items", names);
return result;
}
}
My actual code is more involved as the factory has two dictionaries, one that turns an XElement into a content type to write and another used by the reader to create the actual game objects.
I need to build these factories in code because they map strings to delegates. I have a service that contains several of these factories. The mission is to make these factory classes available to a content processor. Neither the processor itself nor the context it uses as a parameter have any known hooks to attach an IServiceProvider or equivalent.
Any ideas?
I needed to create a data structure essentially on demand without access to the underlying classes as they came from a third party, in this case XNA Game Studio. There is only one way to do this I know of... statically.
public class TestMap : Dictionary<string, string>
{
private static readonly TestMap map = new TestMap();
private TestMap()
{
Add("Logic", "LogicProcessor");
Add("Sequence", "SequenceProcessor");
Add("Key", "KeyProcessor");
Add("KeyVector", "KeyVectorProcessor");
Add("Mouse", "MouseProcessor");
Add("Pad", "PadProcessor");
Add("PadVector", "PadVectorProcessor");
}
public static TestMap Map
{
get { return map; }
}
public IEnumerable<TestContent> Collect(XElement xe, ContentProcessorContext cpc)
{
foreach(var e in xe.Elements().Where(e => ContainsKey(e.Name.ToString())))
{
yield return cpc.Convert<XElement, TestContent>(
e, this[e.Name.ToString()]);
}
}
}
I took this a step further and created content processors for each type of TestBase:
/// <summary>
/// Turns an imported XElement into a TestContent used for a LogicTest
/// </summary>
[ContentProcessor(DisplayName = "LogicProcessor")]
public class LogicProcessor : ContentProcessor<XElement, TestContent>
{
public override TestContent Process(XElement input, ContentProcessorContext context)
{
var result = new TestContent(input);
string key = "Or";
input.GetAttribute("op", key);
result.args.Add("key", key);
var items = TestMap.Map.Collect(input, context);
if (items.Count() < 2) throw new ArgumentNullException(
"LogicProcessor requires at least two items.");
result.args.Add("items", items);
return result;
}
}
Any attempt to reference or access the class such as calling TestMap.Collect will generate the underlying static class if needed. I basically moved the code from LogicTest.Build to the processor. I also carry out any needed validation in the processor.
When I get to reading these classes I will have the ContentService to help.
I need to perform some initialization when new instances of my domain class are created.
class ActivationToken {
String foo
String bar
}
When I do this I want bar to be initialized by code inside ActivationToken:
def tok = new ActivationToken(foo:'a')
I cannot see how to 'override' the 'constructor' to make this happen. I know in this case I could just add a normal constructor but this is just a simple example.
The map constructor is coming from Groovy - not Grails in this case. I did some experimentation, and this is what I came up with:
class Foo {
String name = "bob"
int num = 0
public Foo() {
this([:])
}
public Foo(Map map) {
map?.each { k, v -> this[k] = v }
name = name.toUpperCase()
}
public String toString() {
"$name=$num"
}
}
assert 'BOB=0' == new Foo().toString()
assert 'JOE=32' == new Foo(name:"joe", num: 32).toString()
Basically, it appears that you'll have to manually override the constructors if you need to process the property after construction.
Alternately, you can override individual setters, which is cleaner and safer in general:
class Foo {
String name = "bob"
int num = 0
public void setName(n) {
name = n.toUpperCase()
}
public String toString() {
"$name=$num"
}
}
assert 'bob=0' == new Foo().toString()
assert 'JOE=32' == new Foo(name:"joe", num: 32).toString()
Note that the default value isn't processed, but that should be OK in most instances.
The solution above is also good for cases where initializing an object from parameters in a web request, for example, where you wish to ignore extraneous values, catching Missing property exceptions.
public Foo(Map map) {
try {
map?.each { k, v -> this[k] = v }
}
catch(Exception e){
}
}