I have a function that accepts a Stream. My data is in a large list, running into millions of items.
Is there a simple way I can wrap a sequence in a Stream, returning chunks of my sequence in the stream? One obvious approach is to implement my own stream class that returns chunks of the sequence. Something like :
type SeqStream(sequence:seq<'a>) =
inherit Stream()
default x.Read(buf, offset, count) =
// get next chunk
// yield chunk
Is there a simpler way of doing it? I don't have the means to change the target function that accepts a stream though.
I think that your approach looks good. The only problem is that Stream is a relatively complicated class that has quite a few members and you probably don't want to implement most of them - if you want to pass it to some code that uses some of the additional members, you'll need to make the implementation more complex. Anyway, a simple stream that implements only Read can look like this:
type SeqStream<'a>(sequence:seq<'a>, formatter:'a -> byte[]) =
inherit Stream()
// Keeps bytes that were read previously, but were not used
let temp = ResizeArray<_>()
// Enumerator for reading data from the sequence
let en = sequence.GetEnumerator()
override x.Read(buffer, offset, size) =
// Read next element and add it to temp until we have enough
// data or until we reach the end of the sequence
while temp.Count < size && en.MoveNext() do
temp.AddRange(formatter(en.Current))
// Copy data to the output & return count (may be less then
// required (at the end of the sequence)
let ret = min size temp.Count
temp.CopyTo(0, buffer, offset, ret)
temp.RemoveRange(0, ret)
ret
override x.Seek(offset, dir) = invalidOp "Seek"
override x.Flush() = invalidOp "Flush"
override x.SetLength(l) = invalidOp "SetLength"
override x.Length = invalidOp "Length"
override x.Position
with get() = invalidOp "Position"
and set(p) = invalidOp "Position"
override x.Write(buffer, offset, size) = invalidOp "Write"
override x.CanWrite = false
override x.CanSeek = false
override x.CanRead = true
Note that I added an additional parameter - a function to convert value of the generic type to a byte array. In general, it is difficult to convert anything to bytes (you could use some serialization), so this is probably easier. For example, for integers you can write:
let stream = new SeqStream<_>([ 1 .. 5 ], System.BitConverter.GetBytes)
Related
Functions in Dart are first-class objects, allowing you to pass them to other objects or functions.
void main() {
var shout = (msg) => ' ${msg.toUpperCase()} ';
print(shout("yo"));
}
This made me wonder if there was a way to modify a function a run time, just like an object, prior to passing it to something else. For example:
Function add(int input) {
return add + 2;
}
If I wanted to make the function a generic addition function, then I would do:
Function add(int input, int increment) {
return add + increment;
}
But then the problem would be that the object I am passing the function to would need to specify the increment. I would like to pass the add function to another object, with the increment specified at run time, and declared within the function body so that the increment cannot be changed by the recipient of the function object.
The answer seems to be to use a lexical closure.
From here: https://dart.dev/guides/language/language-tour#built-in-types
A closure is a function object that has access to variables in its
lexical scope, even when the function is used outside of its original
scope.
Functions can close over variables defined in surrounding scopes. In
the following example, makeAdder() captures the variable addBy.
Wherever the returned function goes, it remembers addBy.
/// Returns a function that adds [addBy] to the
/// function's argument.
Function makeAdder(int addBy) {
return (int i) => addBy + i;
}
void main() {
// Create a function that adds 2.
var add2 = makeAdder(2);
// Create a function that adds 4.
var add4 = makeAdder(4);
assert(add2(3) == 5);
assert(add4(3) == 7);
}
In the above cases, we pass 2 or 4 into the makeAdder function. The makeAdder function uses the parameter to create and return a function object that can be passed to other objects.
You most likely don't need to modify a closure, just the ability to create customized closures.
The latter is simple:
int Function(int) makeAdder(int increment) => (int value) => value + increment;
...
foo(makeAdder(1)); // Adds 1.
foo(makeAdder(4)); // Adds 2.
You can't change which variables a closure is referencing, but you can change their values ... if you an access the variable. For local variables, that's actually hard.
Mutating state which makes an existing closure change behavior can sometimes be appropriate, but those functions should be very precise about how they change and where they are being used. For a function like add which is used for its behavior, changing the behavior is rarely a good idea. It's better to replace the closure in the specific places that need to change behavior, and not risk changing the behavior in other places which happen to depend on the same closure. Otherwise it becomes very important to control where the closure actually flows.
If you still want to change the behavior of an existing global, you need to change a variable that it depends on.
Globals are easy:
int increment = 1;
int globalAdder(int value) => value + increment;
...
foo(globalAdd); // Adds 1.
increment = 2;
foo(globalAdd); // Adds 2.
I really can't recommend mutating global variables. It scales rather badly. You have no control over anything.
Another option is to use an instance variable to hold the modifiable value.
class MakeAdder {
int increment = 1;
int instanceAdd(int value) => value + increment;
}
...
var makeAdder = MakeAdder();
var adder = makeAdder.instanceAdd;
...
foo(adder); // Adds 1.
makeAdder.increment = 2;
foo(adder); // Adds 2.
That gives you much more control over who can access the increment variable. You can create multiple independent mutaable adders without them stepping on each other's toes.
To modify a local variable, you need someone to give you access to it, from inside the function where the variable is visible.
int Function(int) makeAdder(void Function(void Function(int)) setIncrementCallback) {
var increment = 1;
setIncrementCallback((v) {
increment = v;
});
return (value) => value + increment;
}
...
void Function(int) setIncrement;
int Function(int) localAdd = makeAdder((inc) { setIncrement = inc; });
...
foo(localAdd); // Adds 1.
setIncrement(2);
foo(localAdd); // Adds 2.
This is one way of passing back a way to modify the local increment variable.
It's almost always far too complicated an approach for what it gives you, I'd go with the instance variable instead.
Often, the instance variable will actually represent something in your model, some state which can meaningfully change, and then it becomes predictable and understandable when and how the state of the entire model changes, including the functions referring to that model.
Using partial function application
You can use a partial function application to bind arguments to functions.
If you have something like:
int add(int input, int increment) => input + increment;
and want to pass it to another function that expects to supply fewer arguments:
int foo(int Function(int input) applyIncrement) => applyIncrement(10);
then you could do:
foo((input) => add(input, 2); // `increment` is fixed to 2
foo((input) => add(input, 4); // `increment` is fixed to 4
Using callable objects
Another approach would be to make a callable object:
class Adder {
int increment = 0;
int call(int input) => input + increment;
}
which could be used with the same foo function above:
var adder = Adder()..increment = 2;
print(foo(adder)); // Prints: 12
adder.increment = 4;
print(foo(adder)); // Prints: 14
I'm trying to draw some boxes in Rascal and trying to give each box its own callback function. On entering the box with the mouse the corresponding string should get displayed in the text element (so hovering box1 should display box1 etc.).
However, at the moment the text does pop up but just displays "box3" for each of the 3 boxes.
Any ideas?
strings = ["box1", "box2", "box3"];
boxes = [ box(
size(100, 100),
onMouseEnter(void() {
output = s;
})
) | s <- strings];
render(hcat([
vcat(boxes),
text(str () {return output;})
]));
Good question, classical problem. The essence of the problem is that Rascal uses "non-capturing closures": this means that functions that are returned from another function share the same context. In your case this is the variable s introduced by s <- strings. This nearly always happens when you create function values in a loop (as you do here). The solution is to wrap another function layer around the returned function.
Here is a simple example:
list[int()] makeClosures()
= [ int() {return i;} | i <- [0,1,2]];
void wrong(){
lst = makeClosures();
println(lst[0]());
println(lst[1]());
println(lst[2]());
}
which will print surprisingly the values 2,2and2`. The solution is, as said, to introduce another function level:
int() makeClosure(int i)
= int() { return i;};
list[int()] makeClosuresOK()
= [ makeClosure(i) | i <- [0,1,2]];
void right(){
lst = makeClosuresOK();
println(lst[0]());
println(lst[1]());
println(lst[2]());
}
now calling right() will print 1, 2, and 3 as expected.
I leave it as an exercise how this is done in your example, but I am prepared to give a solution when you ask for it. Good luck!
If I want to combine multiple same-typed streams into one, I would use Stream::select:
let combined = first_stream.select(second_stream)
However, once one of the streams is exhausted, the other can still produce results for the combined stream. What can I use to exhaust the combined stream once either of the underlying streams is exhausted?
Write your own stream combinator:
use futures::{Async, Poll, Stream}; // 0.1.25
struct WhileBoth<S1, S2>(S1, S2)
where
S1: Stream,
S2: Stream<Item = S1::Item, Error = S1::Error>;
impl<S1, S2> Stream for WhileBoth<S1, S2>
where
S1: Stream,
S2: Stream<Item = S1::Item, Error = S1::Error>,
{
type Item = S1::Item;
type Error = S1::Error;
fn poll(&mut self) -> Poll<Option<Self::Item>, Self::Error> {
match self.0.poll() {
// Return errors or ready values (including the `None`
// that indicates the stream is empty) immediately.
r # Err(_) | r # Ok(Async::Ready(_)) => r,
// If the first stream is not ready, try the second one.
Ok(Async::NotReady) => self.1.poll(),
}
}
}
See also:
How can I add new methods to Iterator?
I'm attempting to build a ROM-based Window function using DSPComplex and FixedPoint types, but seem to keep running into the following error:
chisel3.core.Binding$ExpectedHardwareException: vec element 'dsptools.numbers.DspComplex#32' must be hardware, not a bare Chisel type
The source code for my attempt at this looks like the following:
class TaylorWindow(len: Int, window: Seq[FixedPoint]) extends Module {
val io = IO(new Bundle {
val d_valid_in = Input(Bool())
val sample = Input(DspComplex(FixedPoint(16.W, 8.BP), FixedPoint(16.W, 8.BP)))
val windowed_sample = Output(DspComplex(FixedPoint(24.W, 8.BP), FixedPoint(24.W, 8.BP)))
val d_valid_out = Output(Bool())
})
val win_coeff = Vec(window.map(x=>DspComplex(x, FixedPoint(0, 16.W, 8.BP))).toSeq) // ROM storing our coefficients.
io.d_valid_out := io.d_valid_in
val counter = Reg(UInt(10.W))
// Implicit reset
io.windowed_sample:= io.sample * win_coeff(counter)
when(io.d_valid_in) {
counter := counter + 1.U
}
}
println(getVerilog(new TaylorWindow(1024, fp_seq)))
I'm actually reading the coefficients in from a file (this particular window has a complex generation function that I'm doing in Python elsewhere) with the following sequence of steps
val filename = "../generated/taylor_coeffs"
val coeff_file = Source.fromFile(filename).getLines
val double_coeffs = coeff_file.map(x => x.toDouble)
val fp_coeffs = double_coeffs.map(x => FixedPoint.fromDouble(x, 16.W, 8.BP))
val fp_seq = fp_coeffs.toSeq
Does this mean the DSPComplex type isn't able to be translated to Verilog?
Commenting out the win_coeff line seems to make the whole thing generate (but clearly doesn't do what I want it to do)
I think you should try using
val win_coeff = VecInit(window.map(x=>DspComplex.wire(x, FixedPoint.fromDouble(0.0, 16.W, 8.BP))).toSeq) // ROM storing our coefficients.
which will create hardware values like you want. The Vec just creates a Vec of the type specfied
Imagine a gfor with a seq j...
If I need to use the value of the instance j as a index, who can I do that?
something like:
vector<double> a(n);
gfor(seq j, n){
//Do some calculation and save this on someValue
a[j] = someValue;
}
Someone can help me (again) ?
Thanks.
I've found a solution for this...
if someone had a better option, feel free to post...
First, create a seq with the same size of your gfor instances.
Then, convert that seq in a array.
Now, take the value of that line on array (it's equals the index)
seq sequencia(0, 200);
af::array sqc = sequencia;
//Inside the gfor loop
countLoop = (int) sqc(j).scalar<float>();
Your approach works, but breaks gfors parallelization as converting the index to a scalar forces it to be written from the gpu back to the host, slamming the breaks on the GPU.
You want to do it more like this :
af::array a(200);
gfor(seq j, 200){
//Do some calculation and save this on someValue
a[j] = af::array(someValue); // for someValue a primitive type, say float
}
// ... Now we're safe outside the parallel loop, let's grab the array results
float results[200];
a.host(results) // Copy array from GPU to host, populating a c-type array