how pipe() will performing read and write operations in node js? - stream

var fs = require("fs");
var readStream= fs.createReadStream('input.txt');
var writeStream=fs.createWriteStream('output.txt');
readStream.pipe(writeStream);
console.log("hello...world");
In the above program, readStream and writeStream operations are done by pipe function..what was the internal code and internal process happening in the pipe()?

piping will performs like a bridge in nodejs.
review here: https://nodejs.org/api/stream.html#stream_event_pipe

Related

Can Python gRPC do computation when sending messages out?

Suppose I need to send a large amount of data from the client to the server using python gRPC. And I want to continue the rest computation when sending the message out instead of blocking the code. Is there any way can implement this?
I will illustrate the question by an example using the modified code from the greeter_client.py
for i in range(5):
res=computation()
response = stub.SayHello(helloworld_pb2.HelloRequest(data=res))
I want the computation of the next iteration continue while sending the "res" of last iteration. To this end, I have tried the "async/await", which looks like this
async with aio.insecure_channel('localhost:50051') as channel:
stub = helloworld_pb2_grpc.GreeterStub(channel)
for j in range(5):
res=computation()
response = await stub.SayHello(helloworld_pb2.HelloRequest(data=res))
But the running time is actually the same with the version without async/await. The async/await does not work. I am wondering is there anything wrong in my codes or there are other ways?
Concurrency is different than parallelism. AsyncIO allows multiple coroutines to run on the same thread, but they are not actually computed at the same time. If the thread is given a CPU-heavy work like "computation()" in your snippet, it doesn't yield control back to the event loop, hence there won't be any progress on other coroutines.
Besides, in the snippet, the RPC depends on the result of "computation()", this meant the work will be serialized for each RPC. But we can still gain some concurrency from AsyncIO, by handing them over to the event loop with asyncio.gather():
async with aio.insecure_channel('localhost:50051') as channel:
stub = helloworld_pb2_grpc.GreeterStub(channel)
async def one_hello():
res=computation()
response = await stub.SayHello(helloworld_pb2.HelloRequest(data=res))
await asyncio.gather(*(one_hello() for _ in range(5)))

How to parallelize HTTP requests within an Apache Beam step?

I have an Apache Beam pipeline running on Google Dataflow whose job is rather simple:
It reads individual JSON objects from Pub/Sub
Parses them
And sends them via HTTP to some API
This API requires me to send the items in batches of 75. So I built a DoFn that accumulates events in a list and publish them via this API once they I get 75. This results to be too slow, so I thought instead of executing those HTTP requests in different threads using a thread pool.
The implementation of what I have right now looks like this:
private class WriteFn : DoFn<TheEvent, Void>() {
#Transient var api: TheApi
#Transient var currentBatch: MutableList<TheEvent>
#Transient var executor: ExecutorService
#Setup
fun setup() {
api = buildApi()
executor = Executors.newCachedThreadPool()
}
#StartBundle
fun startBundle() {
currentBatch = mutableListOf()
}
#ProcessElement
fun processElement(processContext: ProcessContext) {
val record = processContext.element()
currentBatch.add(record)
if (currentBatch.size >= 75) {
flush()
}
}
private fun flush() {
val payloadTrack = currentBatch.toList()
executor.submit {
api.sendToApi(payloadTrack)
}
currentBatch.clear()
}
#FinishBundle
fun finishBundle() {
if (currentBatch.isNotEmpty()) {
flush()
}
}
#Teardown
fun teardown() {
executor.shutdown()
executor.awaitTermination(30, TimeUnit.SECONDS)
}
}
This seems to work "fine" in the sense that data is making it to the API. But I don't know if this is the right approach and I have the sense that this is very slow.
The reason I think it's slow is that when load testing (by sending a few million events to Pub/Sub), it takes it up to 8 times more time for the pipeline to forward those messages to the API (which has response times of under 8ms) than for my laptop to feed them into Pub/Sub.
Is there any problem with my implementation? Is this the way I should be doing this?
Also... am I required to wait for all the requests to finish in my #FinishBundle method (i.e. by getting the futures returned by the executor and waiting on them)?
You have two interrelated questions here:
Are you doing this right / do you need to change anything?
Do you need to wait in #FinishBundle?
The second answer: yes. But actually you need to flush more thoroughly, as will become clear.
Once your #FinishBundle method succeeds, a Beam runner will assume the bundle has completed successfully. But your #FinishBundle only sends the requests - it does not ensure they have succeeded. So you could lose data that way if the requests subsequently fail. Your #FinishBundle method should actually be blocking and waiting for confirmation of success from the TheApi. Incidentally, all of the above should be idempotent, since after finishing the bundle, an earthquake could strike and cause a retry ;-)
So to answer the first question: should you change anything? Just the above. The practice of batching requests this way can work as long as you are sure the results are committed before the bundle is committed.
You may find that doing so will cause your pipeline to slow down, because #FinishBundle happens more frequently than #Setup. To batch up requests across bundles you need to use the lower-level features of state and timers. I wrote up a contrived version of your use case at https://beam.apache.org/blog/2017/08/28/timely-processing.html. I would be quite interested in how this works for you.
It may simply be that the extremely low latency you are expecting, in the low millisecond range, is not available when there is a durable shuffle in your pipeline.

How does addressing work in devicetree for a Xilinx CDMA?

Background:
What I want to do is to be able to write from my ARM processor to a BRAM, on a Zynq 7000.
To do this, I have the following components:
-M_AXI_GP0 on PS7 connects to S_AXI_LITE on axi_cdma_0 through an AXI Interconnect
-cdma_introut on axi_cdma_0 connects to IRQ_F2P on PS7 through sys_concat, input 11. This means that this maps to Interrupt 87 on PS7.
-M_AXI on axi_cdma_0 connects to S00_AXI on axi_mem_intercon
-M01_AXI on axi_mem_intercon connects to S_AXI_HP3 on PS7
-M00_AXI on axi_mem_intercon connects to S_AXI on axi_bram_ctrl_0
-BRAM_PORTA on axi_bram_ctrl_0 connects to BRAM_PORTA on blk_mem_gen0
=========================================================================
In my mind, what this setup ought to do is this:
Once a transaction is submitted from the ARM DMA Engine, the Zynq will use GP0 to send a command to the CDMA controller via GP0.
The CDMA controller will receive the commands on its slave AXI_LITE port, and interpret the request to access RAM via HP3.
The CDMA controller will move data through axi_mem_intercon in order to take the transaction data from hp3 on M01_AXI, and send it through M00_AXI to the BRAM Controller
The BRAM controller will take in the AXI-4 input and convert that to the appropriate BRAM port to write the data into the BRAM generated by blk_mem_gen_0
After completing this action, the CDMA will send an interrupt through sys_concat to indicate to the DMA Engine that its work is complete.
After loading this hdl design into the PL fabric, I attempt to submit the transaction to the DMA engine via a kernel module. The result is a timeout, with the DMA engine apparently never finishing the task.
=========================================================================
In my attempts to figure out the problem, I've made these observations:
After attempting a write transaction, which times out, I attempted a read transaction to the same DMA channel, but configured to read data. What I get back is all the data that I had attempted to write. This, to me, seems to indicate that the DMA engine IS writing to somewhere, but isn't recognizing the completion of the task
The BRAM in question is a dual port RAM, and the other port reads the data in the BRAM and toggles LEDs to reflect the data. The LEDs are not toggling when I attempt this write transaction, so it seems as though the DMA transaction is not making it as far as the BRAM
When looking at cat /proc/interrupts, I can see several interrupts, but not GIC 87. As mentioned before, the interrupt line I am using goes to Input 11 of the IRQ concat block. I can confirm that the interrupt line which goes to Input 12 does indeed correspond to GIC 88 from /proc/interrupts, so I believe my understanding of which interrupt I am looking for is correct. So for some reason it is not registering that interrupt on the processor.
=========================================================================
Based on this, I believe my devicetree entry for this CDMA is what is incorrect.
In Vivado, I can see these entries in the Address Editor(Some entries omitted for brevity):
sys_ps7
Data(32 address bits:0x40000000 [1G])
axi_cdma_0 S_AXI_LITE Reg 0x43C0_0000 64K 0x43C0_FFFF
axi_cdma_0
Data(32 address bits : 4G)
axi_bram_ctrl_0 S_AXI Mem0 0xC000_0000 4K 0xC000_0FFF
sys_ps7 S_AXI_HP3 HP3... 0x0000_0000 1G 0x3FFF_FFFF
My attempt to write a devicetree entry is as follows:
axi-cdma#43C00000{
#dma-cells = <0x1>;
compatible = "tst,axi-cdma-ctrl-1.00.a";
reg = <0x10000000 0x1000>;
interrupts = <0x0 0x37 0x4>;
interrupt-parent = <0x1>;
dma-channel#C0000000{
buswidth = <0x20>;
}
Before I added this entry in my kernel module failed to even register a transaction channel, and now it does, so I am fairly certain that the kernel is accepting this entry at least enough to assign a DMA channel. However, I don't understand much about how exactly the devicetree works, specifically with the addressing, so there is a good chance I have written this incorrectly, and that is why my transaction doesn't succeed. Can anyone help me correct my design?
}
Declaring the IP core in device tree is not sufficient. You must also declare your DMA client, as Xilinx does in CDMA test client:
cdmatest_1: cdmatest#1 {
compatible ="xlnx,axi-cdma-test-1.00.a";
dmas = <&axi_cdma_0 0>;
dma-names = "cdma";
} ;
In dmas field, the axi_cdma_0 references the CDMA IP core and the 0 its first dma-channel, as defined in the devicetree:
axi_cdma_0: dma#4e200000 {
#dma-cells = <1>;
clock-names = "s_axi_lite_aclk", "m_axi_aclk";
clocks = <&clkc 15>, <&clkc 15>;
compatible = "xlnx,axi-cdma-1.00.a";
interrupt-parent = <&intc>;
interrupts = <0 31 4>;
reg = <0x4e200000 0x10000>;
xlnx,addrwidth = <0x20>;
xlnx,include-sg ;
dma-channel#4e200000 {
compatible = "xlnx,axi-cdma-channel";
interrupts = <0 31 4>;
xlnx,datawidth = <0x20>;
xlnx,device-id = <0x0>;
xlnx,include-dre ;
xlnx,max-burst-len = <0x10>;
};
};
After that, you should register your client as a platform driver. Again, from CDMA test client source:
static const struct of_device_id xilinx_cdmatest_of_ids[] = {
{ .compatible = "xlnx,axi-cdma-test-1.00.a", },
{ }
};
static struct platform_driver xilinx_cdmatest_driver = {
.driver = {
.name = "xilinx_cdmatest",
.owner = THIS_MODULE,
.of_match_table = xilinx_cdmatest_of_ids,
},
.probe = xilinx_cdmatest_probe,
.remove = xilinx_cdmatest_remove,
};
static int __init cdma_init(void)
{
return platform_driver_register(&xilinx_cdmatest_driver);
}
Please note the compatible field of device tree and of platform driver definition, these strings must match. If you did not do this, the dma_request_slave_channel() cannot reserve a channel from your CDMA IP core. Moreover, ensure you do not use dma_request_channel() which is not supported in xilinx kernel >= 4.0 and will fail to reserve channels properly, the transfer will not complete and the DMA will timeout with no interrupt. I am not sure about observation 1, it might be a caching effect. Try to use dma_alloc_coherent() instead of kmalloc().
PS: In any case, try to make sure your hardware is ok by using a bare metal app if possible.

How do I write data to stdin of child_process in Firefox Add-on SDK?

The child_process Add-on SDK docs state that the child_process module implements the node.js child_process API. So, I'm trying to spawn a sub-process and write data to it as described in the node.js docs section covering the spawn method:
var child_process = require("sdk/system/child_process");
var doiuse = child_process.spawn("/usr/local/bin/doiuse");
doiuse.stdin.write(data);
But I get this error:
TypeError: doiuse.stdin.write is not a function
So, how do I write to a child process in the Firefox Add-on SDK?
In the #jetpack IRC channel, it was suggested to use code like the child_process tests. So, I got it working with:
const { emit } = require('sdk/event/core');
const { spawn } = require('sdk/system/child_process');
var proc = spawn("/bin/cat");
emit(proc.stdin, 'data', "Hello from Add-on code");
emit(proc.stdin, 'end');

Open external file with Electron

I have a running Electron app and is working great so far. For context, I need to run/open a external file which is a Go-lang binary that will do some background tasks.
Basically it will act as a backend and exposing an API that the Electron app will consume.
So far this is what i get into:
I tried to open the file with the "node way" using child_process but i have fail opening the a sample txt file probably due to path issues.
The Electron API expose a open-file event but it lacks of documentation/example and i don't know if it could be useful.
That's it.
How i open an external file in Electron ?
There are a couple api's you may want to study up on and see which helps you.
fs
The fs module allows you to open files for reading and writing directly.
var fs = require('fs');
fs.readFile(p, 'utf8', function (err, data) {
if (err) return console.log(err);
// data is the contents of the text file we just read
});
path
The path module allows you to build and parse paths in a platform agnostic way.
var path = require('path');
var p = path.join(__dirname, '..', 'game.config');
shell
The shell api is an electron only api that you can use to shell execute a file at a given path, which will use the OS default application to open the file.
const {shell} = require('electron');
// Open a local file in the default app
shell.openItem('c:\\example.txt');
// Open a URL in the default way
shell.openExternal('https://github.com');
child_process
Assuming that your golang binary is an executable then you would use child_process.spawn to call it and communicate with it. This is a node api.
var path = require('path');
var spawn = require('child_process').spawn;
var child = spawn(path.join(__dirname, '..', 'mygoap.exe'), ['game.config', '--debug']);
// attach events, etc.
addon
If your golang binary isn't an executable then you will need to make a native addon wrapper.
Maybe you are looking for this ?
dialog.showOpenDialog refer to: https://www.electronjs.org/docs/api/dialog
If using electron#13.1.0, you can do like this:
const { dialog } = require('electron')
console.log(dialog.showOpenDialog({ properties: ['openFile', 'multiSelections'] }))
dialog.showOpenDialog(function(file_paths){
console.info(file_paths) // => this gives the absolute path of selected files.
})
when the above code is triggered, you can see an "open file dialog" like this (diffrent view style for win/mac/linux)
Electron allows the use of nodejs packages.
In other words, import node packages as if you were in node, e.g.:
var fs = require('fs');
To run the golang binary, you can make use of the child_process module. The documentation is thorough.
Edit: You have to solve the path differences. The open-file event is a client-side event, triggered by the window. Not what you want here.
I was also totally struggling with this issue, and almost seven years later the documentation is quite not clear what's the case with Linux.
So, on Linux it falls under Windows treatment in this regard, which means you have to look into process.argv global in the main processor, the first value in the array is the path that fired the app. The second argument, if one exist, is holding the path that requested the app to be opened. For example, here is the output for my test case:
Array(2)
0: "/opt/Blueprint/b-test"
1: "/home/husayngonzalez/2022-01-20.md"
length: 2
So, when you're creating a new window, you check for the length of process.argv and then if it was more than 1, i.e. = 2 it means you have a path that requested to be opened with your app.
Assuming you got your application packaged with the ability to process those files, and also you set the operating system to request your application to open those.
I know this doesn't exactly meet your specification, but it does cleanly separate your golang binary and Electron application.
The way I have done it is to expose the golang binary as a web service. Like this
package main
import (
"fmt"
"net/http"
)
func handler(w http.ResponseWriter, r *http.Request) {
//TODO: put your call here instead of the Fprintf
fmt.Fprintf(w, "HI there from Go Web Svc. %s", r.URL.Path[1:])
}
func main() {
http.HandleFunc("/api/someMethod", handler)
http.ListenAndServe(":8080", nil)
}
Then from Electron just make ajax calls to the web service with a javascript function. Like this (you could use jQuery, but I find this pure js works fine)
function get(url, responseType) {
return new Promise(function(resolve, reject) {
var request = new XMLHttpRequest();
request.open('GET', url);
request.responseType = responseType;
request.onload = function() {
if (request.status == 200) {
resolve(request.response);
} else {
reject(Error(request.statusText));
}
};
request.onerror = function() {
reject(Error("Network Error"));
};
request.send();
});
With that method you could do something like
get('localhost/api/somemethod', 'text')
.then(function(x){
console.log(x);
}

Resources