I am trying to use thingsboard to allow users to request log files from devices. When requested, the devices send the log files to my TB server as telemetry. First, the logs are compressed with gzip and then base64 encoded. I want to have the rule chain decompress these logs and email to the requestor. I've found code to convert the base64 string to a byte array, but I haven't found a way to decompress the resulting byte array. I tried to invoke zlib using:
var zlib = require('zlib');
but it results in an message that 'require' is not defined.
Any hints? What language exactly is the TB rule node environment?
We send it to S3 and then have a link to it available on the TB gui - the user can request a log file(s) on the UX and then a few minutes later can click on the logfile and it downloads to their computer/device as a zip file. Device is linux based.
Related
I wanted to know for any Apache Tika Server instance, how does it handle files that are sent to it from a PUT request such as PDF files or image files when extracting text such as by OCR.
Are they stored at all to disk or are they read directly from the byte array? If they are stored to disk first before processing, are they cleaned up?
I have captured udp log files. I can create the pcap file with captured udp data from log file using pcapDotnet but the newly created pcap files im not able to open veloview.exe. Tool itself getting crashed but same pcap files i can open in wireshark.
Wireshark captures and opens network-level traffic (meaning all kind of packets)
Veloview expects packets generated by Velodyne Lidars, with a specific dataformat (it runs at application level). Please check that your pcap file contains the original datapacket format.
For now, Veloview only reads Velodyne generated data, saved as legacy ".pcap" files (no .pcap.ng, and newer formats). So please try saving in this format.
Best regards,
Bastien Jacquet, VeloView Lead Developer
How to download or view the files sent as multi-part Request (e.g. PUT) via a software tool?
Is there any way to accomplish this with a specific tool like CharlesProxy on macOSX, to download and view files that were sent as a part of request (PUT multipart request)? I typically fix such issues by saving the file to sandbox via code changes. Ideally, need something that can be used by our QA and doesn't require any code modification.
Charles Proxy on macos is sufficient for the most dev/QA needs, such as:
Throttle network
Device debugging
Download response data
...
However, there is no option present to view or download files in HTTP request in Charles Proxy 4.x:
Charles Proxy 4.x (and earlier) allows saving response files, example pdf in this screenshot:
This can be done by editing binary file manually. It's a bit tricky, but can save the file in multipart HTTP request, without any modification to project code.
Here are the steps (verified on Charles v4.2.8 and macOS v10.12.6):
Save request. Right click a recorded HTTP request (the one that send file), and click "Save Request...". This will save the whole HTTP request in binary format.
Inspect Hex representation of request. Left click that recorded HTTP request, and click "Hex" tab of "Request" panel. This will show the binary representation of request, together with some parsed text.
Edit the saved request. Open the saved request (step 1) with editor that support binary, such as Sublime Text. Then, remove all non-image binary code according to the result of step 2. Especially, remove every bytes before (and include) the first empty line (0d0a0d0a in macOS and Windows, 0a0a in Linux), and remove the tail bytes. For example, following screenshot indicates request bytes of step 2, the selected bytes would be deleted (please note the 0d0a bytes, as this experiment is taken on Mac):
...
Save image file. Save the file after step 3 is finished. Then, append filename extension according to the Content-Type value in step 2. In this experiment, the Content-Type is image/png, so .png is appended to the filename.
That's it. You can open the xxx.png file now. It's a pure image file.
Note: this experiment only contain 1 file, but the strategy works when there are multiple file upload in request.
I try to handle large files using Sync API for iOS
My program encrypts files in the stream, and I record the encrypted part (at ~ 320000 bytes) method
[file appendData: data error: nil];
file - variable format DBFile
data - variable format NSData, have size ~320000 bytes
When the file size reaches 80 MB program takes off and does not give an error
In the course of the program produces errors recieve memory warning, although no additional variables are not used
What are some ways to manage large files to dropbox api for ios (and download too)? Or what's my problem here?
I'm developing a download manager using Indy and Delphi XE (The application uses Multithreading to attempt several connections to the server). Everything works fine but sometimes the final downloaded file is broken and when I check downloaded temp files I see that 2 or 3 of them is filled with zero at their end. (Each temp file is download result of each connection).
The larger the file is, the more broken temp files I get as the result.
For example in one of the temp files which was 65,536,000 bytes, only the range of 0-34,359,426 was valid and from 34,359,427 to 64,535,999 it was full of zeros. If I delete those zeros, application will automatically download the missing segments and what I get as the result, well if the problem wouldn't happen again, is the healthy downloaded file.
I want to get rid of those zeros at the end of the temp files without having a lost in download speed.
P.S. I'm using TFileStream and I'm sending it directly to TIdHTTP and downloading the files using GET method.
Additional Info: I handle OnWork event which assigns AWorkCount to a public int64 variable. Each time the file is downloaded, the downloaded file size (That Int64 variable) is logged to a text file and from what the log says is that the file has been downloaded completely (even those zero bytes).
Make sure the server actually supports downloading byte ranges before you request a range to download. If the server does not support ranges, a requested range will be ignored by the server and the entire file will be sent instead. If you are not already doing so, you should be using TIdHTTP.Head() to text for range support before then calling TIdHTTP.Get(). You also need to do this anyway to detect if the remote file has been altered since the last time you downloaded it. Any decent download manager needs to be able to handle things like that.
Also keep in mind that if TIdHTTP knows up front how many bytes are being transferred, it will pre-allocate the size of the destination TStream before then downloading data into it. This is to speed up the transfer and optimize disc I/O when using a TFileStream. So you should NOT use TFileStream to access the same file as the destination for multiple simultaneous downloads, even if they are writing to different areas of the file. Pre-allocating multiple TFileStream objects will likely trample over each other trying to set the file size to different positions. If you need to download a file in multiple pieces simultaneously then either:
1) download each piece to a separate file and copy them into the final file as needed once you have all of the pieces that you need.
2) use a custom TStream class, or Indy's TIdEventStream class, to manage the file I/O yourself so you can ignore TIdHTTP's pre-allocation attempts and ensure that multiple file I/O operatons do not overlap each other incorrectly.