How can I read a file in zig, and run over it line by line?
I did found os.File.openRead, but it seems old cause it says that container 'std.os' has no member called 'File'.
std.io.reader.readUntilDelimiterOrEof lets your read any std.io.reader line by line. You usually get the reader of something like a file by calling it’s reader() method. So for example:
var file = try std.fs.cwd().openFile("foo.txt", .{});
defer file.close();
var buf_reader = std.io.bufferedReader(file.reader());
var in_stream = buf_reader.reader();
var buf: [1024]u8 = undefined;
while (try in_stream.readUntilDelimiterOrEof(&buf, '\n')) |line| {
// do something with line...
}
The std.io.bufferedReader isn’t mandatory but recommended for better performance.
I muddled through this by looking at the Zig library source/docs, so this might not be the most idiomatic way:
const std = #import("std");
pub fn main() anyerror!void {
// Get an allocator
var gp = std.heap.GeneralPurposeAllocator(.{ .safety = true }){};
defer _ = gp.deinit();
const allocator = &gp.allocator;
// Get the path
var path_buffer: [std.fs.MAX_PATH_BYTES]u8 = undefined;
const path = try std.fs.realpath("./src/main.zig", &path_buffer);
// Open the file
const file = try std.fs.openFileAbsolute(path, .{ .read = true });
defer file.close();
// Read the contents
const buffer_size = 2000;
const file_buffer = try file.readToEndAlloc(allocator, buffer_size);
defer allocator.free(file_buffer);
// Split by "\n" and iterate through the resulting slices of "const []u8"
var iter = std.mem.split(file_buffer, "\n");
var count: usize = 0;
while (iter.next()) |line| : (count += 1) {
std.log.info("{d:>2}: {s}", .{ count, line });
}
}
The above is a little demo program that you should be able to drop into the default project created from zig init-exe, it'll just print out it's own contents, with a line number.
You can also do this without allocators, provided you supply the required buffers.
I'd also recommend checking out this great resource: https://ziglearn.org/chapter-2/#readers-and-writers
Note: I'm currently running a development version of Zig from master (reporting 0.9.0), but I think this has been working for the last few official releases.
To open a file and get a file descriptor back
std.os.open
https://ziglang.org/documentation/0.6.0/std/#std;os.open
To read from the file
std.os.read
https://ziglang.org/documentation/0.6.0/std/#std;os.read
I can't find a .readlines() style function in the zig standard library. You'll have to write your own loop to find the \n characters.
Below is a test case that shows how to create a file, write to it then open the same file and read its content.
const std = #import("std");
const testing = std.testing;
const expect = testing.expect;
test "create a file and then open and read it" {
var tmp_dir = testing.tmpDir(.{}); // This creates a directory under ./zig-cache/tmp/{hash}/test_file
// defer tmp_dir.cleanup(); // commented out this line so, you can see the file after execution finished.
var file1 = try tmp_dir.dir.createFile("test_file", .{ .read = true });
defer file1.close();
const write_buf: []const u8 = "Hello Zig!";
try file1.writeAll(write_buf);
var file2 = try tmp_dir.dir.openFile("test_file", .{});
defer file2.close();
const read_buf = try file2.readToEndAlloc(testing.allocator, 1024);
defer testing.allocator.free(read_buf);
try testing.expect(std.mem.eql(u8, write_buf, read_buf));
}
Check out fs package tests on Github or on your local machine under <zig-install-dir>/lib/fs/test.zig.
Also note that test allocator only works for tests. In your actual source code you need to choose an appropriate allocator.
Related
This code is supposed to create a memfd (anonymous file), copy shellcode as a Vec<u8>, then
finally execute using fexecve().
// A method that takes a u8 vector and copies it to a memfd_create file, then executes using fexecve()
use std::ffi::{CStr, CString};
use nix::sys::memfd::{memfd_create, MemFdCreateFlag};
use nix::unistd::fexecve;
use nix::unistd::write;
fn fileless_exec(code: Vec<u8>) {
// Name using CStr
let name = CStr::from_bytes_with_nul(b"memfd\0").unwrap();
// Create a new memfd file.
let fd = memfd_create(&name, MemFdCreateFlag::MFD_CLOEXEC).unwrap();
// Write to the file
let _nbytes = write(fd, &code);
// args for fexecve
let arg1 = CStr::from_bytes_with_nul(b"memfd\0").unwrap();
// enviroment variables
let env = CString::new("").unwrap();
// fexecve
let _ = match fexecve(fd, &[&arg1], &[&env]) {
Ok(_) => {
println!("Success!");
},
Err(e) => {
println!("Error: {}", e);
}
};
}
fn main() {
// Read the file `hello_world` into a vector of bytes.
let code = std::fs::read("/tmp/hello_world").unwrap();
fileless_exec(code);
}
(hello_world is just a simple C hello world example).
The binary executes and writes to stdout normally. How would I capture the output as, say, a String in Rust? I've seen this example do it in C which is ultimately what I'm trying to achieve here.
The whole point here is to execute a file using its fd and capture its output. The input could be coming from anywhere (not always from disk as with the hello_world executable): from a web endpoint, other processes, etc.
I'm aware this code isn't that "Rust"-y.
So following some very bad practices I was able to make this:
// A method that takes a u8 vector and copies it to a memfd_create file.
use std::ffi::{CStr, CString};
use nix::sys::memfd::{memfd_create, MemFdCreateFlag};
use nix::unistd::{read, write, fexecve, dup2, close, fork};
fn fileless_exec(code: Vec<u8>, fd_name: &[u8], stdout: &mut String) {
// Name using CStr
let name = CStr::from_bytes_with_nul(fd_name).unwrap();
// Create a new memfd file.
let fd = memfd_create(&name, MemFdCreateFlag::MFD_CLOEXEC).unwrap();
// Write to the file
let _nbytes = write(fd, &code);
// args for fexecve
let arg1 = CStr::from_bytes_with_nul(fd_name).unwrap();
// enviroment variables
let env = CString::new("").unwrap();
// to capture the output we need to use a pipe
let pipe = nix::unistd::pipe().unwrap();
unsafe {
let mut output = [0u8; 1024];
// fork and exec
let pid = fork().unwrap();
if pid.is_child() {
// dup the read end of the pipe to stdout
dup2(pipe.1, nix::libc::STDOUT_FILENO).unwrap();
// close the write end of the pipe
close(pipe.0).unwrap();
// close the read end of the pipe
close(pipe.1).unwrap();
// fexecve
fexecve(fd, &[&arg1], &[&env]).unwrap();
} else {
// close the read end of the pipe
close(pipe.1).unwrap();
// write to the pipe
let _nbytes = read(pipe.0, &mut output);
// close the write end of the pipe
close(pipe.0).unwrap();
// convert output to a string
*stdout = String::from_utf8(output.to_vec()).unwrap();
}
}
}
fn main() {
// Read the file `/bin/ls` into a vector of bytes.
let code = std::fs::read("/bin/ls").unwrap();
let mut output = String::new();
fileless_exec(code, b"anonymous\0", &mut output);
print!("File output: {}", output);
}
This works for now... thanks for answers
Somewhat old question, but I couldn't find a better answer anywhere. If you want to do this, there is now a crate memfd-exec to do exactly this!
For example (from the docs) we can download an execute a program without ever writing it to disk:
use memfd_exec::{MemFdExecutable, Stdio};
use reqwest::blocking::get;
const URL: &str = "https://novafacing.github.io/assets/qemu-x86_64";
let resp = get(URL).unwrap();
// The `MemFdExecutable` struct is at near feature-parity with `std::process::Command`,
// so you can use it in the same way. The only difference is that you must provide the
// executable contents as a `Vec<u8>` as well as telling it the argv[0] to use.
let qemu = MemFdExecutable::new("qemu-x86_64", resp.bytes().unwrap().to_vec())
// We'll just get the version here, but you can do anything you want with the
// args.
.arg("-version")
// We'll capture the stdout of the process, so we need to set up a pipe.
.stdout(Stdio::piped())
// Spawn the process as a forked child
.spawn()
.unwrap();
// Get the output and status code of the process (this will block until the process
// exits)
let output = qemu.wait_with_output().unwrap();
assert!(output.status.into_raw() == 0);
// Print out the version we got!
println!("{}", String::from_utf8_lossy(&output.stdout));
I am the author of memfd-exec.
Just starting with Deno, I am trying to figure out how to calculate a binary file checksum. It seems to me that the problem is not with the methods provided by the hash module of the standard library, but with the file streaming method and/or the type of the chunks feeding the hash.update method.
I have been trying a few alternatives, related to file opening and chunk types,with no success. A simple example is in the following:
import {createHash} from "https://deno.land/std#0.80.0/hash/mod.ts";
const file= new File(["my_big_folder.tar.gz"], "./my_big_folder.tar.gz");
const iterator = file.stream() .getIterator();
const hash = createHash("md5");
for await( let chunk of iterator){
hash.update(chunk);
}
console.log(hash.toString()); //b35edd0be7acc21cae8490a17c545928
This code compiles and runs with no errors, pity that the result is different from what I get running the functions of the crypto module provided by node and the md5sum provided by linux coreutils. Any suggestion ?
nodejs code:
const crypto = require('crypto');
const fs = require('fs');
const hash = crypto.createHash('md5');
const file = './my_big_folder.tar.gz';
const stream = fs.ReadStream(file);
stream.on('data', data=> { hash.update(data); });
stream.on('end', ()=> {
console.log(hash.digest('hex')); //c18f5eac67656328f7c4ec5d0ef5b96f
});
The same result in bash:
$ md5sum ./my_big_folder.tar.gz
$ c18f5eac67656328f7c4ec5d0ef5b96f ./my_big_folder.tar.gz
on Windows 10 this can be used:
CertUtil -hashfile ./my_big_folder.tar.gz md5
The File API isn't used to read a File in Deno, to do that you need to use the Deno.open API and then turn it into an iterable like this
import {createHash} from "https://deno.land/std#0.80.0/hash/mod.ts";
const hash = createHash("md5");
const file = await Deno.open(new URL(
"./BigFile.tar.gz",
import.meta.url, //This is needed cause JavaScript paths are relative to main script not current file
));
for await (const chunk of Deno.iter(file)) {
hash.update(chunk);
}
console.log(hash.toString());
Deno.close(file.rid);
import { crypto, toHashString } from 'https://deno.land/std#0.176.0/crypto/mod.ts';
const getFileBuffer = (filePath: string) => {
const file = Deno.openSync(filePath);
const buf = new Uint8Array(file.statSync().size);
file.readSync(buf);
file.close();
return buf;
};
const getMd5OfBuffer = (data: BufferSource) => toHashString(crypto.subtle.digestSync('MD5', data));
export const getFileMd5 = (filePath: string) => getMd5OfBuffer(getFileBuffer(filePath));
We are implementing PDFJS to render pdf files on a website.
When trying to initiate a PDFdocument/Viewer as an arrayBuffer, we get al sorts of errors and the file is not rendered.
When opening the same file in the viewer from url (DEFAULT_URL variable), the file renders fine.
There are however some files that do render as streams. Comparing these files in notepad shows they have different encoding/characters.
This piece of code is used to open the file in the viewer:
function rawStringToBuffer( str ) {
var idx, len = str.length, arr = new Array( len );
for ( idx = 0 ; idx < len ; ++idx ) {
arr[ idx ] = str.charCodeAt(idx) & 0xFF;
}
return new Uint8Array( arr ).buffer;
}
function readSingleFile(e) {
var file = e.target.files[0];
if (!file) {
return;
}
var reader = new FileReader();
reader.onload = function(e) {
var contents = e.target.result;
var uint8array = rawStringToBuffer(contents);
pdfjsframe.contentWindow.PDFViewerApplication.open(uint8array,0);
};
reader.readAsText(file);
}
test.pdf helloworld pdf which is not rendered with code above.
test2.pdf helloworld pdf which does rendered with code above.
The behaviour is not browser dependent. The build is b15f335.
Is there something with the code or default configuration of the viewer so that test.pdf can not be rendered by the viewer?
I don't think that your string conversion routine rawStringToBuffer() does what you want. You are reading the file as text, which transforms UTF-8 to UTF-16. But rawStringToBuffer() just takes the low order byte of each UTF-16 character and discards the high order byte, which is not the inverse transform. This will work with 7-bit ASCII data, but not with other characters. The best way to convert a string to UTF-8 is with the TextEncoder API (not supported on all browsers but polyfills are available).
However, converting the data from UTF-8 and back again is unnecessary. Just use FileReader.readAsArrayBuffer() instead of readAsText() to produce your ArrayBuffer directly.
Here's an (untested) replacement function:
function readSingleFile(e) {
var file = e.target.files[0];
if (!file) {
return;
}
var reader = new FileReader();
reader.onload = function(e) {
var contents = e.target.result;
pdfjsframe.contentWindow.PDFViewerApplication.open(contents, 0);
};
reader.readAsArrayBuffer(file);
}
I am currently working on ffsniff extension code. In that I have to save data containing password information into a file in my local system. I have written my code but it is not even creating the file in my local system. (working in mozilla firefox)
Here is my code please help me out.
//// here data variable contains all the information
var fso = new ActiveXObject("Scripting.FileSystemObject");
varFileObject = fso.OpenTextFile("C:\\logs.txt", 2, true,0);
varFileObject.write(data);
varFileObject.close();
after this i tried different code:
Components.utils.import("resource://gre/modules/NetUtil.jsm");
Components.utils.import("resource://gre/modules/FileUtils.jsm");
var file = Components.classes["#mozilla.org/file/directory_service;1"].
getService(Components.interfaces.nsIProperties).
get("Desk", Components.interfaces.nsIFile);
file.append("logs.txt");
var ostream = FileUtils.openSafeFileOutputStream(file)
var converter = Components.classes["#mozilla.org/intl/scriptableunicodeconverter"].
createInstance(Components.interfaces.nsIScriptableUnicodeConverter);
converter.charset = "UTF-8";
var istream = converter.convertToInputStream(data);
}
});
but none of them is working..
Here's a working snippet that creates the destination directory if necessary and writes (overwrites) to file (in this case d:\temp-directory\temp-file.txt):
var {Cc,Ci,Cu}=require("chrome"); //for jetpack sdk.
Cu.import("resource://gre/modules/NetUtil.jsm");
Cu.import("resource://gre/modules/FileUtils.jsm");
var localFile = Cc["#mozilla.org/file/local;1"].createInstance(Ci.nsILocalFile);
var data="test file content";
//localFile.initWithPath("D:\\temp-directory\\temp-file.txt"); //full path is okay if directory exists
localFile.initWithPath("D:\\temp-directory\\"); //otherwise specifiy directory, create it if necessary, and append leaf.
if(!localFile.exists()){
localFile.create(localFile.DIRECTORY_TYPE,FileUtils.PERMS_DIRECTORY);
}
localFile.append("temp-file.txt");
//localFile.createUnique(localFile.NORMAL_FILE_TYPE,FileUtils.PERMS_FILE); //optional: create a new unique file.
asyncSave(localFile,data,onDone);
function asyncSave(file,data,callbackDone){
// file is nsIFile, data is a string, optional: callbackDone(path,leafName,statusCode)
// default flags: FileUtils.openSafeFileOutputStream(file, FileUtils.MODE_WRONLY | FileUtils.MODE_CREATE | FileUtils.MODE_TRUNCATE);
var ostream = FileUtils.openSafeFileOutputStream(file);
var converter = Cc["#mozilla.org/intl/scriptableunicodeconverter"].createInstance(Ci.nsIScriptableUnicodeConverter);
converter.charset = "UTF-8";
var istream = converter.convertToInputStream(data);
// optional: callbackSaved(status).
NetUtil.asyncCopy(istream, ostream, callbackSaved);
function callbackSaved (status) {
if(callbackDone){
if(status===0)callbackDone( file.path, file.leafName, status); //sucess.
else callbackDone( null, null, status); //failure.
};
}
}
function onDone(path,leafName,statusCode){
console.log([statusCode===0?"OK":"error",path,leafName].join("\n"));
}
More information:
https://developer.mozilla.org/en-US/docs/Code_snippets/File_I_O
https://developer.mozilla.org/en-US/docs/Mozilla/JavaScript_code_modules/FileUtils.jsm
https://developer.mozilla.org/en-US/docs/PR_Open
https://developer.mozilla.org/en-US/docs/Mozilla/JavaScript_code_modules/NetUtil.jsm
https://developer.mozilla.org/en-US/docs/XPCOM_Interface_Reference/nsIFile
https://developer.mozilla.org/en-US/docs/XPCOM_Interface_Reference/nsILocalFile
A simple example of how to read/write a file from the filesystem in windows, using Firefox Extension:
// Write File to filesystem
Components.utils.import("resource://gre/modules/osfile.jsm"); // load the OS module
var encoder = new TextEncoder(); // This encoder can be reused for several writes
var array = encoder.encode("just some text"); // Convert the text to an array
var promise = OS.File.writeAtomic("C:\\foo.txt", array,{tmpPath: "foo.txt.tmp"}); // Write the array atomically to "file.txt", using as temporary
alert("URL HOST has been saved");
//Read File from filesystem
var decoder = new TextDecoder(); // This decoder can be reused for several reads
var promise = OS.File.read("C:\\foo.txt"); // Read the complete file as an array
promise = promise.then(
function onSuccess(array) {
alert(decoder.decode(array)); // Convert this array to a text
}
);
This solution is for making file in ubuntu, hope this helps others:
var file = Components.classes["#mozilla.org/file/directory_service;1"].
getService(Components.interfaces.nsIProperties).
get("ProfD", Components.interfaces.nsIFile);
file.append("trick_new");
if( !file.exists() || !file.isDirectory() ) { // if it doesn't exist, create
file.create(Components.interfaces.nsIFile.DIRECTORY_TYPE, 0777);
}
this.log_file = file.path + "/newlog.html";
You can also use text-stream to write to a local file.
function writeTextToFile(text, filename) {
var fileIO = require("sdk/io/file");
var TextWriter = fileIO.open(filename, "w");
if (!TextWriter.closed) {
TextWriter.write(text);
TextWriter.close();
}
}
How does one 'read' a file from a Dart program ?
http://api.dartlang.org/index.html
Dart would be running on the client-side and so taking files as input should be allowed.
You can find a usage of files in Dart's testing framework:
status_file_parser.dart (search for 'File').
In short:
File file = new File(path);
if (!file.existsSync()) <handle missing file>;
InputStream file_stream = file.openInputStream();
StringInputStream lines = new StringInputStream(file_stream);
lines.lineHandler = () {
String line;
while ((line = lines.readLine()) != null) {
...
};
lines.closeHandler = () {
...
};
Note that the API is not yet finalized and could change at any moment.
Edit: API has changed. See Introduction to new IO
Your question implies you want to do this from the client-side, that is, the browser. The dart:io library only works in the stand-alone VM on the command line.
If you do want to read a file from within the VM, there's now an easier way:
import 'dart:io';
main() {
var filename = new Options().script;
var file = new File(filename);
if (!file.existsSync()) {
print("File $filename does not exist");
return;
}
var contents = file.readAsStringSync();
print(contents);
}
If you do not want to block while the whole file is read, you can use the async version of readAsString which returns a Future:
file.readAsString().then((contents) {
print(contents);
});