Capture output of fexecve on memfd_create fd - memory

This code is supposed to create a memfd (anonymous file), copy shellcode as a Vec<u8>, then
finally execute using fexecve().
// A method that takes a u8 vector and copies it to a memfd_create file, then executes using fexecve()
use std::ffi::{CStr, CString};
use nix::sys::memfd::{memfd_create, MemFdCreateFlag};
use nix::unistd::fexecve;
use nix::unistd::write;
fn fileless_exec(code: Vec<u8>) {
// Name using CStr
let name = CStr::from_bytes_with_nul(b"memfd\0").unwrap();
// Create a new memfd file.
let fd = memfd_create(&name, MemFdCreateFlag::MFD_CLOEXEC).unwrap();
// Write to the file
let _nbytes = write(fd, &code);
// args for fexecve
let arg1 = CStr::from_bytes_with_nul(b"memfd\0").unwrap();
// enviroment variables
let env = CString::new("").unwrap();
// fexecve
let _ = match fexecve(fd, &[&arg1], &[&env]) {
Ok(_) => {
println!("Success!");
},
Err(e) => {
println!("Error: {}", e);
}
};
}
fn main() {
// Read the file `hello_world` into a vector of bytes.
let code = std::fs::read("/tmp/hello_world").unwrap();
fileless_exec(code);
}
(hello_world is just a simple C hello world example).
The binary executes and writes to stdout normally. How would I capture the output as, say, a String in Rust? I've seen this example do it in C which is ultimately what I'm trying to achieve here.
The whole point here is to execute a file using its fd and capture its output. The input could be coming from anywhere (not always from disk as with the hello_world executable): from a web endpoint, other processes, etc.
I'm aware this code isn't that "Rust"-y.

So following some very bad practices I was able to make this:
// A method that takes a u8 vector and copies it to a memfd_create file.
use std::ffi::{CStr, CString};
use nix::sys::memfd::{memfd_create, MemFdCreateFlag};
use nix::unistd::{read, write, fexecve, dup2, close, fork};
fn fileless_exec(code: Vec<u8>, fd_name: &[u8], stdout: &mut String) {
// Name using CStr
let name = CStr::from_bytes_with_nul(fd_name).unwrap();
// Create a new memfd file.
let fd = memfd_create(&name, MemFdCreateFlag::MFD_CLOEXEC).unwrap();
// Write to the file
let _nbytes = write(fd, &code);
// args for fexecve
let arg1 = CStr::from_bytes_with_nul(fd_name).unwrap();
// enviroment variables
let env = CString::new("").unwrap();
// to capture the output we need to use a pipe
let pipe = nix::unistd::pipe().unwrap();
unsafe {
let mut output = [0u8; 1024];
// fork and exec
let pid = fork().unwrap();
if pid.is_child() {
// dup the read end of the pipe to stdout
dup2(pipe.1, nix::libc::STDOUT_FILENO).unwrap();
// close the write end of the pipe
close(pipe.0).unwrap();
// close the read end of the pipe
close(pipe.1).unwrap();
// fexecve
fexecve(fd, &[&arg1], &[&env]).unwrap();
} else {
// close the read end of the pipe
close(pipe.1).unwrap();
// write to the pipe
let _nbytes = read(pipe.0, &mut output);
// close the write end of the pipe
close(pipe.0).unwrap();
// convert output to a string
*stdout = String::from_utf8(output.to_vec()).unwrap();
}
}
}
fn main() {
// Read the file `/bin/ls` into a vector of bytes.
let code = std::fs::read("/bin/ls").unwrap();
let mut output = String::new();
fileless_exec(code, b"anonymous\0", &mut output);
print!("File output: {}", output);
}
This works for now... thanks for answers

Somewhat old question, but I couldn't find a better answer anywhere. If you want to do this, there is now a crate memfd-exec to do exactly this!
For example (from the docs) we can download an execute a program without ever writing it to disk:
use memfd_exec::{MemFdExecutable, Stdio};
use reqwest::blocking::get;
const URL: &str = "https://novafacing.github.io/assets/qemu-x86_64";
let resp = get(URL).unwrap();
// The `MemFdExecutable` struct is at near feature-parity with `std::process::Command`,
// so you can use it in the same way. The only difference is that you must provide the
// executable contents as a `Vec<u8>` as well as telling it the argv[0] to use.
let qemu = MemFdExecutable::new("qemu-x86_64", resp.bytes().unwrap().to_vec())
// We'll just get the version here, but you can do anything you want with the
// args.
.arg("-version")
// We'll capture the stdout of the process, so we need to set up a pipe.
.stdout(Stdio::piped())
// Spawn the process as a forked child
.spawn()
.unwrap();
// Get the output and status code of the process (this will block until the process
// exits)
let output = qemu.wait_with_output().unwrap();
assert!(output.status.into_raw() == 0);
// Print out the version we got!
println!("{}", String::from_utf8_lossy(&output.stdout));
I am the author of memfd-exec.

Related

How to read a file in zig?

How can I read a file in zig, and run over it line by line?
I did found os.File.openRead, but it seems old cause it says that container 'std.os' has no member called 'File'.
std.io.reader.readUntilDelimiterOrEof lets your read any std.io.reader line by line. You usually get the reader of something like a file by calling it’s reader() method. So for example:
var file = try std.fs.cwd().openFile("foo.txt", .{});
defer file.close();
var buf_reader = std.io.bufferedReader(file.reader());
var in_stream = buf_reader.reader();
var buf: [1024]u8 = undefined;
while (try in_stream.readUntilDelimiterOrEof(&buf, '\n')) |line| {
// do something with line...
}
The std.io.bufferedReader isn’t mandatory but recommended for better performance.
I muddled through this by looking at the Zig library source/docs, so this might not be the most idiomatic way:
const std = #import("std");
pub fn main() anyerror!void {
// Get an allocator
var gp = std.heap.GeneralPurposeAllocator(.{ .safety = true }){};
defer _ = gp.deinit();
const allocator = &gp.allocator;
// Get the path
var path_buffer: [std.fs.MAX_PATH_BYTES]u8 = undefined;
const path = try std.fs.realpath("./src/main.zig", &path_buffer);
// Open the file
const file = try std.fs.openFileAbsolute(path, .{ .read = true });
defer file.close();
// Read the contents
const buffer_size = 2000;
const file_buffer = try file.readToEndAlloc(allocator, buffer_size);
defer allocator.free(file_buffer);
// Split by "\n" and iterate through the resulting slices of "const []u8"
var iter = std.mem.split(file_buffer, "\n");
var count: usize = 0;
while (iter.next()) |line| : (count += 1) {
std.log.info("{d:>2}: {s}", .{ count, line });
}
}
The above is a little demo program that you should be able to drop into the default project created from zig init-exe, it'll just print out it's own contents, with a line number.
You can also do this without allocators, provided you supply the required buffers.
I'd also recommend checking out this great resource: https://ziglearn.org/chapter-2/#readers-and-writers
Note: I'm currently running a development version of Zig from master (reporting 0.9.0), but I think this has been working for the last few official releases.
To open a file and get a file descriptor back
std.os.open
https://ziglang.org/documentation/0.6.0/std/#std;os.open
To read from the file
std.os.read
https://ziglang.org/documentation/0.6.0/std/#std;os.read
I can't find a .readlines() style function in the zig standard library. You'll have to write your own loop to find the \n characters.
Below is a test case that shows how to create a file, write to it then open the same file and read its content.
const std = #import("std");
const testing = std.testing;
const expect = testing.expect;
test "create a file and then open and read it" {
var tmp_dir = testing.tmpDir(.{}); // This creates a directory under ./zig-cache/tmp/{hash}/test_file
// defer tmp_dir.cleanup(); // commented out this line so, you can see the file after execution finished.
var file1 = try tmp_dir.dir.createFile("test_file", .{ .read = true });
defer file1.close();
const write_buf: []const u8 = "Hello Zig!";
try file1.writeAll(write_buf);
var file2 = try tmp_dir.dir.openFile("test_file", .{});
defer file2.close();
const read_buf = try file2.readToEndAlloc(testing.allocator, 1024);
defer testing.allocator.free(read_buf);
try testing.expect(std.mem.eql(u8, write_buf, read_buf));
}
Check out fs package tests on Github or on your local machine under <zig-install-dir>/lib/fs/test.zig.
Also note that test allocator only works for tests. In your actual source code you need to choose an appropriate allocator.

How to pipe to a process using vala/glib

I'm trying to pipe output from echo into a command using GLib's spawn_command_line_sync method. The problem I've run into is echo is interpreting the entire command as the argument.
To better explain, I run this in my code:
string command = "echo \"" + some_var + "\" | command";
Process.spawn_command_line_sync (command.escape (),
out r, out e, out s);
I would expect the variable to be echoed to the pipe and the command run with the data piped, however when I check on the result it's just echoing everything after echo like this:
"some_var's value" | command
I think I could just use the Posix class to run the command but I like having the result, error and status values to listen to that the spawn_command_line_sync method provides.
The problem is that you are providing shell syntax to what is essentially the kernel’s exec() syscall. The shell pipe operator redirects the stdout of one process to the stdin of the next. To implement that using Vala, you need to get the file descriptor for the stdin of the command process which you’re running, and write some_var to it manually.
You are combining two subprocesses into one. Instead echo and command should be treated separately and have a pipe set up between them. For some reason many examples on Stack Overflow and other sites use the Process.spawn_* functions, but using GSubprocess is an easier syntax.
This example pipes the output of find . to sort and then prints the output to the console. The example is a bit longer because it is a fully working example and makes use of a GMainContext for asynchronous calls. GMainContext is used by GMainLoop, GApplication and GtkApplication:
void main () {
var mainloop = new MainLoop ();
SourceFunc quit = ()=> {
mainloop.quit ();
return Source.REMOVE;
};
read_piped_commands.begin ("find .", "sort", quit);
mainloop.run ();
}
async void read_piped_commands (string first_command, string second_command, SourceFunc quit) {
var output = splice_subprocesses (first_command, second_command);
try {
string? line = null;
do {
line = yield output.read_line_async ();
print (#"$(line ?? "")\n");
}
while (line != null);
} catch (Error error) {
print (#"Error: $(error.message)\n");
}
quit ();
}
DataInputStream splice_subprocesses (string first_command, string second_command) {
InputStream end_pipe = null;
try {
var first = new Subprocess.newv (first_command.split (" "), STDOUT_PIPE);
var second = new Subprocess.newv (second_command.split (" "), STDIN_PIPE | STDOUT_PIPE);
second.get_stdin_pipe ().splice (first.get_stdout_pipe (), CLOSE_TARGET);
end_pipe = second.get_stdout_pipe ();
} catch (Error error) {
print (#"Error: $(error.message)\n");
}
return new DataInputStream (end_pipe);
}
It is the splice_subprocesses function that answers your question. It takes the STDOUT from the first command as an InputStream and splices it with the OutputStream (STDIN) for the second command.
The read_piped_commands function takes the output from the end of the pipe. This is an InputStream that has been wrapped in a DataInputStream to give access to the read_line_async convenience method.
Here's the full, working implementation:
try {
string[] command = {"command", "-options", "-etc"};
string[] env = Environ.get ();
Pid child_pid;
string some_string = "This is what gets piped to stdin"
int stdin;
int stdout;
int stderr;
Process.spawn_async_with_pipes ("/",
command,
env,
SpawnFlags.SEARCH_PATH | SpawnFlags.DO_NOT_REAP_CHILD,
null,
out child_pid,
out stdin,
out stdout,
out stderr);
FileStream input = FileStream.fdopen (stdin, "w");
input.write (some_string.data);
/* Make sure we close the process using it's pid */
ChildWatch.add (child_pid, (pid, status) => {
Process.close_pid (pid);
});
} catch (SpawnError e) {
/* Do something w the Error */
}
I guess playing with the FileStream is what really made it hard to figure this out. Turned out to be pretty straightforward.
Based on previous answers probably an interesting case is to use program arguments to have a general app to pipe any input on it:
pipe.vala:
void main (string[] args) {
try {
string command = args[1];
var subproc = new Subprocess(STDIN_PIPE | STDOUT_PIPE, command);
var data = args[2].data;
var input = new MemoryInputStream.from_data(data, GLib.free);
subproc.get_stdin_pipe ().splice (input, CLOSE_TARGET);
var end_pipe = subproc.get_stdout_pipe ();
var output = new DataInputStream (end_pipe);
string? line = null;
do {
line = output.read_line();
print (#"$(line ?? "")\n");
} while (line != null);
} catch (Error error) {
print (#"Error: $(error.message)\n");
}
}
build:
$ valac --pkg gio-2.0 pipe.vala
and run:
$ ./pipe sort "cc
ab
aa
b
"
Output:
aa
ab
b
cc

How do I extract messages from an unbounded queue every N seconds and spawn them on to a Tokio handler?

I am trying to extract messages (which are futures themselves) from an unbounded queue every N seconds and spawn them into the Tokio handler.
I’ve tried dozens of variations but I cannot seem to find the right approach. It looks like it should be possible, but I always hit a future type mismatch or end up with borrow issues.
This is the code that shows more or less what I want:
let fut = Interval::new_interval(Duration::from_secs(1))
.for_each(|num| vantage_dequeuer.into_future() )
.for_each(|message:VantageMessage |{
handle.spawn(message);
return Ok(());
})
.map_err(|e| panic!("delay errored; err={:?}", e));
core.run(fut);
Complete code:
extern crate futures; // 0.1.24
extern crate tokio; // 0.1.8
extern crate tokio_core; // 0.1.17
use futures::future::ok;
use futures::sync::mpsc;
use futures::{Future, Stream};
use std::thread;
use std::time::Duration;
use tokio::timer::Interval;
use tokio_core::reactor::Core;
type VantageMessage = Box<Future<Item = (), Error = ()> + Send>;
fn main() {
let (enqueuer, dequeuer) = mpsc::unbounded();
let new_fut: VantageMessage = Box::new(ok(()).and_then(|_| {
println!("Message!");
return Ok(());
}));
enqueuer.unbounded_send(new_fut);
let joinHandle = worker(Some(dequeuer));
joinHandle.join();
}
/*
Every second extract one message from dequeuer (or wait if not available)
and spawn it in the core
*/
fn worker(
mut vantage_dequeuer: Option<mpsc::UnboundedReceiver<VantageMessage>>,
) -> thread::JoinHandle<()> {
let dequeuer = dequeuer.take().unwrap();
let joinHandle = thread::spawn(|| {
let mut core = Core::new().unwrap();
let handle = core.handle();
let fut = Interval::new_interval(Duration::from_secs(1))
.for_each(|num| vantage_dequeuer.into_future())
.for_each(|message: VantageMessage| {
handle.spawn(message);
return Ok(());
})
.map_err(|e| panic!("delay errored; err={:?}", e));
core.run(fut);
println!("Returned!");
});
return joinHandle;
}
Playground
error[E0425]: cannot find value `dequeuer` in this scope
--> src/main.rs:33:20
|
33 | let dequeuer = dequeuer.take().unwrap();
| ^^^^^^^^ not found in this scope
error[E0599]: no method named `into_future` found for type `std::option::Option<futures::sync::mpsc::UnboundedReceiver<std::boxed::Box<(dyn futures::Future<Item=(), Error=()> + std::marker::Send + 'static)>>>` in the current scope
--> src/main.rs:38:46
|
38 | .for_each(|num| vantage_dequeuer.into_future())
| ^^^^^^^^^^^
|
= note: the method `into_future` exists but the following trait bounds were not satisfied:
`&mut std::option::Option<futures::sync::mpsc::UnboundedReceiver<std::boxed::Box<(dyn futures::Future<Item=(), Error=()> + std::marker::Send + 'static)>>> : futures::Stream`
Interval and UnboundedReceiver are both streams, so I'd use Stream::zip to combine them:
The zipped stream waits for both streams to produce an item, and then returns that pair. If an error happens, then that error will be returned immediately. If either stream ends then the zipped stream will also end.
extern crate futures; // 0.1.24
extern crate tokio; // 0.1.8
extern crate tokio_core; // 0.1.17
use futures::{
future::ok,
sync::mpsc,
{Future, Stream},
};
use std::{thread, time::Duration};
use tokio::timer::Interval;
use tokio_core::reactor::Core;
type VantageMessage = Box<Future<Item = (), Error = ()> + Send>;
pub fn main() {
let (tx, rx) = mpsc::unbounded();
let new_fut: VantageMessage = Box::new(ok(()).and_then(|_| {
println!("Message!");
Ok(())
}));
tx.unbounded_send(new_fut).expect("Unable to send");
drop(tx); // Close the sending side
worker(rx).join().expect("Thread had a panic");
}
fn worker(queue: mpsc::UnboundedReceiver<VantageMessage>) -> thread::JoinHandle<()> {
thread::spawn(|| {
let mut core = Core::new().unwrap();
let handle = core.handle();
core.run({
Interval::new_interval(Duration::from_secs(1))
.map_err(|e| panic!("delay errored; err={}", e))
.zip(queue)
.for_each(|(_, message)| {
handle.spawn(message);
Ok(())
})
})
.expect("Unable to run reactor");
println!("Returned!");
})
}
Note that this doesn't actually wait for any of the spawned futures to complete before the reactor shuts down. If you want that, I'd switch to tokio::run and tokio::spawn:
fn worker(queue: mpsc::UnboundedReceiver<VantageMessage>) -> thread::JoinHandle<()> {
thread::spawn(|| {
tokio::run({
Interval::new_interval(Duration::from_secs(1))
.map_err(|e| panic!("delay errored; err={}", e))
.zip(queue)
.for_each(|(_, message)| {
tokio::spawn(message);
Ok(())
})
});
println!("Returned!");
})
}

How can I check if std::io::Cursor has unconsumed data?

I am writing a low-level network app that deals with TCP sockets where I often need to process binary data streams. When some data is available, I read it into u8 array, then wrap into std::io::Cursor<&[u8]> and then pass it to handlers. In a handler, I often need to know if there is some more data in the Cursor or not.
Imagine that the handle function receives data and then processes it in chunks using the handle_chunk function. For simplicity, assume that chunk size is fixed at 10 bytes; if the data size is not divisible by 10, it's an error. This simple logic can be implemented in the following way:
fn handle(mut data: Cursor<&[u8]>) {
while !data.empty() {
if let Err(err) = handle_chunk(&mut data) {
eprintln!("Error while handling data: {}", err);
}
}
}
fn handle_chunk(data: &mut Cursor<&[u8]>) -> Result<(), String> {
// Returns Err("unexpected EOF".to_string()) if chunk is incomplete
// ...
}
However, Cursor does not have an empty() method or any other method capable of telling if there is more data to process. The working solution that I could come up with is:
fn handle(data: Cursor<&[u8]>) {
let data = data.into_inner();
let len = data.len();
let mut data = Cursor::new(data);
while (data.position() as usize) < len - 1 {
if let Err(err) = handle_chunk(&mut data) {
eprintln!("Error while handling data: {}", err);
}
}
}
This looks hacky and inelegant though. Is there a better solution? Maybe there is a different tool in the Rust standard library that fits here better than Cursor?
Your code can be simplified by using Cursor::get_ref to avoid breaking up the input and putting it back together:
fn handle(mut data: Cursor<&[u8]>) {
let len = data.get_ref().len();
while (data.position() as usize) < len - 1 {
if let Err(err) = handle_chunk(&mut data) {
eprintln!("Error while handling data: {}", err);
}
}
}
Now, you haven't shown any code that requires a Cursor. Many times, people think it's needed to convert a &[u8] to something that implements Read, but it's not. Read is implemented for &'a [u8]:
use std::io::Read;
fn handle(mut data: &[u8]) {
while !data.is_empty() {
if let Err(err) = handle_chunk(&mut data) {
eprintln!("Error while handling data: {}", err);
}
}
}
fn handle_chunk<R: Read>(mut data: R) -> Result<(), String> {
let mut b = [0; 10];
data.read_exact(&mut b).unwrap();
println!("Chunk: {:?}", b);
Ok(())
}
fn main() {
let d: Vec<u8> = (0..20).collect();
handle(&d)
}
By having mut data: &[u8] and using &mut data, the code will update the slice variable in place to advance it forward. We can't easily go backward though.
an empty() method
Rust style indicates that an empty method would be a verb — this would remove data (if it were possible). The method you want should be called is_empty, as seen on slices.

Delaying a Tokio Stream

Given a Stream, I want to create a new Stream where elements are yielded with a time delay between them.
I tried to write code that does that using tokio_core::reactor::Timeout and the and_then combinator for Streams, but the delay doesn't work: I get all the elements immediately, without a delay.
Here is a self contained example (playground):
extern crate tokio_core;
extern crate futures;
use std::time::Duration;
use futures::{Future, Stream, stream, Sink};
use self::futures::sync::{mpsc};
use tokio_core::reactor;
const NUM_ITEMS: u32 = 8;
fn main() {
let mut core = reactor::Core::new().unwrap();
let handle = core.handle();
let chandle = handle.clone();
let (sink, stream) = mpsc::channel::<u32>(0);
let send_stream = stream::iter_ok(0 .. NUM_ITEMS)
.and_then(move |i: u32| {
let cchandle = chandle.clone();
println!("Creating a timeout object...");
reactor::Timeout::new(Duration::new(1,0), &cchandle)
.map_err(|_| ())
.and_then(|_| Ok(i))
});
let sink = sink.sink_map_err(|_| ());
handle.spawn(sink.send_all(send_stream).and_then(|_| Ok(())));
let mut incoming_items = Vec::new();
{
let keep_messages = stream.for_each(|item| {
incoming_items.push(item);
println!("item = {}", item);
Ok(())
});
core.run(keep_messages).unwrap();
}
assert_eq!(incoming_items, (0 .. NUM_ITEMS).collect::<Vec<u32>>());
}
For completeness, this is the output I get:
Creating a timeout object...
Creating a timeout object...
item = 0
Creating a timeout object...
item = 1
Creating a timeout object...
item = 2
Creating a timeout object...
item = 3
Creating a timeout object...
item = 4
Creating a timeout object...
item = 5
Creating a timeout object...
item = 6
item = 7
I suspect that the problem is somewhere in these lines:
reactor::Timeout::new(Duration::new(1,0), &cchandle)
.map_err(|_| ())
.and_then(|_| Ok(i))
It is possible that I don't really wait on the returned Timeout object, though I'm not sure how to solve it.
As I suspected, the problem was was the manipulation (using and_then) of the newly created Timeout. We either need to first unwrap the result from the call to reactor::Timeout::new, which could become messy if done manually, or use into_future, to convert the result into a Future, and then work with it using Future combinators.
Code for solving the problem:
extern crate tokio_core;
extern crate futures;
use std::time::Duration;
use futures::{Future, Stream, stream, Sink, IntoFuture};
use self::futures::sync::{mpsc};
use tokio_core::reactor;
const NUM_ITEMS: u32 = 8;
fn main() {
let mut core = reactor::Core::new().unwrap();
let handle = core.handle();
let chandle = handle.clone();
let (sink, stream) = mpsc::channel::<u32>(0);
let send_stream = stream::iter_ok(0 .. NUM_ITEMS)
.and_then(move |i: u32| {
let cchandle = chandle.clone();
println!("Creating a timeout object...");
reactor::Timeout::new(Duration::new(1,0), &cchandle)
.into_future()
.and_then(move |timeout| timeout.and_then(move |_| Ok(i)))
.map_err(|_| ())
});
let sink = sink.sink_map_err(|_| ());
handle.spawn(sink.send_all(send_stream).and_then(|_| Ok(())));
let mut incoming_items = Vec::new();
{
let keep_messages = stream.for_each(|item| {
incoming_items.push(item);
println!("item = {}", item);
Ok(())
});
core.run(keep_messages).unwrap();
}
assert_eq!(incoming_items, (0 .. NUM_ITEMS).collect::<Vec<u32>>());
}
Note that two and_then are being used. The first one unwraps the Result obtained from calling reactor::Timeout::new. The second one actually waits for the Timeout to fire.

Resources