I've got the following struct in Rust that I want to return from my service:
#[derive(PartialEq,Clone,Default)]
pub struct Response {
pub names: ::protobuf::RepeatedField<String>,
}
and I set its names field (see the doc about its type) in the following way:
use protobuf::{Message, RepeatedField};
use futures::future::Future;
use futures::Stream;
fn create_response(names: impl Stream<Item = String, Error = CustomError>) -> Result<Response, CustomError>
let names = names.collect().wait()?;
let mut response = Response::new();
response.set_names(RepeatedField::from_vec(names));
Ok(response)
}
Now I want to optimise this piece of code and avoid blocking collect operation. Is there any way to stream that names directly into response.names field (i.e., avoid the step of creating a names vector)?
there are relevant pieces of code from protobuf sources:
/// Wrapper around vector to avoid deallocations on clear.
pub struct RepeatedField<T> {
vec: Vec<T>,
len: usize,
}
/// Create a contained with data from given vec.
#[inline]
pub fn from_vec(vec: Vec<T>) -> RepeatedField<T> {
let len = vec.len();
RepeatedField { vec: vec, len: len }
}
So no, your code is already easy and optimal: it creates the vector and then just moves it to the structure without unnecessary copying.
Related
With the Rocket web framework I can parse, error check, and use the data of a form with fields known ahead of time, like this:
#[derive(Debug, FromForm)]
struct Submission<'v> {
foo: &'v str,
bar: i32,
baz: f32,
qux: bool,
}
#[post("/", data = "<form>")]
fn post<'r>(form: Form<Contextual<'r, Submission<'r>>>) -> Template {
if let Some(ref submission) = form.value {
println!("valid: {:?}", submission);
}
Template::render("base", &form.context)
}
How would I do the same thing, if the names, types, and number of fields isn't fixed at compile time (instead being populated from a JSON).
Running cargo expand to view the output of the FromForm derive yields about 315 lines of code for just that 4-variable struct (likely too much to replicate), and although it's possible to manually parse the raw multipart/form-data string with
fn post<'r>(form: String) -> Template {
that seems to undermine the point of using rocket.
Let's say I have a function defined in Rust, which looks like this:
#[no_mangle]
pub unsafe extern "C" fn do_something(
my_value: *mut MyStruct,
some_param: c_uint,
content: *mut *mut u8,
length: *mut c_uint,
capacity: *mut c_uint,
) -> *mut MyStruct {
// Do something and obtain an `ffi_result`...
let heap_data = ffi_result.as_mut_ptr();
// These values are passed as "out" parameters.
*length = ffi_result.len() as c_uint;
*capacity = ffi_result.capacity() as c_uint;
*content = heap_data;
// We intentionally "leak" this data to the heap.
// The caller is responsible for cleaning it up by calling another function.
std::mem::forget(ffi_result);
std::boxed::Box::into_raw(value_of_type_my_struct)
}
It takes in a pointer to a struct, a simple integer parameter, several out parameters that can later be used to create an Array and it returns a pointer to a struct.
Now I compile the rust library into a static library for the target aarch64-apple-ios. I set up a XCode project, add the static library as a dependency as explained here with an "Objective-C Bridging Header" where I import the following header file
#ifndef libmy_project_h
#define libmy_project_h
#include <stdint.h>
struct myStruct;
struct myStruct *do_something(struct myStruct *state, int someParam, char **content, int *length, int *capacity);
#endif
Up until this point everything seems to work fine and I have already successfully used this procedure for a whole bunch of other functions. However in this special case I can not figure out how to call this function from swift. I need to call the function from swift and pass content, length and capacity as out parameters so that I can later use the pointers to create an Array in Swift like so.
This is what I tried so far:
var content = UnsafeMutablePointer<UnsafeMutablePointer<CChar>?>(UnsafeMutablePointer(bitPattern: 0))
var length = UnsafeMutablePointer<Int32>(bitPattern: 0)
var capacity = UnsafeMutablePointer<Int32>(bitPattern: 0)
let my_struct = do_something(my_struct, Int32(some_param), content, length, capacity)
let buffer = UnsafeRawBufferPointer(start: content?.pointee, count: Int(length!.pointee))
var data = Array(repeating: UInt8(0), count: Int(length!.pointee))
data.withUnsafeMutableBytes { arrayPtr in
arrayPtr.copyBytes(from: buffer)
}
However now when I execute this swift snippet, I get an EXC_BAD_ACCESS error, which I think occurs because the pointers I manually created do not belong to the adress space of my application. How can I create pointers that I can use as out parameters?
P.S. For reference here is the same interop code in C#:
[DllImport("my_dll")]
private static extern IntPtr do_something(IntPtr state, uint someParam, out IntPtr content, out uint length, out uint capacity);
Which can be called like so:
IntPtr contentPointer;
uint length, capacity;
IntPtr my_struct = do_something(state, myParam, out contentPointer, out length, out capacity);
byte[] rawContent = new byte[length];
Marshal.Copy(contentPointer, rawContent, 0, (int)length);
// Free the data owned by rust with another FFI call:
free_do_something_result(contentPointer, length, capacity);
var length = UnsafeMutablePointer<Int32>(bitPattern: 0)
You need to pass storage for your out-parameters. This is defining a null pointer. When Rust tries to write the result to address 0, it crashes, since you don't have access to write there.
Instead of creating two layers of pointers, create a value of the type you want, and then pass the address (&) of that value; this will add the extra layer of pointer automatically.
// Create storage
var content: UnsafeMutablePointer<CChar>? // Could be NULL, so Optional
var length: Int32 = 0
var capacity: Int32 = 0
// Pass as references
do_something(&content, &length, &capacity)
// Copy the data
let data = Array(UnsafeRawBufferPointer(start: content, count: Int(length)))
content is still a pointer here because the thing being updated is a pointer. You're not providing storage for content, Rust is. But you do need to provide storage for the pointer (and that's what this does).
I can't compile your code because it's missing a lot (an MCVE would be much better here), so I can't test that this is doing exactly what you mean, but it should be close.
In your example, you're leaking the memory, but since your C# calls free_do_something_result (which I assume cleans it up), I assume you're actually doing the same in the Swift.
I want to write a parser.
It seems practical to me to have a mutable Iterator that I can pass around to different parser functions.
I've tried to illustrated a simplified approach, which compiles but is not ideal yet.
fn main() {
let tokens = vec!["fIrSt".to_string(), "SeConD".to_string(), "tHiRd".to_string(), "FoUrTh".to_string()];
let parsed = parse_input(tokens);
println!("{}", parsed);
}
fn parse_input(tokens: Vec<String>) -> String {
let mut tokens_iter = tokens.iter();
let upps = parse_upper(&mut tokens_iter);
let lowers = parse_lower(&mut tokens_iter);
upps + &lowers
}
fn parse_upper(tokens_iter: &mut Iterator<Item=&String>) -> String {
let mut result = String::new();
let token_1 = tokens_iter.next().unwrap().to_uppercase();
let token_2 = tokens_iter.next().unwrap().to_uppercase();
result.push_str(&token_1);
result.push_str(&token_2);
result
}
fn parse_lower(tokens_iter: &mut Iterator<Item=&String>) -> String {
let mut result = String::new();
let token_1 = tokens_iter.next().unwrap().to_lowercase();
let token_2 = tokens_iter.next().unwrap().to_lowercase();
result.push_str(&token_1);
result.push_str(&token_2);
result
}
How the example works:
Let's say I have some input, that has already been tokenized. Here it is represented by the tokens vector (Vec<String>).
Inside the outer parse_input function, the Vec gets transformed into an Iterator and then passed into different, specific parser functions. Here: parse_upper and parse_lower. In real life those could be "parse_if_statement" or "parse_while_loop" but which part of the Iterator gets worked on is not relevant for the question.
What is relevant is, that every call to next advances the cursor on the Iterator. So that every function consumes the pieces it needs.
This example compiles and gives the output: FIRSTSECONDthirdfourth
I would like to be able to peek() into the Iterator, before I pass it to a function. This is necessary to determine which function should actually be called. But everything I have tried with using a Peekable instead of an Iterator resulted in total lifetime and borrow chaos.
Any suggestions on how to pass a Peekable instead of an Iterator in this case?
Maybe using a Peekable as function parameter is a bad idea in the first place. Or maybe my Iterator approach is already wrong. All suggestions/hints are welcome.
I am trying to extract the extension of a file from a given String path.
The following piece of code works, but I was wondering if there is a cleaner and more idiomatic Rust way to achieve this:
use std::path::Path;
fn main() {
fn get_extension_from_filename(filename: String) -> String {
//Change it to a canonical file path.
let path = Path::new(&filename).canonicalize().expect(
"Expecting an existing filename",
);
let filepath = path.to_str();
let name = filepath.unwrap().split('/');
let names: Vec<&str> = name.collect();
let extension = names.last().expect("File extension can not be read.");
let extens: Vec<&str> = extension.split(".").collect();
extens[1..(extens.len())].join(".").to_string()
}
assert_eq!(get_extension_from_filename("abc.tar.gz".to_string()) ,"tar.gz" );
assert_eq!(get_extension_from_filename("abc..gz".to_string()) ,".gz" );
assert_eq!(get_extension_from_filename("abc.gz".to_string()) , "gz");
}
In idiomatic Rust the return type of a function that can fail should be an Option or a Result. In general, functions should also accept slices instead of Strings and only create a new String where necessary. This reduces excessive copying and heap allocations.
You can use the provided extension() method and then convert the resulting OsStr to a &str:
use std::path::Path;
use std::ffi::OsStr;
fn get_extension_from_filename(filename: &str) -> Option<&str> {
Path::new(filename)
.extension()
.and_then(OsStr::to_str)
}
assert_eq!(get_extension_from_filename("abc.gz"), Some("gz"));
Using and_then is convenient here because it means you don't have to unwrap the Option<&OsStr> returned by extension() and deal with the possibility of it being None before calling to_str. I also could have used a lambda |s| s.to_str() instead of OsStr::to_str - it might be a matter of preference or opinion as to which is more idiomatic.
Notice that both the argument &str and the return value are references to the original string slice created for the assertion. The returned slice cannot outlive the original slice that it is referencing, so you may need to create an owned String from this result if you need it to last longer.
What's more idiomatic than using Rust's builtin method for it?
Path::new(&filename).extension()
I am using a 3rd party C library in my iOS application, which I am in the process of converting from Objective-C to Swift. I hit an obstacle when attempting to read one of the structs returned by the C library in Swift.
The struct looks similar to this:
typedef unsigned int LibUint;
typedef unsigned char LibUint8;
typedef struct RequestConfiguration_ {
LibUint8 names[30][128];
LibUint numberNames;
LibUint currentName;
} RequestConfiguration;
Which is imported into Swift as a Tuple containing 30 Tuples of 128 LibUint8 values. After a long time of trial and error using nested withUnsafePointer calls, I eventually began searching for solutions to iterating a Tuple in Swift.
What I ended up using is the following functions:
/**
* Perform iterator on every children of the type using reflection
*/
func iterateChildren<T>(reflectable: T, #noescape iterator: (String?, Any) -> Void) {
let mirror = Mirror(reflecting: reflectable)
for i in mirror.children {
iterator(i.label, i.value)
}
}
/**
* Returns a String containing the characters within the Tuple
*/
func libUint8TupleToString<T>(tuple: T) -> String {
var result = [CChar]()
let mirror = Mirror(reflecting: tuple)
for child in mirror.children {
let char = CChar(child.value as! LibUint8)
result.append(char)
// Null reached, skip the rest.
if char == 0 {
break;
}
}
// Always null terminate; faster than checking if last is null.
result.append(CChar(0))
return String.fromCString(result) ?? ""
}
/**
* Returns an array of Strings by decoding characters within the Tuple
*/
func libUint8StringsInTuple<T>(tuple: T, length: Int = 0) -> [String] {
var idx = 0
var strings = [String]()
iterateChildren(tuple) { (label, value) in
guard length > 0 && idx < length else { return }
let str = libUint8TupleToString(value)
strings.append(str)
idx++
}
return strings
}
Usage
func handleConfiguration(config: RequestConfiguration) {
// Declaration types are added for clarity
let names: [String] = libUint8StringsInTuple(config.names, config.numberNames)
let currentName: String = names[config.currentName]
}
My solution uses reflection to iterate the first Tuple, and reflection to iterate the second, because I was getting incorrect strings when using withUnsafePointer for the nested Tuples, which I assume is due to signage. Surely there must be a way to read the C strings in the array, using an UnsafePointer alike withUsafePointer(&struct.cstring) { String.fromCString(UnsafePointer($0)) }.
To be clear, I'm looking for the fastest way to read these C strings in Swift, even if that involves using Reflection.
Here is a possible solution:
func handleConfiguration(var config: RequestConfiguration) {
let numStrings = Int(config.numberNames)
let lenStrings = sizeofValue(config.names.0)
let names = (0 ..< numStrings).map { idx in
withUnsafePointer(&config.names) {
String.fromCString(UnsafePointer<CChar>($0) + idx * lenStrings) ?? ""
}
}
let currentName = names[Int(config.currentName)]
print(names, currentName)
}
It uses the fact that
LibUint8 names[30][128];
are 30*128 contiguous bytes in memory. withUnsafePointer(&config.names)
calls the closure with $0 as a pointer to the start of that
memory location, and
UnsafePointer<CChar>($0) + idx * lenStrings
is a pointer to the start of the idx-th subarray. The above code requires
that each subarray contains a NUL-terminated UTF-8 string.
The solution suggested by Martin R looks good to me and, as far as I can see from my limited testing, does work. However, as Martin pointed out, it requires that the strings be NUL-terminated UTF-8. Here are two more possible approaches. These follow the principle of handling the complexity of C data structures in C instead of dealing with it in Swift. Which of these approaches you choose depends on what specifically you are doing with RequestConfiguration in your app. If you are not comfortable programming in C, then a pure Swift approach, like the one suggested by Martin, might be a better choice.
For the purposes of this discussion, we will assume that the 3rd party C library has the following function for retrieving RequestConfiguration:
const RequestConfiguration * getConfig();
Approach 1: Make the RequestConfiguration object available to your Swift code, but extract names from it using the following C helper function:
const unsigned char * getNameFromConfig(const RequestConfiguration * rc, unsigned int nameIdx)
{
return rc->names[nameIdx];
}
Both this function's signature and the RequestConfiguration type must be available to the Swift code via the bridging header. You can then do something like this in Swift:
var cfg : UnsafePointer<RequestConfiguration> = getConfig()
if let s = String.fromCString(UnsafePointer<CChar>(getNameFromConfig(cfg, cfg.memory.currentName)))
{
print(s)
}
This approach is nice if you need the RequestConfiguration object available to Swift in order to check the number of names in multiple places, for example.
Approach 2: You just need to be able to get the name at a given position. In this case the RequestConfiguration type does not even need to be visible to Swift. You can write a helper C function like this:
const unsigned char * getNameFromConfig1(unsigned int idx)
{
const RequestConfiguration * p = getConfig();
return p->names[idx];
}
and use it in Swift as follows:
if let s = String.fromCString(UnsafePointer<CChar>(getNameFromConfig1(2)))
{
print(s)
}
This will print the name at position 2 (counting from 0). Of course, with this approach you might also want to have C helpers that return the count of names as well as the current name index.
Again, with these 2 approaches it is assumed the strings are NUL-terminated UTF-8. There are other approaches possible, these are just examples.
Also please note that the above assumes that you access RequestConfiguration as read-only. If you also want to modify it and make the changes visible to the 3rd party library C code, then it's a different ballgame.