I need to download HTML code from some web page. What is the best way to approach this task? As I understand there are very few working web frameworks for Rust right now and hyper is the one most people use? But after searching it's documentation I couldn't find a way. The closest I got is this
extern crate hyper;
use hyper::Client;
fn main() {
let client = Client::new();
let res = client.get("http://www.bloomberg.com/")
.send()
.unwrap();
println!("{:?}", res);
}
But it returns Response, which doesn't seem to contain any code from HTML body.
Note: this answer is outdated!
I don't have the time to update this with every hyper release. But please see my answer to a very related question: How can I download a website's content into a string?
It's a bit hidden: The Response type implements the trait Read. One method of Read is read_to_string which reads everything into the a String. That's a simple way you can get the body.
extern crate hyper;
use hyper::Client;
use std::io::Read;
fn main() {
let client = Client::new();
let mut res = client.get("http://www.bloomberg.com/")
.send()
.unwrap();
let mut body = String::new();
res.read_to_string(&mut body).expect("failed to read into string");
println!("{}", body);
}
Currently Rustdoc (the HTML documentation of Rust) is a little bit misleading because Rust beginners think that trait implementations don't add any important functionality. This is not true, so better look out for it. However, the hyper documentation could be better...
Related
What would be the equivalent in dart of the following javascript snippet?
myPeerConnection.createOffer().then(function(offer) {
return myPeerConnection.setLocalDescription(offer);
})
The straight translation:
final offer = await myPeerConnection.createOffer();
myPeerConnection.setLocalDescription(offer);
does not compile because createOffer() eventually returns a RtcSessionDescription and setLocalDescription accepts only a Map.
And RtcSessionDescription does not have a nice way to convert to a Map in the API.
Am I missing anything? Thanks!
After trying a few things, this works in my use case (at least in Chrome):
final offer = await myPeerConnection.createOffer();
myPeerConnection.setLocalDescription({"sdp": offer.sdp, "type": offer.type ?? 'offer'});
My Twilio function is in danger of getting too large and unwieldy. I'd like to break it up into smaller functions, and have the 'master' function call the other functions to get data as needed.
I don't see anything in the documentation about this, and the few tests I've tried have not been successful. Is there an easy/best way to go about doing this? Thanks!
this is an example of how to include code from another function:
including function's body
exports.handler = function(context, event, callback) {
let path = Runtime.getFunctions().helper.path;
let helper = require(path);
let output = helper.output_init();
}
included function's body (the name of this function needs to be 'helper' to work on this example)
function output_init(){
let output = new Twilio.Response();
output.setStatusCode(200);
output.appendHeader('Content-Type', 'application/json');
return output;
}
module.exports ={
output_init: output_init,
};
hope this helps
there is a discussion around this topic on a Google Groups forum and the details are provided from the documentation below:
Runtime Client
https://www.twilio.com/docs/runtime/client#functions
"The Function object enables developers to compose complex applications by allowing them to include and execute code stored in multiple Functions."
In JavaScript we have something like .toString which can convert the entire function object to string.
Do we have something similar on IOS?
For example, in JavaScript if we have function like this, after converting it with .toString and printing the value in console we see the entire function object.
function sum(a, b)
{
return a + b;
}
console.log(sum.toString());
// expected output:
// "function sum(a, b)
// {
//return a + b;
// }"
Can we do something similar for IOS? I tried String (describing :Function) in Swift but that didn't work and gave me output as (Function) but not the complete structure like we get in JavaScript .toString.
public func say_hello()
{
print("Hello, World!")
}
String(describing: say_hello))
//Output:(Function)
Despite the many comments explaining why that's not possible (nor feasible in many cases), I want to point out that you can use JavaScript code in your Swift app and thus use the serialization mechanism of that language. Have a look at JSContext for details. This of course won't make things simpler, but it does give extra flexibility with injecting/changing/extending functionality at runtime.
This is not possible from Swift/Objc
I used to do something like this:
HttpResponse res = req.response;
String dataReceived;
await req.listen((List<int> buffer) {
dataReceived = new String.fromCharCodes(buffer);
}).asFuture();
Map data = JSON.decode(dataReceived);
When I needed UTF8 support, I modified it to:
Map data = JSON.decode(await new Utf8Codec().decodeStream(request));
Kevin Moore suggested to encode/decode like this:
https://dartpad.dartlang.org/1d229cfdc1c1fd2ab877
So I've got:
Map data;
await request.listen((List<int> buffer) {
data = JSON.fuse(UTF8).decode(buffer);
}).asFuture();
Not sure that I need the asFuture():
Map data;
await request.listen((List<int> buffer) => data = JSON.fuse(UTF8).decode(buffer));
Or do I? And this method requires that I encode it into bytes on the client side:
sendData: new JsonUtf8Encoder().convert({'model': message, 'authToken': app.authToken}))
What are the benefits of this? Isn't it more to send over the wire?
I believe Shelf and/or the new RPC lib would handle this stuff for me? Shall I move to one of those? Right now, it's all homegrown.
HttpRequest is a Stream<List<int>>. You don't want to use listen because you'll only get the first "chunk" of data.
Instead you'll want to do something like this:
import 'dart:async';
import 'dart:convert';
main() async {
var input = {'a':1, 'b':2};
var decoder = JSON.fuse(UTF8).decoder;
var json = await decoder.bind(toByteStream(input)).single;
print(json);
}
Stream<List<int>> toByteStream(json) =>
_encoder.bind(new Stream.fromIterable([json]));
final _encoder = new JsonUtf8Encoder();
https://dartpad.dartlang.org/9807d0c5ed89360c9f53
Yes as you can see on https://github.com/dart-lang/shelf/blob/master/lib/src/message.dart#L136 shelf defaults to UTF-8
I am likely biased but I would definitely recommend moving over to shelf. You have several options depending on what you prefer, like:
shelf_rpc as you mentioned. I haven't used it but likely full featured API support
shelf_bind if you simply want to bind a handler function parameter to a JSON body. This is lower level, more flexible and less prescriptive but does less. e.g.
router.post('/foo', (#RequestBody() Foo foo) => ...)
shelf_rest. Adds higher level more prescriptive API support (similar to shelf_rpc).
full frameworks like redstone, mojito etc. These do more for you but you need to buy into more
Had a chat w/ Kevin to better understand his answer, and thought it best to share my learnings as a new answer.
HttpRequest is always a Stream<List<int>> – a streamed list of integers. Those integers are bytecodes, and this is commonly referred to as a bytestream. You can be sure that no matter what API you use to send data over the wire, that it is sent as a bytestream.
The HttpRequest.request() method accepts sendData in several forms...
* If specified, `sendData` will send data in the form of a [ByteBuffer],
* [Blob], [Document], [String], or [FormData] along with the HttpRequest.
Source:
https://api.dartlang.org/apidocs/channels/stable/dartdoc-viewer/dart:html.HttpRequest#id_request
...but these are just abstractions, and ultimately your data is sent as a Stream<List<int>> bytestream.
So on the server we first set up a decoder that will decode both JSON and UTF8 (for correct char handling), and then we bind that to the HttpRequest request, which is a bytestream. I think single just serves to ensure we throw an exception if we received more than one data event. Here's all the code we need to interpret an HttpRequest:
import 'dart:async';
import 'dart:convert';
static handleRequest(HttpRequest request) async {
var decoder = JSON.fuse(UTF8).decoder;
var data = await decoder.bind(request).single;
print('The decoded data received is:\n\n$data');
}
How do I use F#'s built-in support for async operations classes exposing the Event-based Asynchronous Pattern such as WebClient class?
let Download(url : Uri) =
let client = new WebClient()
let html = client.DownloadString(url)
html
When I try to change this to use "let!" in an async block (say as described in Soma's recent post)
let Download(url : Uri) =
async {
let client = new WebClient()
let! html = client.DownloadStringAsync(url)
return html }
I get an error message:
Type constraint mismatch. The type unit is not compatible with type Async<'a> The type 'unit' is not compatible with the type 'Async<'a>'
Edit: I'm really asking about the general question of using *Async() methods, WebClient is just an easy example. Microsoft says "... you should expose asynchronous features using the Event-based Asynchronous Pattern [ as opposed to BeginFoo()/EndFoo() ] whenever possible ..." so I would think there should be an easy way to consume an arbitrary *Async() method from F#.
The WebClient.DownloadStringAsync method is part of the .NET framework. It'll raise an event to signal its progress, and its return type is unit, so you don't want to use it, and there's no advantage in wrapping it in an async object.
The F# PowerPack defines an extension method, val webclient.AsyncDownloadString : uri -> Async{string}:
let Download(url : Uri) =
async {
let client = new WebClient()
client.Encoding <- Encoding.GetEncoding("utf-8")
let! html = client.AsyncDownloadString(url)
return html }
Unfortunately, the choice of name clashes with the existing webclient method, which can understandably cause confusion. However, I believe all of the F# async extensions begin with Async*.
[Edit to add in response to comments:]
Usually, .NET uses the BeginFoo / EndFoo pattern for concurrency. If the types are right, you can just use Async.BuildPrimitive beginMethod endMethod, which will return an Async wrapper for the method.
Sometimes objects don't use this pattern, like the WebClient, and you actually have to use Async.AwaitEvent to wait for an event to be fired, or write your own loop to repeatedly check to see if a bool is set. Here's a nice article on converting events to Async objects.
For what its worth, if you have F# installed, you should also have the source code which will give you an idea of how the F# team implements their async extensions. On my machine, the relevant file is located at:
C:\Program Files\FSharp-1.9.6.16\source\fsppack\FSharp.PowerPack\AsyncOperations.fs