How can I write uploaded multipart files to disk? - dart

I'm trying to handle file upload through multipart requests with Aqueduct. Aqueduct has now an example on how to handle multipart requests here:
https://aqueduct.io/docs/http/request_and_response/#example-multipartform-data
The example explains, how to get the header and the content of the files. However it doesn't explain how to write the content into a file on disk.
How can I write the content of the files uploaded to disk?
Below an example that shows what I want to achieve, but doesn't work:
import 'dart:io';
import 'package:aqueduct/aqueduct.dart';
import 'package:mime/mime.dart';
class MediaUploadController extends ResourceController {
MediaUploadController() {
acceptedContentTypes = [ContentType("multipart", "form-data")];
}
#Operation.post()
Future<Response> postMultipartForm() async {
final transformer = MimeMultipartTransformer(request.raw.headers.contentType.parameters["boundary"]);
final bodyStream = Stream.fromIterable([await request.body.decode<List<int>>()]);
final parts = await transformer.bind(bodyStream).toList();
for (var part in parts) {
final String contentType = part.headers["content-type"];
// Write content to disk
final content = await part.toList();
final fileName = DateTime.now().millisecondsSinceEpoch.toString() + ".jpg";
var file = new File('data/' + fileName);
var sink = file.openWrite();
sink.write(content);
sink.close();
}
return new Response.ok({});
}
}

This below actually worked. Additionally to the mime package, I have also added the http_server package to pubspec.yaml, because it makes it easier to handle the multipart form data.
dependencies:
aqueduct: ^3.0.1
mime: ^0.9.6+2
http_server: ^0.9.8+1
Then I studied some other frameworks to see how they handled writing to the file. It's so complicated to get how this multipart stuff and streams work together. But at last after nearly a week, the light at the end of the tunnel... until the next questions pop up. Most likely 10 minutes down the line :)
import 'dart:io';
import 'package:aqueduct/aqueduct.dart';
import 'package:mime/mime.dart';
import 'package:http_server/http_server.dart';
class MediaUploadController extends ResourceController {
MediaUploadController() {
acceptedContentTypes = [ContentType("multipart", "form-data")];
}
#Operation.post()
Future<Response> postMultipartForm() async {
final transformer = MimeMultipartTransformer(request.raw.headers.contentType.parameters["boundary"]);
final bodyStream = Stream.fromIterable([await request.body.decode<List<int>>()]);
final parts = await transformer.bind(bodyStream).toList();
for (var part in parts) {
HttpMultipartFormData multipart = HttpMultipartFormData.parse(part);
final ContentType contentType = multipart.contentType;
final content = multipart.cast<List<int>>();
final filePath = "data/" + DateTime.now().millisecondsSinceEpoch.toString() + ".jpg";
IOSink sink = File(filePath).openWrite();
await for (List<int> item in content) {
sink.add(item);
}
await sink.flush();
await sink.close();
}
return new Response.ok({});
}
}

Related

Java How to format URL as a String to connect with JSoup Malformed URL error

I have a program that connects to a user defined URL from a TextField and scrapes the images on that web page. The user defined URL is gotten from the textfield via .getText() and assigned to a String. The String is then used to connect to the Web page with JSoup and puts the webpage into a document.
String address = labelforAddress.getText();
try {
document = Jsoup.connect(address).get();
}catch(IOException ex){
ex.printStackTrace();
}
I've tried differently formatted URLS: "https://www.", "www.", "https://" but everything I use throws the malformed URL error.
Someone please show me how to get the text from the TextField the correct way.
Below is the entire code.
package sample;
import javafx.application.Application;
import javafx.fxml.FXMLLoader;
import javafx.scene.Parent;
import javafx.scene.Scene;
import javafx.scene.control.Button;
import javafx.scene.control.Label;
import javafx.scene.control.TextField;
import javafx.scene.layout.GridPane;
import javafx.stage.FileChooser;
import javafx.stage.Stage;
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
import org.jsoup.select.Elements;
import java.io.*;
import java.net.URL;
import java.util.ArrayList;
import java.util.LinkedList;
import java.util.List;
import java.util.zip.ZipEntry;
import java.util.zip.ZipOutputStream;
public class Main extends Application {
Document document;
LinkedList<String> imageURLList = new LinkedList<String>();
ArrayList<File> fileList = new ArrayList<File>();
int fileCount = 1;
#Override
public void start(Stage primaryStage) throws Exception{
Parent root = FXMLLoader.load(getClass().getResource("sample.fxml"));
primaryStage.setTitle("Webpage Photo Scraper");
GridPane gp = new GridPane();
Label labelforAddress = new Label("URL");
GridPane.setConstraints(labelforAddress, 0,0);
TextField URLAddress = new TextField();
GridPane.setConstraints(URLAddress, 1,0);
Button scrape = new Button("Scrape for Photos");
GridPane.setConstraints(scrape, 0,1);
scrape.setOnAction(event->{
String address = labelforAddress.getText();
try {
document = Jsoup.connect(address).get();
}catch(IOException ex){
ex.printStackTrace();
}
Elements imgTags = document.getElementsByAttributeValueContaining("src", "/CharacterImages");
for(Element imgTag: imgTags){
imageURLList.add(imgTag.absUrl("src"));
}
for(String url: imageURLList){
File file = new File("C:\\Users\\Andrei\\Documents\\file" + fileCount + ".txt");
downloadFromURL(url, file);
fileList.add(file);
fileCount++;
}
});
Button exportToZipFile = new Button("Export to Zip File");
GridPane.setConstraints(exportToZipFile, 0,2);
exportToZipFile.setOnAction(event -> {
FileChooser fileChooser = new FileChooser();
FileChooser.ExtensionFilter exfilt = new FileChooser.ExtensionFilter("Zip Files", ".zip");
fileChooser.getExtensionFilters().add(exfilt);
try{
FileOutputStream fos = new FileOutputStream(fileChooser.showSaveDialog(primaryStage));
ZipOutputStream zipOut = new ZipOutputStream(fos);
for(int count = 0; count<=fileList.size()-1; count++){
File fileToZip = new File(String.valueOf(fileList.get(count)));
FileInputStream fis = new FileInputStream(fileToZip);
ZipEntry zipEntry = new ZipEntry(fileToZip.getName());
zipOut.putNextEntry(zipEntry);
byte[] bytes = new byte[1024];
int length;
while((length = fis.read(bytes)) >= 0) {
zipOut.write(bytes, 0, length);
}
fis.close();
}
zipOut.close();
fos.close();
}catch(IOException e1){
e1.printStackTrace();
}
});
primaryStage.setScene(new Scene(gp, 300, 275));
primaryStage.show();
gp.getChildren().addAll(exportToZipFile, labelforAddress, scrape, URLAddress);
}
public static void downloadFromURL(String url, File file){
try {
URL Url = new URL(url);
BufferedInputStream bis = new BufferedInputStream(Url.openStream());
FileOutputStream fis = new FileOutputStream(file);
byte[] buffer = new byte[1024];
int count = 0;
while((count = bis.read(buffer, 0,1024)) !=-1){
fis.write(buffer, 0, count);
}
fis.close();
bis.close();
}catch(IOException e){
e.printStackTrace();
}
}
public static void main(String[] args) {
launch(args);
}
}
Your text field containing the value entered by user is stored in URLAddress object but you always try to get the url from labelforAddress object which is a label always containing "URL" text.
So the solution is to use:
String address = URLAddress.getText();
If you read carefully error message it would help you to find the cause, because it always displays the value it considers wrong. In this case I see:
Caused by: java.net.MalformedURLException: no protocol: URL
and it shows the unrecognized address is: URL.
If you encounter this kind of error next time try:
debugging the aplication in runtime to see values of each variable
logging variable values in the console to see if variables contain values you expect

How do I send multiple API request once?

currently, I used http, I found a way to send multiple requests once to react using axios.
axios.all([
axios.get('http://google.com'),
axios.get('http://apple.com')
])
.then(axios.spread((googleRes, appleRes) => {
// do something with both responses
});
Like this Is that any way to send multiple requests once?
#mezoni answer is correct. But this is less code with caching also.
import 'dart:async';
import 'dart:convert';
import 'package:http/http.dart' as http;
void main() async {
final urlList = ['http://google.com', 'http://apple.com'];
final responses = await Future.wait(
urlList.map((String url) {
return http.get(url);
}),
);
final List<dynamic> caches = responses.map((response) {
return json.decode(response.body);
}).toList();
}

Make a http request in dart whith dart:io

Hey I'm a beginner and I want to interact with an API with dart:io for fetch JSON files I can fetch the data with this code :
final HttpClient client = HttpClient();
client.getUrl(Uri.parse("https://api.themoviedb.org/3/movie/76341?api_key=fbe54362add6e62e0e959f0e7662d64e&language=fr"))
.then((HttpClientRequest request) {
return request.close();
})
.then((HttpClientResponse response) {
Map a;
print(a);
But I want to have a Map whith the JSON but I can't do it. If I could get a String that contains the JSON I could do it with
json.decode();
also know that the answer is stored in an int list that represents the utf8 values of the characters so with utf8.decode(responce.toList()) I can get the utf8 value but responce.toList() return a Future but even if it may be easy I don't know how to get the list.
import 'dart:convert';
import 'dart:io';
void main() async {
final client = HttpClient();
final request = await client.getUrl(Uri.parse(
'https://api.themoviedb.org/3/movie/76341?api_key=fbe54362add6e62e0e959f0e7662d64e&language=fr'));
final response = await request.close();
final contentAsString = await utf8.decodeStream(response);
final map = json.decode(contentAsString);
print(map);
}

How to write a `ByteData` instance to a File in Dart?

I am using Flutter to load an "asset" into a File so that a native application can access it.
This is how I load the asset:
final dbBytes = await rootBundle.load('assets/file');
This returns an instance of ByteData.
How can I write this to a dart.io.File instance?
ByteData is an abstraction for:
A fixed-length, random-access sequence of bytes that also provides
random and unaligned access to the fixed-width integers and floating
point numbers represented by those bytes.
As Gunter mentioned in the comments, you can use File.writeAsBytes. It does require a bit of API work to get from ByteData to a List<int>, however.
import 'dart:async';
import 'dart:io';
import 'dart:typed_data';
Future<void> writeToFile(ByteData data, String path) {
final buffer = data.buffer;
return new File(path).writeAsBytes(
buffer.asUint8List(data.offsetInBytes, data.lengthInBytes));
}
I've also filed an issue to make the docs on Flutter more clear for this use case.
you need to have path_provider package installed, then
This should work :
import 'dart:async';
import 'dart:io';
import 'dart:typed_data';
import 'package:path_provider/path_provider.dart';
final dbBytes = await rootBundle.load('assets/file'); // <= your ByteData
//=======================
Future<File> writeToFile(ByteData data) async {
final buffer = data.buffer;
Directory tempDir = await getTemporaryDirectory();
String tempPath = tempDir.path;
var filePath = tempPath + '/file_01.tmp'; // file_01.tmp is dump file, can be anything
return new File(filePath).writeAsBytes(
buffer.asUint8List(data.offsetInBytes, data.lengthInBytes));
}
//======================
to get your file :
var file;
try {
file = await writeToFile(dbBytes); // <= returns File
} catch(e) {
// catch errors here
}
Hope this helps,
Thank you.
to search flutter ByteData to List<int> then found here, but not fully answer my question:
how to convert ByteData to List<int> ?
after self investigate, solution is:
use .cast<int>()
ByteData audioByteData = await rootBundle.load(audioAssetsFullPath);
Uint8List audioUint8List = audioByteData.buffer.asUint8List(audioByteData.offsetInBytes, audioByteData.lengthInBytes);
List<int> audioListInt = audioUint8List.cast<int>();
or 2. use .map
ByteData audioByteData = await rootBundle.load(audioAssetsFullPath);
Uint8List audioUint8List = audioByteData.buffer.asUint8List(audioByteData.offsetInBytes, audioByteData.lengthInBytes);
List<int> audioListInt = audioUint8List.map((eachUint8) => eachUint8.toInt()).toList();
For those looking to write bytes (aka Uint8List) instead of ByteData please note that ByteData is a wrapper for Uint8List.
From /runtime/lib/typed_data.patch:
#patch
class ByteData implements TypedData {
#patch
#pragma("vm:entry-point")
factory ByteData(int length) {
final list = new Uint8List(length) as _TypedList;
_rangeCheck(list.lengthInBytes, 0, length);
return new _ByteDataView(list, 0, length);
}
#patch
class Uint8List {
#patch
#pragma("vm:exact-result-type", _Uint8List)
factory Uint8List(int length) native "TypedData_Uint8Array_new";
}
If you are using the latter type you can use the answer provided by Rami and modify the return as follow:
import 'dart:async';
import 'dart:io';
import 'dart:typed_data';
import 'package:path_provider/path_provider.dart';
Future<File> writeToFile(Uint8List data) async {
(...)
return new File(filePath).writeAsBytes(data);
}

Generate one file for a list of parsed files using source_gen in dart

I have a list of models that I need to create a mini reflective system.
I analyzed the Serializable package and understood how to create one generated file per file, however, I couldn't find how can I create one file for a bulk of files.
So, how to dynamically generate one file, using source_gen, for a list of files?
Example:
Files
user.dart
category.dart
Generated:
info.dart (containg information from user.dart and category.dart)
Found out how to do it with the help of people in Gitter.
You must have one file, even if empty, to call the generator. In my example, it is lib/batch.dart.
source_gen: ^0.5.8
Here is the working code:
The tool/build.dart
import 'package:build_runner/build_runner.dart';
import 'package:raoni_global/phase.dart';
main() async {
PhaseGroup pg = new PhaseGroup()
..addPhase(batchModelablePhase(const ['lib/batch.dart']));
await build(pg,
deleteFilesByDefault: true);
}
The phase:
batchModelablePhase([Iterable<String> globs =
const ['bin/**.dart', 'web/**.dart', 'lib/**.dart']]) {
return new Phase()
..addAction(
new GeneratorBuilder(const
[const BatchGenerator()], isStandalone: true
),
new InputSet(new PackageGraph.forThisPackage().root.name, globs));
}
The generator:
import 'dart:async';
import 'package:analyzer/dart/element/element.dart';
import 'package:build/build.dart';
import 'package:source_gen/source_gen.dart';
import 'package:glob/glob.dart';
import 'package:build_runner/build_runner.dart';
class BatchGenerator extends Generator {
final String path;
const BatchGenerator({this.path: 'lib/models/*.dart'});
#override
Future<String> generate(Element element, BuildStep buildStep) async {
// this makes sure we parse one time only
if (element is! LibraryElement)
return null;
String libraryName = 'raoni_global', filePath = 'lib/src/model.dart';
String className = 'Modelable';
// find the files at the path designed
var l = buildStep.findAssets(new Glob(path));
// get the type of annotation that we will use to search classes
var resolver = await buildStep.resolver;
var assetWithAnnotationClass = new AssetId(libraryName, filePath);
var annotationLibrary = resolver.getLibrary(assetWithAnnotationClass);
var exposed = annotationLibrary.getType(className).type;
// the caller library' name
String libName = new PackageGraph.forThisPackage().root.name;
await Future.forEach(l.toList(), (AssetId aid) async {
LibraryElement lib;
try {
lib = resolver.getLibrary(aid);
} catch (e) {}
if (lib != null && Utils.isNotEmpty(lib.name)) {
// all objects within the file
lib.units.forEach((CompilationUnitElement unit) {
// only the types, not methods
unit.types.forEach((ClassElement el) {
// only the ones annotated
if (el.metadata.any((ElementAnnotation ea) =>
ea.computeConstantValue().type == exposed)) {
// use it
}
});
});
}
});
return '''
$libName
''';
}
}
It seems what you want is what this issue is about How to generate one output from many inputs (aggregate builder)?
[Günter]'s answer helped me somewhat.
Buried in that thread is another thread which links to a good example of an aggregating builder:
1https://github.com/matanlurey/build/blob/147083da9b6a6c70c46eb910a3e046239a2a0a6e/docs/writing_an_aggregate_builder.md
The gist is this:
import 'package:build/build.dart';
import 'package:glob/glob.dart';
class AggregatingBuilder implements Builder {
/// Glob of all input files
static final inputFiles = new Glob('lib/**');
#override
Map<String, List<String>> get buildExtensions {
/// '$lib$' is a synthetic input that is used to
/// force the builder to build only once.
return const {'\$lib$': const ['all_files.txt']};
}
#override
Future<void> build(BuildStep buildStep) async {
/// Do some operation on the files
final files = <String>[];
await for (final input in buildStep.findAssets(inputFiles)) {
files.add(input.path);
}
String fileContent = files.join('\n');
/// Write to the file
final outputFile = AssetId(buildStep.inputId.package,'lib/all_files.txt');
return buildStep.writeAsString(outputFile, fileContent);
}
}

Resources