sorry for bother again, im having some trouble in chunk some files to send them as a POST request. I made this function to post text to dpaste.com:
void dpaste_create(WiFiClientSecure *client, HTTPClient *http, String title, String data) {
http->begin(*client, "https://dpaste.com/api/");
http->addHeader("Authorization", "TOKEN");
http->addHeader("Content-type", "application/x-www-form-urlencoded");
http->POST("content=" + data + "&title=" + title + "&expiry_days=30");
Serial.printf("[DEBUG]: %s\n", http->getString().c_str());
http->end();
}
But now im trying to send the data of a file (400kb) but i only have 40kb of ram. Any have some idea how to chunk and send the sizes with "Transfer-Encoding: chunked"?
Any suggestion is appreciated. Thanks a lot for the time.
Related
Using express and Node.js, I'm using the twitter streaming API and the needle npm package for accessing APIs to pull tweets related to keywords. The streaming is functional and I am successfully pulling tweets using the following (simplified) code:
const needle = require('needle');
const TOKEN = // My Token
const streamURL = 'https://api.twitter.com/2/tweets/search/stream';
function streamTweets() {
const stream = needle.get(streamURL, {
headers: {
Authorization: `Bearer ${TOKEN}`
}
});
stream.on('data', (data) => {
try {
const json = JSON.parse(data); // This line appears to be causing my error
const text = json.data.text;
} catch (error) {
console.log("error");
}
});
}
However, no matter which search term I use (and the subsequent large or small volume of tweets coming through), the catch block will consistently log 1-3 errors per minute, which look like this:
SyntaxError: Unexpected end of JSON input
at JSON.parse (<anonymous>)
at PassThrough.<anonymous> (C:\Users\danie\OneDrive\Documents\Personal-Projects\twitter-program\server.js:56:31)
at PassThrough.emit (events.js:315:20)
at addChunk (internal/streams/readable.js:309:12)
at readableAddChunk (internal/streams/readable.js:284:9)
at PassThrough.Readable.push (internal/streams/readable.js:223:10)
at PassThrough.Transform.push (internal/streams/transform.js:166:32)
at PassThrough.afterTransform (internal/streams/transform.js:101:10)
at PassThrough._transform (internal/streams/passthrough.js:46:3)
at PassThrough.Transform._read (internal/streams/transform.js:205:10).
I've seen previous advice which says that data can be fired in multiple chunks, and to push the chunks to an array i.e. something like the following:
let chunks = [];
stream.on('data', (dataChunk) => {
chunks.push(dataChunk);
}).on('end',() => {
// combine chunks to create JSON object
})
But this didn't work either (may have been my implementation but I don't think so) and now I'm wondering if it's perhaps an error with the twitter API, because most of the tweet objects do come through correctly. I should note that the streamTweets() function above is called from an async function, and I am also wondering if that is having something to do with it.
Has anyone else encountered this error? Or does anyone have any idea how I might be fix it? Ideally i'd like 100% of the tweets to stream correctly.
Thanks in advance!
For future readers, this error is triggered by Twitter's heartbeat message that is sent every 20 seconds. Per the documentation:
The endpoint provides a 20-second keep alive heartbeat (it will look like a new line character).
Adding a guard against parsing the empty string will prevent the JSON parsing error.
if (data === "")
return
An empty string is invalid JSON, hence the emitted error.
Now, acknowledging that the heartbeat exists, it may be beneficial to add read_timeout = 20 * 1000 in the needle request to avoiding a stalled program with no data, be that due to a local network outage or DNS miss, etc.
I'm playing around with the Twilio MMS API, and I'm wondering if there's a simple way to show a Gif attachment after the body text?
So far I've only been able to send the Gif before the text.
I understand the behavior you are talking about. I'm sure you've figured it out by now, but if not, here is a hack:
Send a blank text file first, which means put it last, as the files in this Twilio SMS/MMS (twiml too) are processed in reverse order.
Twilio Functions Example (node.js)
exports.handler = function(context, event, callback) {
var client = context.getTwilioClient();
let dataMMS = {
to: ['+15556667777'],
from: '+15551112222', // your twilio num
body: 'I am the body',
mediaUrl: [
'http://s3.amazonaws.com/blog.invisionapp.com/uploads/2014/12/motion-example.gif',
'https://sepia-jaguar-8734.twil.io/assets/blank.csv'
],
contentType: [
'image/gif',
'text/csv'
]
};
client.messages.create(dataMMS).then(function(response){
console.log('Message Sent : ' + response.status);
return callback(null, 200); //response.status, queued = good (200)
}).catch(function(err){
console.log('An error is happening: ' + err);
return callback(null, err);
});
}
Extra:
This gif link was selected from a random link https://www.invisionapp.com/blog/7-tips-for-designing-awesome-gifs/. It was selected because it was nice and small, ~25 kb.
In my testing with Twilio and my carrier, I found gifs < ~100 kb worked and if I remember correctly they are not run thru a compressor like png/jpegs.
Note to Twilio employee: The Twilio public asset is in a dummy sub account, I'd be happy to replace it with an asset not related to me.
Oh yeah, make sure your urls have proper mime type encoding: https://www.twilio.com/docs/api/messaging/accepted-mime-types
I think Twilio is forgiving for .jpg and .png , as they are run thru compressors...
ps This will give you an extra blank line in your text block, you can get rid of it. Share your answer when you discover how, cause I've shared enough today.
Hoping to find an example of using Flutter to send a form post to Node.js and pass a string that is JSON and an image file.
I have been trying http.MultipartRequest, but the node service does not get anything in files or fields.
Not sure what I am doing wrong, so any guidance would be great.
var _url = Uri.parse('http://$baseurl/chat/messages/send');
var request = new http.MultipartRequest('POST', _url);
request.fields['json'] = json;
request.files.add(new http.MultipartFile.fromString(widget.mychat.msgkey, myimagefile.path));
Am I missing an easier way? Tried to BASE64 encode the image and add as a string in JSON, but the server keeps crashing even after I changed the amount of client request nginx would allow. I need it to work for an image or video sent from mobile.
I was having the exact same issue. The problem is not with MultipartRequest but the nodejs server.
The body-parser module you are using in the nodejs API can not handle multipart/form-data , it can only handle x-www-form-urlencoded
The solution is : You have to use module like multer in your nodejs API
const multer = require('multer')
let upload = multer({ storage: multer.memoryStorage() });
router.route('/signup').post(upload.single('avatar'),function (req, res) {
console.log('signup page');
console.log(req.file);
console.log(req.body);
//res.status(200).json({"success": "Signup page"});
res.end('hello world');
});
Found a nice solution here http://derpturkey.com/node-multipart-form-data-explained/
I am trying to use refit to upload to azure blob storage from a Xamarin iOS application. This is the interface configuration I am using for Refit:
[Headers("x-ms-blob-type: BlockBlob")]
[Put("/{fileName}")]
Task<bool> UploadAsync([Body]byte[] content, string sasTokenKey,
[Header("Content-Type")] string contentType);
Where the sasTokenKey parameter looks like this:
"/content-default/1635839001660743375-66f93195-e923-4c8b-a3f1-5f3f9ba9dd32.jpeg?sv=2015-04-05&sr=b&sig=Up26vDxQikFqo%2FDQjRB08YtmK418rZfKx1IHbYKAjIE%3D&se=2015-11-23T18:59:26Z&sp=w"
This is how I am using Refit to call the azure blob server:
var myRefitApi = RestService.For<IMyRefitAPI>("https://myaccount.blob.core.windows.net");
myRefitApi.UploadAsync(photoBytes, sasTokenKey, "image/jpeg"
However I am getting the follow error:
Response status code does not indicate success: 403 (Server failed to
authenticate the request. Make sure the value of Authorization header is
formed correctly including the signature.)
The SAS url is working fine if I call it directly like this
var content = new StreamContent(stream);
content.Headers.Add("Content-Type", "jpeg");
content.Headers.Add("x-ms-blob-type", "BlockBlob");
var task = HttpClient.PutAsync(new Uri(sasTokenUrl), content);
task.Wait();
So basically I am just trying to do the same thing using Refit.
Any idea how to get Refit working with Azure Blob Storage?
Thanks!
[UPDATE] I am now able to upload the bytes to the azure blob server but something seems to be wrong with the byte data because I am not able to view the image. Here is the code I am using to convert to byte array.
byte[] bytes;
using (var ms = new MemoryStream())
{
stream.Position = 0;
stream.CopyTo(ms);
ms.Position = 0;
bytes = ms.ToArray();
}
[UPDATE] Got it fixed by using stream instead of byte array!
I see %2F and %3D and I'm curious if refit is encoding those a second time. Try sending the token without encoding it.
This is incorrect use of Authorization header. You use Authorization header when you want to authorize the requests using account key. If you have the Shared Access Signature then you really don't need this header as the authorization information is included in the SAS itself. You can simply use the SAS URL for uploading files.
I want the user to be able to upload a file via my application. I don't have DB access, all my data calls get completed via a web-service that another person is writing. I needed to secure the web service, so I've consumed it & exposed it via WebAPI, & added OAuth security.
Now to my problem.
I've written the following.
public Task<FileResult> Post()
{
if (Request.Content.IsMimeMultipartContent())
{
var task = Request.Content.ReadAsByteArrayAsync().ContinueWith(
o =>
{
var result = this.Client.UploadPicture(this.UserId, o.Result);
if (result.ResultCode == 0)
{
return new FileResult()
{
Message = "Success",
FileId = result.ServerId
};
}
throw new HttpResponseException(...);
});
return task;
}
...
}
I'm pretty much a noob when it comes to WebAPI & multithreading (I'm not sure why this needs to be handled async? I'm sure there is a reason, but for now I'd just like a working example and get to the why later..).
My code is loosely based on some R&D & samples I've found on the net, but i haven't come across a scenario like I'm needing to complete... Yet it doesn't seem like I'm doing something out of the ordinary...
Upload a file to the server, and pass the image byte[] object to either sql or another service?
In this line
var result = this.Client.UploadPicture(this.UserId, o.Result);
I'm uploading a byte[] array of something....
Then later (the retrieval method works, I've managed to retrieve & view a test image)
When retrieving the byte array of the "image" i uploaded i get an array of idk what.. EG, i get a valid result of something, but it ain't no picture. Which leads me to believe that the uploaded data is bogus :|
O_o
How to get the image byte[]?
Mime Multipart is more than just your array of bytes. It also has metadata and boundary stuff. You need to treat it as MultiPartContent and then extract the image byte array out of that.
Filip has a blog post on the subject here.