I am using transloadit to process image uploads with rails. I have included all the fields (fields: "*") so they are submitted with the params. I would now like to use them in the assembly instructions to rename the files. See the relevant excerpt of the instructions,
"export": {
"use": [
"base",
"large",
"medium",
"thumb"
],
"robot": "/s3/store",
"key": "********",
"secret": "********",
"bucket": "********",
"path": "${unique_original_prefix}/${previous_step.name}/${fields.coach[name]}.${file.ext}"
}
However this does not work. The resulting files are,
5e
/f88480973a11e49ecf65da10504cf1
/base
/.jpg
/large
/.jpg
/medium
/.jpg
/thumb
/.jpg
What am I doing wrong?
Bonus:
Also is there a way to parameterize the field values with transloadit or should I just have a hidden input field which gets set with the proper value when the form is submitted. This I guess would also allow me to circumvent the first problem but somehow that feels dirty.
you are using ${unique_original_prefix}, which works like:
This is like ${unique_prefix} with the exception that two files that are encoding results of the same uploaded file (the original file) will have the same prefix value here.
And then for ${unique_prefix}:
A unique 33-character prefix used to avoid file name collisions, such as “f2/d3eeeb67479f11f8b091b04f6181ad”
Please notice how both these assembly variables will create a two char subdirectory, which you are seeing in your results. And the rest of the path is expected.
If you'd like a unique prefix without the two-char subfolders, please use ${assembly.id} instead of ${unique_original_prefix}.
About the bonus, you can just add a "fields" object in your assembly params (just like you are adding "auth" and "steps"). Those will also be available as fields. You could for example add your own unique prefix by sending it in that object and then using it as ${fields.my_custom_unique_prefix}. Just make sure you use "my_custom_unique_prefix" as the key in your fields object.
Kind regards,
Tim
Co-Founder Transloadit
#tim_kos
Related
I'm currently coding a transition from a system that used hand-crafted JSON files to one that can automatically generate the JSON files. The old system works; the new system works; what I need to do is transfer data from the old system to the new one.
The JSON files are used by an iOS app to provide functionality, and have never been read by our server software in Ruby On Rails before. To convert between the original system and the new system, I've started work on parsing the existing JSON files.
The problem is that one of my first two sample files has trailing commas in the JSON:
{ "sample data": [1, 2, 3,] }
This apparently went through just fine with the iOS app, because that file has been in use for a while. Now I need some way to parse the data provided in the file in my Ruby on Rails server, which (quite rightfully) throws an exception over the illegal trailing comma in the JSON file.
I can't just JSON.parse the code, because the parser, quite rightfully, rejects it as invalid JSON. Is there some way to parse it -- either an option I can pass to JSON.parse, or a gem that adds something, etc etc? Or do I need to report back that we're going to have to hand-fix the broken files before the automated process can process them?
Edit:
Based on comments and requests, it looks like some additional data is called for. The JSON files in question are stored in .zip files on S3, stored via ActiveStorage. The process I'm writing needs to download, unpack, and parse the zip files, using the 'manifest.json' file as a key to convert the archived file into a database structure with multiple, smaller files stored on S3 instead of a single zip that contains everything. A (very) long term goal is for clients to stop downloading a unitary zip file, and instead download the files individually. The first step towards that is to break the zip files up on the server, which means the server needs to read in the zip files. A more detailed sample of the data follows. (Note that the structure contains several design decisions I later came to regret; one of the original ideas was to be able to re-use files rather than pack multiple copies of the same identical file, but YAGNI bit me in the rear there)
The following includes comments that are not legal in JSON format:
{
"defined_key": [
{
"name": "Object_with_subkeys",
"key": "filename",
"subkeys": [
{
"id":"1"
},
{
"id":"2"
},
{
"id":"3" // references to identifier on another defined key
}, // Note trailing comma
]
}
],
"another_defined_key":[
{
"identifier": "should have made parent a hash with id as key instead of an array",
"data":"metadata",
"display_name":"Names: Can be very arbitrary",
"user text":"Wait for the right {moment}", // I actually don't expect { or } in the strings, but they're completely legal and may have been used
"thumbnail":"filename-2.png",
"video-1":"filename-3.mov"
}
]
}
The problem is that your are trying to parse something that looks a lot like JSON but is not actually JSON as defined by the spec.
Arrays- An array structure is a pair of square bracket tokens surrounding zero or more values. The values are separated by commas.
Since you have a trailing comma another value is also expected and most JSON parsers will raise an error due to this violation
All that being said json-next will parse this appropriately maybe give that a shot.
It can parse JSON like representations that completely violate the JSON spec depending on the flavor you use. (HanSON, SON, JSONX as defined in the gem)
Example:
json = "{ \"sample data\": [1, 2, 3,] }")
require 'json/next'
HANSON.parse(json)
#=> {"sample data"=>[1, 2, 3]}
but the following is equivalent and completely violates spec
JSONX.parse("{ \"sample data\": [1 2 3] }")
#=> {"sample data"=>[1, 2, 3]}
So if you choose this route do not expect to use this to validate the JSON data or structure in any fashion and you could end up with unintended results.
I have been using Karate and RestAssured for sometime. There are advantages and downside of both tools of course. Right now I have a RestAssured project where I have Request and Response object and POJOs. My requests wraps my endpoint and send my POJOs to those endpoint. I do all my Headers, etc configuration in an abstract layer. In case I need to override them, I override them during the test. If not, Its a two lines of code for me to trigger an endpoint.
My way of working with happy path and negative path of an edpoint is that I initialize the POJO before every test with new values in the constructor. Then I override the value that I want in test scope. For example, if I want to test a negative case for password field, I set this field to empty string during the test. But other fields are already set to some random stuff before the test.
But I dont know how to achieve this with Karate.
Karate allows me to create a JSON representation of my request body and define my parameters as seen below example.
{
"firstName": "<name>",
"lastName": "<lastName>",
"email": "<email>",
"role": <role>
}
Then in every test I have to fill all the fields with some data.
|token |value|
|name |'canberk'|
|lastName |''|
|email |'canberk#blbabla.com'|
|role |'1'|
and
|token |value|
|name |''|
|lastName |'akduygu'|
|email |'canberk#blbabla.com'|
|role |'1'|
It goes on like this.
It's ok with a 4 fields JSON body but when the body starts to have more than 20 fields, it become a pain to initialise every field for every test.
Does Karate have a way of achieving this problem with a predefined steps of I need to come up with a solution?
There are advantages and downside of both tools of course.
I'm definitely biased, but IMHO the only disadvantage of Karate compared to REST-assured is that you don't get compile time safety :) I hope that you have seen this comparison.
Karate has multiple ways to do what you want. Here's what I would do.
create a JSON file that has all your "happy path" values set
use the read() syntax to load the file (which means this is re-usable across multiple tests)
use the set keyword to update only the field for your scenario or negative test
You can get even more fancier if you use embedded expressions.
create a JSON file that has all your "happy path" values set and the values you want to vary look like foo: '##(foo)'
before using read() you init some variables for e.g. * def foo = 'bar' and if you use null that JSON key will even be removed from the JSON
read() the JSON. it is ready for use !
You can refer to this file that demonstrates some of these concepts for XML, and you may get more ideas: xml.feature
Say I have a ruby model which has a name and age attribute. A GET request for one of these objects returns something like this when using rails generate scaffold:
{
"id": 1,
"name": "foo",
"age": 21,
"parent_id": 1
}
By default a POST/PUT to this resource expects:
{
"user": {
"name": "foo",
"age": 21,
"parent_id": 1
}
}
When using nested resources configured in routes the default behaviour is to add the parent id outside of this nested hash too, e.g.: PUT /parents/1/users:
{
"parent_id": 1,
"user": {
"name": "foo",
"age": 21
}
}
I can go to the controller simply enough and alter what parameters are expected, but I'd like to know why that is the case and if I risk breaking anything if changing it.
More specifically this is a Rails API and I'd like to add swagger doc generation to the API, so having this asymmetrical request body is annoying.
So in summary my questions are:
What are the advantages of this, why is it the Rails default and what do I risk breaking by changing it?
How best to add swagger support to the API in a way which doesn't have different GET responses vs PUT/POST (which seems like bad design to me, but maybe I'm wrong)?
How best/should I make the API automatically add the parent id when making a call like POST /parents/1/users, because again the default generation doesn't support it and I'm wondering if there's a reason
What are the advantages of this?
This is perhaps an opinion-based answer, which is generally frowned upon by StackOverflow, but here's my 2 cents.
In the GET request, you are simply being returned a resource. So the attributes are all you need to know:
{
"id": 1,
"name": "foo",
"age": 21,
"parent_id": 1
}
On the other hand, for this PUT request:
{
"parent_id": 1,
"user": {
"name": "foo",
"age": 21
}
}
You can think of the parameters as being split into two "sections": The parent_id (which would normally get sent as a path param, not part of the request body!) is something to "search/filter" by, whereas the user params are the attributes of the user resource to update.
This logical separation of concerns is particularly useful in the context of web forms (which is what Rails was originally/primarily designed for), especially when dealing with complex queries or "nested" attributes.
what do I risk breaking by changing it?
Nothing really.
That format, however, was "optimised" for the context of RESTful APIs and web forms.
If you'd rather use some other format then go ahead; Rails isn't forcing you to use anything here. Just beware that a naive "better design" may come back to bite you down the line.
How best to add swagger support to the API in a way which doesn't have different GET responses vs PUT/POST (which seems like bad design to me, but maybe I'm wrong)?
You can design the API any way you like. If you want "flat parameters" everywhere, then just build the Rails application like that.
How best/should I make the API automatically add the parent id when making a call like POST /parents/1/users, because again the default generation doesn't support it and I'm wondering if there's a reason
I'm not sure what you mean by "the default generation doesn't support it". The default generation of what? The swagger docs? The rails application?
Anyway... That should be implemented as a path parameter. The swagger docs should look something like this:
/parents/{parent_id}/users:
get:
description: '.....'
parameters:
- name: parent_id
in: path
description: 'ID of the parent'
required: true
type: integer
Tom Lord’s answer and note is probably better than mine.
My guess is that this mimics the behaviour of HTTP. If you GET data, you can add parameters (?name=foo). However, if you POST data, you tend to put the payload in the body of the request. And not have any parameters in the URL.
It’s likely that Rails thinks that you’re going put that JSON object into the body of the request. Whereas the GET request it’s going to split the key/values apart and send as parameters.
The advantages of keeping it the way they are is that it’ll avoid a gotcha later. I’d argue this is always the best thing to do in programming, but also especially something like Rails. But, if you’re making an API I can see why you’d want to let people send data as parameters rather than a body that needs validating.
As for Swagger, let the user know they need to send the data as a JSON string, and then use the parameters feature as expected.
Last one is a bit tricky. I guess it’s up to the design of your API. You could pass it as part of the request. Maybe take a look through sometihng like RESTful API Design to clarify your goal.
Two questions:
When you define a new route and you want to protect it by requiring an api_key to be set in the header, do you do it by specifying a security section under that route in the yaml file, or do you put it under parameters (or in both)? If it works by putting it under parameters, what's the point of putting it under security?
The yaml file can tell you that certain routes are protected by an api_key, but it doesn't specify the value of that key (as far as I know). Does it mean I need the write some custom middleware that intercepts my routes and checks the validity of the key, or it can be auto-generated by one of the swagger tools?
Thanks.
Im currently on the same journey as you. I do know this:
What to add in my swagger.json to accept api key
https://github.com/OAI/OpenAPI-Specification/blob/master/versions/2.0.md#securityRequirementObject
"security": [
{
"api_key": []
}
]
what to add in my controller ...
https://github.com/apigee-127/swagger-tools/blob/master/docs/Middleware.md
Pretty much use swagger-tools to use the middleware.swaggerSecurity.
What they dont tell you is how to validate the api_key
Well I found that it's simply taking the 'scopes' parameter which holds the actual key and comparing that against a list of valid keys that you have stored somewhere.
What I dont get is how to allow the flow to go to the next function (like 'next()' in express) or stop if there is an invalid API key supplied. There is a 'callback' parameter, but I dont know how to use it (yet).
I hope this works.
You do need to implement the security checks on your own. Swagger will help you define them, but implementing is your job.
I would like to develop URL's which look like the following:
http://mysite.com/products/1/best-product-in-the-world
Where all i need to get to the proper record is the following route:
http://mysite.com/products/1
When I add the product description piece to the URL ("best-product-in-the-world") I get URL encoding issues. I've tried to use Server.UrlEncode when constructing this portion of my URL in an ActionLink(...):
<%= Html.ActionLink(item.Subject, "../Post/Detail",
new { id = item.ID,
descriptiveUrl = Server.UrlEncode(Product.ShortDescription) },
new { rel = "canonical",
title = Product.ShortDescription,
#class = "product-hyperlink" })%>
But this renders regularly encoded elements for special characters and spaces, much like the following:
http://localhost:2392/Products/Detail/1/best+product+in+the+world253f
...which creates a 400, bad request exception. Not sure I've done the question justice, but can provide further clarification if need be.
Update: this post's URL is as follows, and i'm trying to do something very similar!
http://stackoverflow.com/questions/1148955/creating-search-engine-friendly-urls-in-asp-net-mvc
In a deeper Google search, I found the following link for generating slugs:
http://www.intrepidstudios.com/blog/2009/2/10/function-to-generate-a-url-friendly-string.aspx
Thanks #Rob and #Coding the Wheel for giving me the terminology I really needed to find this answer!
A simple option would be to add a property to your model object with an accessor that normalises the appropriate field (short description in this case) down to a suitable "slug"; that is, the bit of junk text after the identifier. You then use this when constructing the URI.
The normalisation process might be as simple as removing any non-alphanumeric characters and replacing spaces with hyphens.
The standard practice here is to store a 'slug' with each post that will function as the post's outward-facing URL. For example, your slug for the above post would be:
best-product-in-the-world
A decent CMS will do this for you automatically, and allow you to tweak the slug before saving.