Adding not validated dictionary with python eve along with an image - eve

I want to import images in mongoDB along with any dictionary. The dictionary should provide image tags, of which types, numbers and names I can't know at the moment I define the schema.
I'm trying to add a dictionary in eve without success:
curl -F"attr={\"a\":1}" -F "img_id=2asdasdasd" -F "img_data=#c:\path\
1.png;type=image/png" http://127.0.0.1:5000/images
{"_status": "ERR", "_issues": {"attr": "must be of dict type"}, "_error": {"message": "Insertion failure: 1 document(s)
contain(s) error(s)", "code": 422}}
My schema definition looks like that:
'schema': {
#Fixed attributes
'original_name': {
'type': 'string',
'minlength': 4,
'maxlength': 1000,
},
'img_id': {
'type': 'string',
'minlength': 4,
'maxlength': 150,
'required': True,
'unique': True,
},
'img_data': {
'type': 'media'
},
#Additional attributes
'attr': {
'type': 'dict'
}
}
Is it possible at all? Should the schema for dicts be fixed?
EDIT
I wanted to add image first and the dictionary after it, but get an error in PATCH request:
C:\Windows\SysWOW64>curl -X PATCH -i -H "Content-Type: application/json" -d "{\
"img_id\":\"asdasdasd\", \"attr\": {\"a\": 1}}" http://localhost:5000/images/asd
asdasd
HTTP/1.0 405 METHOD NOT ALLOWED
Content-Type: application/json
Content-Length: 106
Server: Eve/0.7.4 Werkzeug/0.9.4 Python/2.7.3
Date: Wed, 28 Jun 2017 22:55:54 GMT
{"_status": "ERR", "_error": {"message": "The method is not allowed for the requested URL.", "code": 405}}

I have posted an issue on Github for the same situation. However I have came with a workaround.
Override the dict validator:
class JsonValidator(Validator):
def _validate_type_dict(self, field, value):
if type(value) is dict:
pass
try:
json.loads(value)
except:
self._error(value, "Invalid JSON")
app = Eve(validator=JsonValidator)
Next, add an insert hook:
def multi_request_json_parser(documents):
for item in documents:
if 'my_json_field' in item.keys():
item['my_json_field'] = json.loads(item['my_json_field'])
app.on_insert_myendpoint += multi_request_json_parser
The dict validator must be overidden, because else the insert hook will not be called due to an validation error.

Related

How to retreive all the records from elasticsearch in rails

There is an upper limit on the number of docs you can get from elastic search(that is 10000). we can use "scroll" to retrieve all the records. Does anyone know how to embed this in code?
There is this method scroll
https://github.com/elastic/elasticsearch-ruby/blob/4608fd144277941003de71a0cdc24bd39f17a012/elasticsearch-api/lib/elasticsearch/api/actions/scroll.rb
But I don't know how to use it. Could you explain how to use it?
I have tried the "scan". But it is no longer supported in Elasticsearch anymore.
# Open the "view" of the index
response = client.search index: 'test', search_type: 'scan', scroll: '5m', size: 10
# Call `scroll` until results are empty
while response = client.scroll(scroll_id: response['_scroll_id'], scroll: '5m') and not
response['hits']['hits'].empty? do
puts response['hits']['hits'].map { |r| r['_source']['title'] }
end
Your code should work, but as you mentioned the scan parameter for search_type is not necessary. I just ran this locally with some test data and it worked:
# scroll.rb
require 'elasticsearch'
client = Elasticsearch::Client.new
response = client.search(index: 'articles', scroll: '10m')
scroll_id = response['_scroll_id']
while response['hits']['hits'].size.positive?
response = client.scroll(scroll: '5m', body: { scroll_id: scroll_id })
puts(response['hits']['hits'].map { |r| r['_source']['title'] })
end
Output:
$ ruby scroll.rb
Title 297
Title 298
Title 299
Title 300
...
You can fiddle around with the value for the scroll parameter, but something like this should work for you too.
paragraph from elastic official docs :
We no longer recommend using the scroll API for deep pagination. If
you need to preserve the index state while paging through more than
10,000 hits, use the search_after parameter with a point in time
(PIT).
Scroll Official Doc Link
I recommend to use pagination.
you can use
that limitation in number of hits is for performance inprovements, you can use pagination, Its much faster.
in this way you can use start point with form key or use search_after key with sort and PIT(point in time for prevent from inconsistent result). and you can determinate you hits size key with 10 for faster query time.
Pagination Official Doc Link
for instantiate PIT ID:
POST /test/_pit?keep_alive=1m
for instantiate pagination:
GET /test/_search
{
"size": 10,
"query": {
"match" : {
"user.id" : "elkbee"
}
},
"pit": {
"id": "46ToAwMDaWR5BXV1aWQyKwZub2RlXzMAAAAAAAAAACoBYwADaWR4BXV1aWQxAgZub2RlXzEAAAAAAAAAAAEBYQADaWR5BXV1aWQyKgZub2RlXzIAAAAAAAAAAAwBYgACBXV1aWQyAAAFdXVpZDEAAQltYXRjaF9hbGw_gAAAAA==",
"keep_alive": "1m"
},
"sort": [
{"#timestamp": {"order": "asc", "format": "strict_date_optional_time_nanos", "numeric_type" : "date_nanos" }}
]
}
for get rest of data in pagination:
there is sort key in the result, put it in the search_after
GET /test/_search
{
"size": 10,
"query": {
"match" : {
"user.id" : "elkbee"
}
},
"pit": {
"id": "46ToAwMDaWR5BXV1aWQyKwZub2RlXzMAAAAAAAAAACoBYwADaWR4BXV1aWQxAgZub2RlXzEAAAAAAAAAAAEBYQADaWR5BXV1aWQyKgZub2RlXzIAAAAAAAAAAAwBYgACBXV1aWQyAAAFdXVpZDEAAQltYXRjaF9hbGw_gAAAAA==",
"keep_alive": "1m"
},
"sort": [
{"#timestamp": {"order": "asc", "format": "strict_date_optional_time_nanos"}}
],
"search_after": [
"2021-05-20T05:30:04.832Z", #you can find this value from sort key in response
4294967298
],
"track_total_hits": false
}

Ruby Grape: How to send array of hashes as param

I sends an array of hashes.
[{question_id: 1_id, answer:1_String},{question_id: 2_id, answer:2_String}]
and I used this code in my API file:
requires :profile_setting, type: Array[Hash], desc: "[{question_id: 1_id, answer: '1_String'},{question_id: 2_id, answer: '2_String'}]"
params: [{question_id: 1_id, answer:1_String},{question_id: 2_id, answer:2_String}]
response:
{
"error": "profile_setting is invalid"
}
how to send a Array of multiple hashes.
Your JSON misses quotes around keys and values.
Should be
[{
"question_id": "1 _id",
"answer": "1 _String"
},
{
"question_id": "2 _id",
"answer": "2 _String"
}]

Remove metadata from json results

I'm using ActiveResource to establish a REST connection with rails 4.2 to an ADS Advantage server using the WebPlatform from ADS. It returns json with "__metadata". How can I remove the "__metadata"?
{
"__metadata": {
"uri": "http://.....",
"key_fields": "ID",
"rows_affected": 0,
"last_autoinc": 0
},
In my class I have added self.include_format_in_path = false, to remove the .json from the end of the uri.
Thanks.
you can achieve this in the following steps:
parse the JSON:
parsed_json = JSON.parse('{ "__metadata": { "uri": "http://.....", "key_fields": "ID", "rows_affected": 0, "last_autoinc": 0 }}')
then you will get a hash type and you just need to get the inside of __metadata:
result = parsed_json['__metadata']
then you can just return it or print it:
puts result.to_json
#=> {"uri"=>"http://.....", "key_fields"=>"ID", "rows_affected"=>0, "last_autoinc"=>0}

ember-data: server side code for removing an associated object

I 'm working with revision 12 of ember-data RESTAdapter and using the rails-api gem.
I have these models:
App.TransportDocumentRow = DS.Model.extend,
productName: DS.attr 'string'
transportDocument: DS.belongsTo('App.TransportDocument')
App.TransportDocument = DS.Model.extend
number: DS.attr 'string'
transportDocumentRows: DS.hasMany('App.TransportDocumentRow')
configured in this way:
DS.RESTAdapter.map 'App.TransportDocument', {
transportDocumentRows: { embedded: 'always' }
}
(i'm using embedded: always becauseif i don't my document rows are committed with document_id = 0, as asked here
Consider i have already created a transport document (id: 1) with 2 rows. If i delete a row (with id: 1), the result would be a PUT request to /transport_documents/1.
The JSON sent with this put would be something like this:
{"transport_document"=>
{"number"=>"1", "transport_document_rows"=>
[
{"id"=>2, "product_name"=>"aaaa", "transport_document_id"=>1}
]
}, "id"=>"1"
}
while rails would expect something like this:
{"transport_document"=>
{"number"=>"1", "transport_document_rows"=>
[
{"id"=>1, "_delete"=>1}
{"id"=>2, "product_name"=>"aaaa", "transport_document_id"=>1}
]
}, "id"=>"1"
}
Is there a way specified in active_model_serializers to do this?
Or should i make some manual transformations my controller?
Or should i change the payload so that ember produces the correct request?

rails extract data from simple json response

I need to extract some data from a JSON response i'm serving up from curb.
Previously I wasn't calling symbolize_keys, but i thought that would make my attempt work.
The controller action:
http = Curl.get("http://api.foobar.com/thing/thing_name/catalog_items.json?per_page=1&page=1") do|http|
http.headers['X-Api-Key'] = 'georgeBushSucks'
end
pre_keys = http.body_str
#foobar = ActiveSupport::JSON.decode(pre_keys).symbolize_keys
In the view (getting undefined method `current_price' )
#foobar.current_price
I also tried #foobar.data[0]['current_price'] with the same result
JSON response from action:
{
"data": {
"catalog_items": [
{
"current_price": "9999.0",
"close_date": "2013-05-14T16:08:00-04:00",
"open_date": "2013-04-24T11:00:00-04:00",
"stuff_count": 82,
"minimum_price": "590000.0",
"id": 337478,
"estimated_price": "50000.0",
"name": "This is a really cool name",
"current_winner_id": 696969,
"images": [
{
"thumb_url": "http://foobar.com/images/93695/thumb.png?1365714300",
"detail_url": "http://foobar.com/images/93695/detail.png?1365714300",
"position": 1
},
{
"thumb_url": "http://foobar.com/images/95090/thumb.jpg?1366813823",
"detail_url": "http://foobar.com/images/95090/detail.jpg?1366813823",
"position": 2
}
]
}
]
},
"pagination": {
"per_page": 1,
"page": 1,
"total_pages": 131,
"total_objects": 131
}
}
Please note that accessing hash's element in Rails work in models. To use it on hash, you have to use OpenStruct object. It's part of standard library in rails.
Considering, #foobar has decoded JSON as you have.
obj = OpenStruct.new(#foobar)
obj.data
#=> Hash
But, note that, obj.data.catalog_items willn't work, because that is an hash, and again not an OpenStruct object. To aid this, we have recursive-open-struct, which will do the job for you.
Alternative solution [1]:
#foobar[:data]['catalog_items'].first['current_price']
But, ugly.
Alternative solution [2]:
Open Hash class, use method_missing ability as :
class Hash
def method_missing(key)
self[key.to_s]
end
end
Hope it helps. :)

Resources