I've got a route that returns details about a features on a user's account:
// games[{keys:games}].features[{integers:indices}]
{
$type : "atom",
value : {
id: "6",
count: "1",
...
}
}
There's also a route that returns generic details about specific features:
// features[{integers:features}]
{
$type : "atom",
value : {
name : "fooga",
max : 10,
...
}
}
I don't want to merge the generic feature data into the user-specific data because that will be a bunch of data duplication, but I also want to be able to get it all in a single request
What's a smart way to structure my routes/returned data so that games[{keys:games}].features[{integers:indices}] can return a useful reference to features[{integers:features}]?
I tried splitting them up like this:
// games[{keys:games}].features[{integers:indices}].details
{
$type : "atom",
value : {
id: "6",
count: "1",
...
}
}
// games[{keys:games}].features[{integers:indices}].meta
{
$type : "ref",
value : [
"features",
"15"
]
}
but I couldn't figure out a way to resolve the .meta reference w/o writing redundant-seeming paths like ...features.0.meta.[name,max,...]. Ideally the ref would just return an atom because it's a small amount of data.
I ended up structuring it like this:
games[{keys:games}].features[{integers:indices}].details
games[{keys:games}].features[{integers:indices}].feature
features[{keys:games}][{integers:features}].details
Ugly paths, but ¯\_(ツ)_/¯
Related
Sample Data:
"fields": {
"key1":0,
"key2":"no",
"Lang": {
"en": {
"firstName": "Vikrant",
"lastName":"Thakur"
},
"ch": {
"firstName": "维克兰特",
"lastName":"塔库尔"
}
}
}
Expected Response:
"fields": {
"Lang": {
"en": {
"firstName": "Vikrant",
"lastName":"Thakur"
}
}
}
I have added the following in my search-definition demo.sd:
struct lang {
field firstName type string {}
field lastName type string {}
}
field Lang type map <string, lang> {
indexing: summary
struct-field key {
indexing: summary | index | attribute
}
}
I want to write a yql query something like this (This doesn't work):
http://localhost:8080/search/?yql=select Lang.en from sources demo where key2 contains 'no';
My temporary workaround approach
I have implemented a custom searcher in MySearcher.java, through which I am able to extract the required sub-field and set a new field 'defaultLang', and remove the 'Lang' field. The response generated by the searcher:
"fields": {
"defaultLang": {
"firstName": "Vikrant",
"lastName":"Thakur"
}
}
I have written the following in MySearcher.java:
for (Hit hit: result.hits()) {
String language = "en"; //temporarily hard-coded
StructuredData Lang = (StructuredData) hit.getField("Lang");
Inspector o = Lang.inspect();
for (int j=0;j<o.entryCount();j++){
if (o.entry(j).field("key").asString("").equals(language)){
SlimeAdapter value = (SlimeAdapter) o.entry(j).field("value");
hit.setField("defaultLang",value);
break;
}
}
hit.removeField("Lang");
}
Edit-1: A more efficient way instead is to make use of the Inspectable interface and Inspector, like above (Thanks to #Jo Kristian Bergum)
But, in the above code, I am having to loop through all the languages to filter out the required one. I want to avoid this O(n) time-complexity and make use of the map structure to access it in O(1). (Because the languages may increase to 1000, and this would be done for each hit.)
All this is due to the StructuredData data type I am getting in the results. StructureData doesn't keep the Map Structure and rather gives an array of JSON like:
[{
"key": "en",
"value": {
"firstName": "Vikrant",
"lastName": "Thakur"
}
}, {
"key": "ch",
"value": {
"firstName": "维克兰特",
"lastName": "塔库尔"
}
}]
Please, suggest a better approach altogether, or any help with my current one. Both are appreciated.
The YQL sample query I guess is to illustrate what you want as that syntax is not valid. Picking a given key from the field Lang of type map can be done as you do in your searcher but deserializing into JSON and parsing the JSON is probably inefficient as StructuredData implements the Inspectable interface and you can inspect it directly without the need to go through JSON format. See https://docs.vespa.ai/documentation/reference/inspecting-structured-data.html
Modified the question to explain better:
I have two Falcor models from two different HttpDataSource, like below:
First model (User model):
const user_model = new falcor.Model(
{
source: new HttpDataSource('http://localhost:3000/api/userManagement')
});
user_model.get(['user', 'list'])
OUTPUT1:
{
"jsonGraph": {
"user": {
"list": {
"$type": "atom",
"value": {
"users": [...]
}
}
}
}
}
Second model (Role model):
const role_model = new falcor.Model(
{
source: new HttpDataSource('http://localhost:3000/api/roleManagement')
});
role_model.get(['role', 'list'])
OUTPUT2:
{
"jsonGraph": {
"role": {
"list": {
"$type": "atom",
"value": {
"roles": [...]
}
}
}
}
}
Is there a way to combine all these Falcor models into a single model?
The purpose is, if I try to do user_model.get(['user', 'list']) more than once it would get the data from Falcor-Model-Cache (after the first fetch from DB).
But if I try to do role_model.get(['user', 'list']), then I have to hit the DB again to get the data (inorder to store the same User list in role_model cache).
So instead if there is a way like below:
all_model = user_model + role_model
then I can do all_model.get(['user', 'list']) (or) all_model.get(['role', 'list']). So basically I would have only one combined Falcor-Model-Cache at the browser end.
Hope the question is more clear now.
You must use forkJoin
forkJoin(model1.source,model2.source).subscribe(res=>{
//in res[0] you have the response of model1.source
//in res[1] you have the response of model2.source
let data={...res[0],...res[1]}
//in data you have all the properties
}
I'm going to create app to share photo albums, using firebase.
Now I've issue with securing common objects.
My data object is
"Album" : {
"AbAY6YVhy6MLyvVjyYC517v62o22" : {
"-KeD0I9C-esXW2zA4uH_" : {
"backgroundPattern" : 0,
"collaboratorsIDS" :
[ "cS5O4Klt8CXrrKLJPVkHMaSTltW2","J55nZlr4SSPHL5GW7c7yrUkbAUl1", "LQ5mNECXAMQi9AoLCXh8GMsihf12" ],
"date" : "2017-03-02",
"direction" : 2,
"imageUrl" : "",
"ownerID" : "AbAY6YVhy6MLyvVjyYC517v62o22",
"scrapBookID" : "-KeD0I9C-esXW2zA4uH_",
"title" : "Test"
}
}
And question is, how to set .read and .write rules for albums. Main point is that collaboratorsIDS is user ID's that should have access to album.
"Album": {
"$albumId" : {
"$pushKey" : {
".read":"data.child('collaboratorsIDS').val().contains(auth.uid)"
}
}
With this, you can allow access to your node for collaboratorsIDS
I'm fairly new to Firebase and have a rankings app, where my structure currently looks like the following:
{
"Rankings" : {
"-KFGX5H3rLSnpPvupakm" : {
"Sports Teams" : {
"Red sox" : 1,
"Warriors" : 3,
"Yankees" : 2
}
},
"-KFGZkwAIl817CLDLmMp" : {
"Beers" : {
"Bud light" : 3,
"Coors" : 2,
"Pbr" : 4
}
}
}
}
I'm using childIDs so I can sort these chronologically. If I want to search rankings by name, how can I bypass the child ID to do so?
For instance, if a user searches for rankings using the term "Sports," how can I traverse my Rankings tree by searching for all rankings containing "Sports"?
This type of deep querying on dynamic paths is not possible with Firebase (nor with many other NoSQL databases). What you'll need to do is set up a so-called index, that maps the keys that you want to search for to the values that you want to find.
{
"Rankings" : {
"-KFGX5H3rLSnpPvupakm" : {
"Sports Teams" : {
"Red sox" : 1,
"Warriors" : 3,
"Yankees" : 2
}
},
"-KFGZkwAIl817CLDLmMp" : {
"Beers" : {
"Bud light" : 3,
"Coors" : 2,
"Pbr" : 4
}
}
},
"SearchTerms": {
"Red sox" : {
"-KFGX5H3rLSnpPvupakm": true
},
"Warriors" : {
"-KFGX5H3rLSnpPvupakm": true
},
"Yankees" : {
"-KFGX5H3rLSnpPvupakm": true
},
"Bud light" : {
"-KFGZkwAIl817CLDLmMp": true
},
"Coors" : {
"-KFGZkwAIl817CLDLmMp": true
},
"Pbr" : {
"-KFGZkwAIl817CLDLmMp": true
}
}
}
This process is called denormalizing your data and it's described in this blog post, in the Firebase documentation on structuring data and in this article on NoSQL data modeling. And in probably half a dozen similar answers I've given recently.
I was able to get the values of the child node-id's.
Here is my sample code:
LOCATION_URL.queryOrderedByChild("email").queryEqualToValue(FIRAuth.auth()?.currentUser?.email).observeEventType(.Value, withBlock: { snapShot in
let enumerator = snapShot.children
while let rest = enumerator.nextObject() {
var shot = rest.childSnapshotForPath(rest.key)
// child node ID
print(shot.key)
}
})
I hope this helps you..
Best regards,
Nazar Medeiros
Elasticsearch experts,
I have been unable to find a simple way to just tell ElasticSearch to insert the _timestamp field for all the documents that are added in all the indices (and all document types).
I see an example for specific types:
http://www.elasticsearch.org/guide/reference/mapping/timestamp-field/
and also see an example for all indices for a specific type (using _all):
http://www.elasticsearch.org/guide/reference/api/admin-indices-put-mapping/
but I am unable to find any documentation on adding it by default for all documents that get added irrespective of the index and type.
Elasticsearch used to support automatically adding timestamps to documents being indexed, but deprecated this feature in 2.0.0
From the version 5.5 documentation:
The _timestamp and _ttl fields were deprecated and are now removed. As a replacement for _timestamp, you should populate a regular date field with the current timestamp on application side.
You can do this by providing it when creating your index.
$curl -XPOST localhost:9200/test -d '{
"settings" : {
"number_of_shards" : 1
},
"mappings" : {
"_default_":{
"_timestamp" : {
"enabled" : true,
"store" : true
}
}
}
}'
That will then automatically create a _timestamp for all stuff that you put in the index.
Then after indexing something when requesting the _timestamp field it will be returned.
Adding another way to get indexing timestamp. Hope this may help someone.
Ingest pipeline can be used to add timestamp when document is indexed. Here, is a sample example:
PUT _ingest/pipeline/indexed_at
{
"description": "Adds indexed_at timestamp to documents",
"processors": [
{
"set": {
"field": "_source.indexed_at",
"value": "{{_ingest.timestamp}}"
}
}
]
}
Earlier, elastic search was using named-pipelines because of which 'pipeline' param needs to be specified in the elastic search endpoint which is used to write/index documents. (Ref: link) This was bit troublesome as you would need to make changes in endpoints on application side.
With Elastic search version >= 6.5, you can now specify a default pipeline for an index using index.default_pipeline settings. (Refer link for details)
Here is the to set default pipeline:
PUT ms-test/_settings
{
"index.default_pipeline": "indexed_at"
}
I haven't tried out yet, as didn't upgraded to ES 6.5, but above command should work.
You can make use of default index pipelines, leverage the script processor, and thus emulate the auto_now_add functionality you may know from Django and DEFAULT GETDATE() from SQL.
The process of adding a default yyyy-MM-dd HH:mm:ss date goes like this:
1. Create the pipeline and specify which indices it'll be allowed to run on:
PUT _ingest/pipeline/auto_now_add
{
"description": "Assigns the current date if not yet present and if the index name is whitelisted",
"processors": [
{
"script": {
"source": """
// skip if not whitelisted
if (![ "myindex",
"logs-index",
"..."
].contains(ctx['_index'])) { return; }
// don't overwrite if present
if (ctx['created_at'] != null) { return; }
ctx['created_at'] = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss").format(new Date());
"""
}
}
]
}
Side note: the ingest processor's Painless script context is documented here.
2. Update the default_pipeline setting in all of your indices:
PUT _all/_settings
{
"index": {
"default_pipeline": "auto_now_add"
}
}
Side note: you can restrict the target indices using the multi-target syntax:
PUT myindex,logs-2021-*/_settings?allow_no_indices=true
{
"index": {
"default_pipeline": "auto_now_add"
}
}
3. Ingest a document to one of the configured indices:
PUT myindex/_doc/1
{
"abc": "def"
}
4. Verify that the date string has been added:
GET myindex/_search
An example for ElasticSearch 6.6.2 in Python 3:
from elasticsearch import Elasticsearch
es = Elasticsearch(hosts=["localhost"])
timestamp_pipeline_setting = {
"description": "insert timestamp field for all documents",
"processors": [
{
"set": {
"field": "ingest_timestamp",
"value": "{{_ingest.timestamp}}"
}
}
]
}
es.ingest.put_pipeline("timestamp_pipeline", timestamp_pipeline_setting)
conf = {
"settings": {
"number_of_shards": 2,
"number_of_replicas": 1,
"default_pipeline": "timestamp_pipeline"
},
"mappings": {
"articles":{
"dynamic": "false",
"_source" : {"enabled" : "true" },
"properties": {
"title": {
"type": "text",
},
"content": {
"type": "text",
},
}
}
}
}
response = es.indices.create(
index="articles_index",
body=conf,
ignore=400 # ignore 400 already exists code
)
print ('\nresponse:', response)
doc = {
'title': 'automatically adding a timestamp to documents',
'content': 'prior to version 5 of Elasticsearch, documents had a metadata field called _timestamp. When enabled, this _timestamp was automatically added to every document. It would tell you the exact time a document had been indexed.',
}
res = es.index(index="articles_index", doc_type="articles", id=100001, body=doc)
print(res)
res = es.get(index="articles_index", doc_type="articles", id=100001)
print(res)
About ES 7.x, the example should work after removing the doc_type related parameters as it's not supported any more.
first create index and properties of the index , such as field and datatype and then insert the data using the rest API.
below is the way to create index with the field properties.execute the following in kibana console
`PUT /vfq-jenkins
{
"mappings": {
"properties": {
"BUILD_NUMBER": { "type" : "double"},
"BUILD_ID" : { "type" : "double" },
"JOB_NAME" : { "type" : "text" },
"JOB_STATUS" : { "type" : "keyword" },
"time" : { "type" : "date" }
}}}`
the next step is to insert the data into that index:
curl -u elastic:changeme -X POST http://elasticsearch:9200/vfq-jenkins/_doc/?pretty
-H Content-Type: application/json -d '{
"BUILD_NUMBER":"83","BUILD_ID":"83","JOB_NAME":"OMS_LOG_ANA","JOB_STATUS":"SUCCESS" ,
"time" : "2019-09-08'T'12:39:00" }'