How do I convert data from Laravel DB connection to GeoJSON - geojson

Currently making a project in Laravel that uses MapboxGLJS. I've currently got a database server that I'm connected to that contains comments which I need to convert to a GeoJSON FeatureCollection that includes the ID and the spacial data. I've seen an example of the code to do this which I'll provide below but when I try to use said code and try to use the addSource Mapbox method it comes back with Error: Input data is not a valid GeoJSON object..
CommentController.php
...
public function all(){
$comments = Comment::whereNotNull('user_id')->get();
$mapFeatures = array();
$mapFeatures['type'] = 'FeatureCollection';
$mapFeatures['name'] = 'comments';
$mapFeatures['crs'] = array(
'type' => 'name',
'properties' => array(
'name' => 'urn:ogc:def:crs:OGC:1.3:CRS84'
),
);
$mapFeatures['features'] = array();
foreach ($comments as $comment) {
$mapItem = array(
'type' => 'Feature',
'properties' => array(
'id' => $comment->id,
),
'geometry' => $comment->location
);
array_push($mapFeatures['features'], $mapItem);
}
return json_encode($mapFeatures);
}
...
Using Postman I collected the following from the api request:
{
"type": "FeatureCollection",
"name": "comments",
"crs": {
"type": "name",
"properties": {
"name": "urn:ogc:def:crs:OGC:1.3:CRS84"
}
},
"features": [
{
"type": "Feature",
"properties": {
"id": 143
},
"geometry": "0101000020E6100000E17A14AE47E111C085EB51B81E054A40"
},
...
]
}
Running the data through https://geojsonlint.com/ it comes back with Line 1: old-style crs member is not recommended, this object is equivalent to the default and should be removed. Also stating that geometry was expected as an object but got a string which I assume is to do with the crs property not decoding the geometry correctly.
Is there a different crs that I need in order to get the geometry to be correctly decoded?
I unfortunately cannot change the data on the database to include a lat/long geometry as the current data is being used by another project which relies on it being in this format.

This:
"0101000020E6100000E17A14AE47E111C085EB51B81E054A40"
is not a GeoJSON geometry. I'm not sure exactly what it is. It looks like PostGIS's native format (see here) but I don't know what that is called or how to convert from it outside PostGIS.
A GeoJSON geometry would look like:
{
"type": "LineString",
"coordinates": [[...]]
}
If you have access to the PostGIS queries, you should use the ST_AsGeoJSON function.
This has nothing to do with CRS's - the message is just telling you not to bother adding that crs property, assuming that your data is in EPSG:4326.

Discovered there is a package specifically to fix this issue available here:
https://github.com/mstaack/laravel-postgis
This just needs to be installed and referenced in the Controller that gets the coordinates.

Related

Twitter API 2.0 - Unable to fetch user.fields

I am using API version 2.0 and unable to fetch the user.fields results. All other parameters seem to be returning results correctly. I'm following this documentation.
url = "https://api.twitter.com/2/tweets/search/all"
query_params = {
"query": "APPL",
"max_results": "10",
"tweet.fields": "created_at,lang,text,author_id",
"user.fields": "name,username,created_at,location",
"expansions": "referenced_tweets.id.author_id",
}
response = requests.request("GET", url, headers=headers, params=query_params).json()
Sample result:
{
'author_id': '1251347502013521925',
'text': 'All conspiracy. But watch for bad news on Apple. Such a vulnerable stocktechnically for the biggest market cap # $2.1T ( Thanks Jay). This is the glue for the bulls. But, they stopped innovating when Steve died, built a fancy office and split the stock. $appl',
'lang': 'en',
'created_at': '2021-06-05T02:33:48.000Z',
'id': '1401004298738311168',
'referenced_tweets': [{
'type': 'retweeted',
'id': '1401004298738311168'
}]
}
As you can see, the following information is not returned: name, username, and location.
Any idea how to retrieve this info?
Your query does actually return the correct data. I tested this myself.
A full example response will be structured like this:
{
"data": [
{
"created_at": "2021-06-05T02:33:48.000Z",
"lang": "en",
"id": "1401004298738311168",
"text": "All conspiracy. But watch for bad news on Apple. Such a vulnerable stocktechnically for the biggest market cap # $2.1T ( Thanks Jay). This is the glue for the bulls. But, they stopped innovating when Steve died, built a fancy office and split the stock. $appl",
"author_id": "1251347502013521925",
"referenced_tweets": [
{
"type": "retweeted",
"id": "1401004298738311168"
}
]
}
],
"includes": {
"users": [
{
"name": "Gary Casper",
"id": "1251347502013521925",
"username": "Hisel1979",
"created_at": "2020-07-11T13:39:58.000Z"
}
]
}
}
The sample result you provided comes from within the data object. However, the expanded object data will be nested in the includes object (in your case name, username, and location). The corresponding user object can be referenced via the author_id field.

Zapier - add data to JSON response (App development)

We are creating a Zapier app to expose our APIs to the public, so anyone can use it. The main endpoint that people are using returns a very large and complex JSON object. Zapier, it looks like, has a really difficult time parsing nested complex JSON. But it does wonderful with a very simple response object such as
{ "field": "value" }
Our data that is being returned has this structure and we want to move some of the fields to the root of the response so it's easily parsed by Zapier.
"networkSections": [
{
"identifier": "Deductible",
"label": "Deductible",
"inNetworkParameters": [
{
"key": "Annual",
"value": " 600.00",
"message": null,
"otherInfo": null
},
{
"key": "Remaining",
"value": " 600.00",
"message": null,
"otherInfo": null
}
],
"outNetworkParameters": null
},
So, can we do something to return for example the remaining deductible?
I got this far (adding outputFields) but this returns an array of values. I'm not sure how to parse through this array either in the Zap or in the App.
{key: 'networkSections[]inNetworkParameters[]key', label: 'xNetworkSectionsKey',type: 'string'},
ie this returns an array of "Annual", "Remaining", etc
Great question. In this case, there's a lot going on, and outputFields can't quite handle it all. :(
In your example, inNetworkParameters contains an array of objects. Throughout our documentation, we refer to these as line items. These lines items can be passed to other actions, but the different expected structures presents a bit of a problem. The way we've handled this is by letting users map line-items from one step's output to another step's input per field. So if step 1 returns
{
"some_array": [
{
"some_key": "some_value"
}
]
}
and the next step needs to send
{
"data": [
{
"some_other_key": "some_value"
}
]
}
users can accomplish that by mapping some_array.some_key to data.some_other_key.
All of that being said, if you want to always return a Remaining Deductible object, you'll have to do it by modifying the result object itself. As long as this data is always in that same order, you can do something akin to
var data = z.JSON.parse(bundle.response.content);
data["Remaining Deductible"] = data.networkSections[0].inNetworkParameters[1].value;
return data;
If the order differs, you'll have to implement some sort of search to find the objects you'd like to return.
I hope that all helps!
Caleb got me where I wanted to go. For completeness this is the solution.
In the creates directory I have a js file for the actual call. The perform part is below.
perform: (z, bundle) => {
const promise = z.request({
url: 'https://api.example.com/API/Example/' + bundle.inputData.elgRequestID,
method: 'GET',
headers: {
'content-type': 'application/json',
}
});
return promise.then(function(result) {
var data = JSON.parse(result.content);
for (var i=0; i<data.networkSections.length; i++) {
for (var j=0; j<data.networkSections[i].inNetworkParameters.length; j++) {
// DEDUCT
if (data.networkSections[i].identifier == "Deductible" &&
data.networkSections[i].inNetworkParameters[j].key == "Annual")
data["zAnnual Deductible"] = data.networkSections[i].inNetworkParameters[j].value;
} // inner for
} // outer for
return data;
});

Drag and drop a GeoJSON with linked CRS

I have a GeoJSON file that looks like this:
{
"type": "FeatureCollection",
"crs": {
"type": "link",
"properties": {
"href": "http://spatialreference.org/ref/epsg/32198/proj4/",
"type": "proj4"
}
},
"features": [
{
"type": "Feature",
"geometry": {
"type": "Point",
"coordinates": [200000, 20000]
},
"properties": {
"id": 1,
"name": "foo"
}
}
]
}
As you can see, the crs definition uses the link type, which is documented here: http://geojson.org/geojson-spec.html#linked-crs
I'm dropping the file in an OL3 map that has the ol.interaction.DragDrop interaction enabled, but it fails to load it. OpenLayers 3 doesn't currently support this type of crs definition, thus the reason it fails to load it. It supports those of type name and EPSG, see: https://github.com/openlayers/ol3/blob/master/src/ol/format/geojsonformat.js#L484 (snippet below):
if (crs.type == 'name') {
return ol.proj.get(crs.properties.name);
} else if (crs.type == 'EPSG') {
// 'EPSG' is not part of the GeoJSON specification, but is generated by
// GeoServer.
// TODO: remove this when http://jira.codehaus.org/browse/GEOS-5996
// is fixed and widely deployed.
return ol.proj.get('EPSG:' + crs.properties.code);
} else {
goog.asserts.fail('Unknown crs.type: ' + crs.type);
return null;
}
Looking at it, I don't know if it would be possible to natively support the link type inside OpenLayers directly, as it would require to do an asynchronous request to fetch the projection definition within code that's synchronous. I suspect that I'm stuck with this problem.
I'm looking for an alternative to approach the problem, or maybe I'm just wrong about the fact that it could be possible to support this (with a proper patch) natively in OL3.
Any hint ?
Alexandre, your best bet is to avoid using the old GeoJSON linked CRS (which is very poorly supported by software) and either 1) convert your data to GeoJSON's default WGS84 long/lat – this is the best option by far or 2) use a CRS name like "urn:ogc:def:crs:EPSG::32198".

How to make elasticsearch add the timestamp field to every document in all indices?

Elasticsearch experts,
I have been unable to find a simple way to just tell ElasticSearch to insert the _timestamp field for all the documents that are added in all the indices (and all document types).
I see an example for specific types:
http://www.elasticsearch.org/guide/reference/mapping/timestamp-field/
and also see an example for all indices for a specific type (using _all):
http://www.elasticsearch.org/guide/reference/api/admin-indices-put-mapping/
but I am unable to find any documentation on adding it by default for all documents that get added irrespective of the index and type.
Elasticsearch used to support automatically adding timestamps to documents being indexed, but deprecated this feature in 2.0.0
From the version 5.5 documentation:
The _timestamp and _ttl fields were deprecated and are now removed. As a replacement for _timestamp, you should populate a regular date field with the current timestamp on application side.
You can do this by providing it when creating your index.
$curl -XPOST localhost:9200/test -d '{
"settings" : {
"number_of_shards" : 1
},
"mappings" : {
"_default_":{
"_timestamp" : {
"enabled" : true,
"store" : true
}
}
}
}'
That will then automatically create a _timestamp for all stuff that you put in the index.
Then after indexing something when requesting the _timestamp field it will be returned.
Adding another way to get indexing timestamp. Hope this may help someone.
Ingest pipeline can be used to add timestamp when document is indexed. Here, is a sample example:
PUT _ingest/pipeline/indexed_at
{
"description": "Adds indexed_at timestamp to documents",
"processors": [
{
"set": {
"field": "_source.indexed_at",
"value": "{{_ingest.timestamp}}"
}
}
]
}
Earlier, elastic search was using named-pipelines because of which 'pipeline' param needs to be specified in the elastic search endpoint which is used to write/index documents. (Ref: link) This was bit troublesome as you would need to make changes in endpoints on application side.
With Elastic search version >= 6.5, you can now specify a default pipeline for an index using index.default_pipeline settings. (Refer link for details)
Here is the to set default pipeline:
PUT ms-test/_settings
{
"index.default_pipeline": "indexed_at"
}
I haven't tried out yet, as didn't upgraded to ES 6.5, but above command should work.
You can make use of default index pipelines, leverage the script processor, and thus emulate the auto_now_add functionality you may know from Django and DEFAULT GETDATE() from SQL.
The process of adding a default yyyy-MM-dd HH:mm:ss date goes like this:
1. Create the pipeline and specify which indices it'll be allowed to run on:
PUT _ingest/pipeline/auto_now_add
{
"description": "Assigns the current date if not yet present and if the index name is whitelisted",
"processors": [
{
"script": {
"source": """
// skip if not whitelisted
if (![ "myindex",
"logs-index",
"..."
].contains(ctx['_index'])) { return; }
// don't overwrite if present
if (ctx['created_at'] != null) { return; }
ctx['created_at'] = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss").format(new Date());
"""
}
}
]
}
Side note: the ingest processor's Painless script context is documented here.
2. Update the default_pipeline setting in all of your indices:
PUT _all/_settings
{
"index": {
"default_pipeline": "auto_now_add"
}
}
Side note: you can restrict the target indices using the multi-target syntax:
PUT myindex,logs-2021-*/_settings?allow_no_indices=true
{
"index": {
"default_pipeline": "auto_now_add"
}
}
3. Ingest a document to one of the configured indices:
PUT myindex/_doc/1
{
"abc": "def"
}
4. Verify that the date string has been added:
GET myindex/_search
An example for ElasticSearch 6.6.2 in Python 3:
from elasticsearch import Elasticsearch
es = Elasticsearch(hosts=["localhost"])
timestamp_pipeline_setting = {
"description": "insert timestamp field for all documents",
"processors": [
{
"set": {
"field": "ingest_timestamp",
"value": "{{_ingest.timestamp}}"
}
}
]
}
es.ingest.put_pipeline("timestamp_pipeline", timestamp_pipeline_setting)
conf = {
"settings": {
"number_of_shards": 2,
"number_of_replicas": 1,
"default_pipeline": "timestamp_pipeline"
},
"mappings": {
"articles":{
"dynamic": "false",
"_source" : {"enabled" : "true" },
"properties": {
"title": {
"type": "text",
},
"content": {
"type": "text",
},
}
}
}
}
response = es.indices.create(
index="articles_index",
body=conf,
ignore=400 # ignore 400 already exists code
)
print ('\nresponse:', response)
doc = {
'title': 'automatically adding a timestamp to documents',
'content': 'prior to version 5 of Elasticsearch, documents had a metadata field called _timestamp. When enabled, this _timestamp was automatically added to every document. It would tell you the exact time a document had been indexed.',
}
res = es.index(index="articles_index", doc_type="articles", id=100001, body=doc)
print(res)
res = es.get(index="articles_index", doc_type="articles", id=100001)
print(res)
About ES 7.x, the example should work after removing the doc_type related parameters as it's not supported any more.
first create index and properties of the index , such as field and datatype and then insert the data using the rest API.
below is the way to create index with the field properties.execute the following in kibana console
`PUT /vfq-jenkins
{
"mappings": {
"properties": {
"BUILD_NUMBER": { "type" : "double"},
"BUILD_ID" : { "type" : "double" },
"JOB_NAME" : { "type" : "text" },
"JOB_STATUS" : { "type" : "keyword" },
"time" : { "type" : "date" }
}}}`
the next step is to insert the data into that index:
curl -u elastic:changeme -X POST http://elasticsearch:9200/vfq-jenkins/_doc/?pretty
-H Content-Type: application/json -d '{
"BUILD_NUMBER":"83","BUILD_ID":"83","JOB_NAME":"OMS_LOG_ANA","JOB_STATUS":"SUCCESS" ,
"time" : "2019-09-08'T'12:39:00" }'

Breeze delete failing, json has mysterious "undefined": false

I am making a website that follows John Papa's Code Camper SPA Jumpstart Pluralsight course. My database/entities has the following hierarchy:
Proficiency contains Action(s) and Level(s).
When I delete a "Proficiency", I get the following server side error:
"Object reference not set to an instance of an object."
Looking at the json JObject saveBundle in the BreezeController, I am seeing a mysterious:
"undefined": false,
in every entity. None of my entities have a Boolean in them. Just like in Code Camper, I am adding an "isPartial" in the constructor of each entity as shown in the code below.
var proficiencyConstructor = function () {
this.isPartial = false;
}
metadataStore.registerEntityTypeCtor('Proficiency', proficiencyConstructor, proficiencyInitializer);
function proficiencyInitializer(proficiency) {
var empty = "00000000-0000-0000-0000-000000000000";
if (proficiency.id() === empty) {
proficiency.id(breeze.core.getUuid());
}
};
My gut says the mysterious "undefined":false is the "isPartial" property. According to the documentation, the "Breeze adds the isPartial property to the Entity metadata as an unmapped property. The values of unmapped properties are not transmitted to the service." I am stuck. Anyone recommend things I can do to figure this out?
Thanks,
Dan
Here is a sample from the savebundle
{ "entities":
[
{ "Id": "a0223d7c-35e5-458f-ba83-65ec7ec189fa", "Name": "AST Prof0", "IsEnabled": true, "Description": "AST Prof0", "ProficiencyType": "TBD", "ApplicationId": "7ba4b47f-06a3-4ceb-bca6-de3fd3699bbd", "undefined": false, "entityAspect": { "entityTypeName": "Proficiency:#LobGame.Model", "entityState": "Deleted", "originalValuesMap": { "IsPartial": true }, "autoGeneratedKey": null } },
This is likely due to a bug fix that is now in breeze 1.2.8. This fixed it for me.
From their release notes:
Bug fix for the case where a save involving a delete would fail when
that save also involved a modification to an unmapped property.
http://www.breezejs.com/documentation/download

Resources