How can I import geojson values in a mysql column using load data infile? for example I have a file as follows (simplified for brevity):
{"type":"FeatureCollection", "features": [
{"type":"Feature","geometry":{"type":"Polygon","coordinates":
[
[
[31.287890625000017,-22.40205078125001],
[31.429492187500017,-22.298828125],
[32.37109375,-21.33486328125001]
]
] },"properties":{"NAME":"Zimbabwe"}
},
{"type":"Feature","geometry":{"type":"Polygon","coordinates":
[
[
[30.39609375,-15.64306640625],
[30.3505859375,-15.349707031250006],
[30.231835937500023,-14.990332031250006]
]
]},"properties":{"NAME":"Zambia"}
}
]
}
Currently when I do the following:
LOAD DATA LOCAL INFILE 'C:/Users/Downloads/countries.geojson' INTO TABLE countries (geo_json);
I get error:
Invalid JSON text: "Invalid value." at position 2668 in value for column 'countries.geo_json'
How can I load each feature into a table which has a json column called geo_json. Then I want to extract the name for each feature and add to a name column?
Related
I got the following .geojson file from Overpass API:
{
"type": "FeatureCollection",
"generator": "overpass-ide",
"copyright": "The data included in this document is from www.openstreetmap.org. The data is made available under ODbL.",
"timestamp": "2022-07-18T07:57:39Z",
"features": [
{
"type": "Feature"
...
I converted this simply with gdf = geopandas.read_file(filename) into a GeoDataFrame object. This all worked great.
The GeoDataFrame object now contains all elements that were listed inside of "features" within the .geojson.
But now I wondered if there is any way for me to access the value for "timestamp" on this GeoDataFrame object? Or does the GeoDataFrame object not even contain this key/value-pair anymore?
gdf["timestamp"] does not work.
I could use the json-library like shown below, but then I would have to load the entire file again which seems like a huge waste of resources.
import json
f = open('filename.geojson', encoding="utf8")
data = json.load(f)
timestamp = data["timestamp"]
Any help would be much appreciated.
i need to export a collection that have a field "Data" like :
"Data" : BinData(0,"AAAAAAAAxFBQKFgBBvodksodfosdqgDw...")
when i export with mongoexport in json output file the field became :
"Data":
{"$binary":
{"base64":"AAAAAAAAxFBQKFgBBvodksodfosdqgDw...","subType":"00"}
}
is there a way to force
}
is there a way to force to create .json in the mongoexport command to generate the output tag of "Data" like this ?: BinData(0,"AAAAAAAAxFBQKFgBBvodksodfosdqgDw...")
I have Multiple folder and files which from FileSystem (linked service) on Azure Data Factory. and my activity is references on link: https://www.sqlservercentral.com/articles/working-with-get-metadata-activity-in-azure-data-factory
for now I'm using process metadata FileName and LastModified per file like this:
and then i'm using stored-procedure on ADF like this:
ALTER PROCEDURE [dbo].[SP_FileSystemMonitoring]
(
-- Add the parameters for the stored procedure here
#FLAG int,
#FILE_NAME nvarchar(100),
#LAST_MODIFIED datetime
)
AS
BEGIN
-- SET NOCOUNT ON added to prevent extra result sets from
-- interfering with SELECT statements.
SET NOCOUNT ON
-- Insert statements for procedure here
IF ( #FILE_NAME IS NOT NULL )
BEGIN
UPDATE [dwh].[FileSystemMonitoring]
SET STATUS = #FLAG,
PROCESS_DATE = DATEADD(HH, 7, Getdate()),
REPORT_DATE = DATEADD(hh,7,(DATEADD(dd,-1,GETDATE()))),
LAST_MODIFIED = #LAST_MODIFIED
WHERE FILE_NAME = #FILE_NAME
but, I want on 1 activity can get metadata on 1 folder and then then file that folder insert to Azure SQL Database, for example
folderA/file1.txt
folderA/file2.txt
On that Azure SQL Table like this:
--------------------------
File_Name | Last_Modified
--------------------------
file1.txt | 2021-12-19 13:45:56
file2.txt | 2021-12-18 10:23:32
I have no idea, because I'm confuse how to mapping on that sink on Azure SQL Table. Thanks before ...
Confused by your question, is it that you want to get the details of the file or folder from the get metadata activity? Or do you want to enumerate/store the child items of a root folder?
If you simply want to reference the items from Get Metadata, add a dynamic expression that navigates the output value to the JSON property you seek. For example:
#activity('Get Metadata Activity Name').output.lastModified
#activity('Get Metadata Activity Name').output.itemName
You can pass each of the above expressions as values to your stored procedure parameters. NOTE: 'Get Metadata Activity Name' should be renamed to the name of your activity.
The output JSON of this activity is like so and will grow depending on what you select to return in the Get Metadata activity. In my example I'm also including childItems.
{
"exists": true,
"lastModified": "2021-03-04T14:00:01Z",
"itemName": "some-container-name",
"itemType": "Folder",
"childItems": [{
"name": "someFilePrefix_1640264640062_24_12_2021_1640264640.csv",
"type": "File"
}, {
"name": "someFilePrefix_1640286000083_24_12_2021_1640286000.csv",
"type": "File"
}
],
"effectiveIntegrationRuntime": "DefaultIntegrationRuntime (Australia Southeast)",
"executionDuration": 0,
"durationInQueue": {
"integrationRuntimeQueue": 0
},
"billingReference": {
"activityType": "PipelineActivity",
"billableDuration": [{
"meterType": "AzureIR",
"duration": 0.016666666666666666,
"unit": "Hours"
}
]
}
}
if you want to store the child files, then you can either parse childItems as an nvarchar JSON value into your stored procedure and then enumerate the JSON array in SQL.
You could also use ADF an enumerate the same childItems property using a ForEach activity for each file. You simply enumerate over:
#activity('Get Metadata Activity Name').output.childItems
You can then call the SP for each file referencing the nested item as:
#item().name
You'll also still be able to reference any of the root parameters from the original get metadata activity within the ForEach activity.
I'm applying number formatting to a PDF document, which works fine in a 'parent' record in the JSON data, but fails when used inside a table. How can I get the $formatNumber expression to work inside an array of data?
The data is being merged into an invoice document, and in one of the columns of the "Invoice Line Items" I'd like to format the number with the expression $formatNumber, '#,###.00'.
Here is the merge syntax in my word document:
This formatting works fine for the parent data:
{{ $formatNumber(QuoteLineItems.totalSize, '#,###.00') }}
This formatting fails when used inside a table (the output in the PDF si "NaN")
{{ $formatNumber(QuoteLineItems.records.AmountNet, '#,###.00') }}
Image: Comparison of Success vs Failure syntax
Image: PDF Output contains NaN in table
here is the sample input data I am using inside the word plugin document tagger:
{
"QuoteLineItems":
{
"records":
[
{
"AmountGross": 47897,
"AmountNet": 44556,
"Name": "Sample Charges"
},
{
"AmountGross": 4968,
"AmountNet": 4621,
"Name": "Sample 2 Charges"
}
],
"done": true,
"totalSize": 2
},
"Name": "Sample Invoice"
}
Sorry for the delay in answering this. This is a known bug (I'm going to check on the status). For now I suggest doing your number format in your code before sending it to the API. If that doesn't make sense, let me know and I'll clarify.
A CSV containing the following rows is imported into mongodb via CSV using the mongoimport tool:
object_1_id,field1,field2
52db7f026956b996a0000001,apples,oranges
52db7f026956b996a0000001,pears,plums
The fields are imported into the collection Object2.
After import the rows are confirmed to exist via the console.
#<Object2 _id: 52e0713417bcabcb4d09ad12, _type: nil, field1: "apples", field2: "oranges", object_1_id: "52db7f026956b996a0000001">
#<Object2 _id: 52e0713517bcabcb4d09ad76, _type: nil, field1: "pears", field2: "plums", object_1_id: "52db7f026956b996a0000001">
Object2 can access Object1 via object_1_id:
> o = Object2.first
#<Object2 _id: 52e0713417bcabcb4d09ad12, _type: nil, field1: "apples", field2: "oranges", object_1_id: "52db7f026956b996a0000001">
> o1 = o.object_1
#<Object1 _id: "52db7f026956b996a0000001", other_fields: "text and stuff">
But Object1 cannot see any of the Object2 rows that were imported with mongoimport. It can see all rows that have been created via the console or other means:
> o1.object_2s.count
10
> o1.object_2s.find("52e0713417bcabcb4d09ad12")
Mongoid::Errors::DocumentNotFound:
Document not found for class Object2 with id(s) 52e0713417bcabcb4d09ad12.
TL;DR Object1 doesn't appear to recognise child models imported via mongoimport, despite the child correctly storing the parent ID and being able to identify its parent.
As per mu is too short's comment the ids were being imported as Strings instead of BSON ObjectIds.
mongoexport and mongoimport (I was only using the latter) only support strings and numbers (See: https://stackoverflow.com/a/15763908/943833).
In order to import data with type from a CSV you have to use Extended JSON dumps as explained in the above link.
Quick and dirty solution:
1) Export the collection you want to import as JSON using mongoexport:
mongoexport -d database -c collection -o output.json
2) Grab the first line of the export file. It should look something like this:
{ "_id" : { "$oid" : "52dfe0106956b9ee6e0016d8" }, "column2" : "oranges", "column1" : "apples", "object_1_id" : { "$oid" : "52dfe0106956b9ee6e0016d8" }, "updated_at" : { "$date" : 1390403600994 }, "created_at" : { "$date" : 1390403600994 } }
3) The _id field as well as any other fields you don't want to import.
4) Use your language of choice to generate a JSON file using the JSON snippet as a template for each line.
5) Import the new JSON file using mongoimport:
mongoimport -d database -c collection --type json --file modified.json
This will preserve types better than CSV is capable. I'm not sure whether it is as reliable as using mongodump and mongorestore but they aren't an option for me since my CSV file comes from elsewhere.