Importing data to influx DB with DDL/DML & setting retention policy - influxdb

I have data in a text file as below...
DML
CONTEXT-DATABASE: import
CONTEXT-RETENTION-POLICY: oneyear
FO,type=FreshOrders avg_FreshOrders=1479.0840707964599,p95_FreshOrders=2589,FreshOrders=226 1593648000000
FO,type=FreshOrders avg_FreshOrders=2242.86170212766,p95_FreshOrders=5622,FreshOrders=188 1593475200000
FO,type=FreshOrders avg_FreshOrders=1682.3375,p95_FreshOrders=2738,FreshOrders=160 1593734400000
I placed this text file in a folder called data inside influxDB. My requirement is to import last 3 lines of data into influx DB. I am navigating into influxDB folder C:\Users\Administrator\Downloads\influxdb-1.8.0_windows_amd64\influxdb-1.8.0-1 and running the below command in command prompt to import it
influx -import -path=C:\Users\Administrator\Downloads\influxdb-1.8.0_windows_amd64\influxdb-1.8.0-1\data\import.txt -precision=ns
But data is not getting imported and getting below error. Can someone please help me in resolving this issue?
C:\Users\Administrator\Downloads\influxdb-1.8.0_windows_amd64\influxdb-1.8.0-1>influx -import -path=C:\Users\Administrator\Downloads\influxdb-1.8.0_windows_amd64\influxdb-1.8.0-1\data\import.txt -precision=s
2020/07/13 16:34:25 error: error parsing query: found DML, expected SELECT, DELETE, SHOW, CREATE, DROP, EXPLAIN, GRANT, REVOKE, ALTER, SET, KILL at line 1, char 1
2020/07/13 16:34:25 error: error parsing query: found CONTEXT, expected SELECT, DELETE, SHOW, CREATE, DROP, EXPLAIN, GRANT, REVOKE, ALTER, SET, KILL at line 1, char 1
2020/07/13 16:34:25 error: error parsing query: found CONTEXT, expected SELECT, DELETE, SHOW, CREATE, DROP, EXPLAIN, GRANT, REVOKE, ALTER, SET, KILL at line 1, char 1
2020/07/13 16:34:25 error: error parsing query: found FO, expected SELECT, DELETE, SHOW, CREATE, DROP, EXPLAIN, GRANT, REVOKE, ALTER, SET, KILL at line 1, char 1
2020/07/13 16:34:25 error: error parsing query: found FO, expected SELECT, DELETE, SHOW, CREATE, DROP, EXPLAIN, GRANT, REVOKE, ALTER, SET, KILL at line 1, char 1
2020/07/13 16:34:25 error: error parsing query: found FO, expected SELECT, DELETE, SHOW, CREATE, DROP, EXPLAIN, GRANT, REVOKE, ALTER, SET, KILL at line 1, char 1


Change the content of your file as below:
# DML
# CONTEXT-DATABASE: import
# CONTEXT-RETENTION-POLICY: oneyear
FO,type=FreshOrders avg_FreshOrders=1479.0840707964599,p95_FreshOrders=2589,FreshOrders=226 1593648000000
FO,type=FreshOrders avg_FreshOrders=2242.86170212766,p95_FreshOrders=5622,FreshOrders=188 1593475200000
FO,type=FreshOrders avg_FreshOrders=1682.3375,p95_FreshOrders=2738,FreshOrders=160 1593734400000

Related

How to upload Polygons from GeoPandas to Snowflake?

I have a geometry column of a geodataframe populated with polygons and I need to upload these to Snowflake.
I have been exporting the geometry column of the geodataframe to file and have tried both CSV and GeoJSON formats, but so far I either always get an error the staging table always winds up empty.
Here's my code:
design_gdf['geometry'].to_csv('polygons.csv', index=False, header=False, sep='|', compression=None)
import sqlalchemy
from sqlalchemy import create_engine
from snowflake.sqlalchemy import URL
engine = create_engine(
URL(<Snowflake Credentials Here>)
)
with engine.connect() as con:
con.execute("PUT file://<path to polygons.csv> #~ AUTO_COMPRESS=FALSE")
Then on Snowflake I run
create or replace table DB.SCHEMA.DESIGN_POLYGONS_STAGING (geometry GEOGRAPHY);
copy into DB.SCHEMA."DESIGN_POLYGONS_STAGING"
from #~/polygons.csv
FILE_FORMAT = (TYPE = CSV FIELD_DELIMITER = '|' SKIP_HEADER = 1 compression = None encoding = 'iso-8859-1');
Generates the following error:
"Number of columns in file (6) does not match that of the corresponding table (1), use file format option error_on_column_count_mismatch=false to ignore this error File '#~/polygons.csv.gz', line 3, character 1 Row 1 starts at line 2, column "DESIGN_POLYGONS_STAGING"[6] If you would like to continue loading when an error is encountered, use other values such as 'SKIP_FILE' or 'CONTINUE' for the ON_ERROR option. For more information on loading options, please run 'info loading_data' in a SQL client."
Can anyone identify what I'm doing wrong?

Inspired by #Simeon_Pilgrim's comment I went back to Snowflake's documentation. There I found an example of converting a string literal to a GEOGRAPHY.
https://docs.snowflake.com/en/sql-reference/functions/to_geography.html#examples
select to_geography('POINT(-122.35 37.55)');
My polygons looked like strings describing Polygons more than actual GEOGRAPHYs so I decided I needed to be treating them as strings and then calling TO_GEOGRAPHY() on them.
I quickly discovered that they needed to be explicitly enclosed in single quotes and copied into a VARCHAR column in the staging table. This was accomplished by modifying the CSV export code:
import csv
design_gdf['geometry'].to_csv(<path to polygons.csv>,
index=False, header=False, sep='|', compression=None, quoting=csv.QUOTE_ALL, quotechar="'")
The staging table now looks like:
create or replace table DB.SCHEMA."DESIGN_POLYGONS_STAGING" (geometry VARCHAR);
I ran into further problems copying into the staging table related to the presence of a polygons.csv.gz file I must have uploaded in a previous experiment. I deleted this file using:
remove #~/polygons.csv.gz
Finally, converting the staging table to GEOGRAPHY
create or replace table DB.SCHEMA."DESIGN_GEOGRAPHY_STAGING" (geometry GEOGRAPHY);
insert into DB.SCHEMA."DESIGN_GEOGRAPHY"
select to_geography(geometry)
from DB.SCHEMA."DESIGN_POLYGONS_STAGING"
and I wound up with a DESIGN_GEOGRAPHY table with a single column of GEOGRAPHYs in it. Success!!!

Neo4Js client error apoc.load.csv failed procedure. caused by ArrayIndexOutofBoundException 1

I am newbie to NEO4J. I am using neo4j version 3.5.6 community edition and apoc plugins version 3.5.0.4 .I have a CSV file in default import folder
NR_Nodes_Agent_I_20190331_tmp.csv. For testing purpose I have written a cypher query
CALL apoc.load.csv('NR_Nodes_Agent_I_20190331_tmp.csv') yield map as row return row;
but I am getting bellow error
Neo.ClientError.Procedure.ProcedureCallFailed: Failed to invoke procedure apoc.load.csv: Caused by: java.lang.ArrayIndexOutOfBoundsException: 1
I did some research on it but haven't successful to solve
I uncommented statement in neo4j.conf file
dbms.directories.import=import
dbms.security.allow_csv_import_from_file_urls=true
dbms.security.procedures.whitelist=apoc.coll.*,apoc.load.*,apoc.*

Make sure you have this setting in your neo4j.conf file:
apoc.import.file.enabled=true
Make sure your CSV file is well-formed.
For example, this CSV file would cause the same ArrayIndexOutOfBoundsException: 1 error message (notice that the single data row is missing a second value, since it has one fewer comma than the header):
a,b
1
On the other hand, this CSV file would work, even though the data row has no value after the comma):
a,b
1,
The query result would be:
╒════════════════╕
│"row" │
╞════════════════╡
│{"a":"1","b":""}│
└────────────────┘
And if the data row had a second value, like this:
a,b
1,2
Then the query result would be:
╒═════════════════╕
│"row" │
╞═════════════════╡
│{"a":"1","b":"2"}│
└─────────────────┘

Importing a DB file in Neo4J

I am trying to install the reactome DB into Neo4j so I can make a graph from it. I keep getting the same error message regardless of my syntax. The DB folder is stored in neo4J\bin\reactome.
I have basically been using this cypher command and all sorts of permutations of it:
neo4j-admin restore --from=neo4j\bin\reactome --database= reactome.graphdb –force=true"
and, regardless of how I do it, I get this error leading me to think it is something more than the syntax:
Neo.ClientError.Statement.SyntaxError: Invalid input 'e': expected <init> (line 1, column 1 (offset: 0))
"eo4j-admin restore --from=neo4j\bin\reactome --database= reactome.graphdb –force=true""
^

The neo4j-admin tool must be run from the command line -- it is NOT a Cypher operation.

To install the Reactome DB with Neo4j, click on 'open folder/import' in Neo4j. Then go up one directory, and go to '/data/databases/graph.db/'. Dump all the contents of the Reactome graph DB folder (uncompressed) inside the directory.

Error when Executing GraphQL query in Neo4J

I've quite a bit of exerience with Neo4J but a noob with graphql
I created my graphql schema by running:
CALL graphql.idl(null)
I have a node type with three labels. I tried to run the following query in graphiql and got the same error. Due to the stack, I wondered if graphiql was adding meta and moved to the neo4j browser - same error.
The query:
CALL graphql.execute('mutation { createArrival(uuid:"graphql")}')
The error:
Failed to invoke procedure `graphql.execute`: Caused by: java.lang.RuntimeException: Error executing GraphQL Query:
ExceptionWhileDataFetching{path=[createArrival]exception=org.neo4j.graphdb.QueryExecutionException: Invalid input 'n': expected whitespace, comment, '{', node labels, MapLiteral, a parameter, a relationship pattern, '(', '.', '=' or "+=" (line 1, column 69 (offset: 68))
"CREATE (node:Arrival) SET node = {properties} SET node:`Event`, SET node:`Configuration`"
^locations=[SourceLocation{line=1, column=12}]}
I'm probably doing something really obviously wrong but any help would be appreciated

Confirmed as a bug by #michael-hunger

dask dataframe to_parquet throws error

I am trying to save task dataframe to parquet on HDFS. However it fails with error :Exception: TypeError('expected list of bytes',)
I am also providing object_encoding argument as {"anomaly":"json","sensor_name":"json"}.
Here is the columns in dataframe: Index(['original_value', 'anomaly', 'anomaly_bin', 'sensor_name'], dtype='object')
Columns sensor_name and anomaly are string. Other columns are float.
eg: [18.0 'N' 0.0 'settemp']
I also tried to save it as CSV in HDFS but the api failed with error: Exception: ValueError('url type not understood:
Path to CSV as: hdfs://ip:port/some path
It will be great if some one can guide me in right direction.

Resources