I have a file with data in a .log file. how can i get inserted into the database like the NOAA_water_database they suggest in the documentation.
the link to the doc is below
https://docs.influxdata.com/influxdb/v1.7/query_language/data_download/
CLI command influx has an option called -import to import data to influx. Refer more details here.
In the same document you mentioned in question there is a command to import data to influx
influx -import -path=NOAA_data.txt -precision=s -database=NOAA_water_database
Inserting the data from a log file to influx DB is possible if the lines of a log file are in line protocol.
Please refer below URL :
https://community.influxdata.com/t/writing-logs-manually-into-influxdb/6247/4
Related
I'm going to put the csv file into the bucket using influxdb v2.1.
Attempting to insert a simple example file results in the following error:
error in csv.from(): failed to read metadata: failed to read annotations: expected annotation datatype
The csv file that I was going to write is as follows.
#datatype measurement,tag,double,dateTime:RFC3339
m,host,used_percent,time
mem,host1,64.23,2020-01-01T00:00:00Z
mem,host2,72.01,2020-01-01T00:00:00Z
mem,host1,62.61,2020-01-01T00:00:10Z
mem,host2,72.98,2020-01-01T00:00:10Z
mem,host1,63.40,2020-01-01T00:00:20Z
mem,host2,73.77,2020-01-01T00:00:20Z
This is the example data in the official document of influxdata.
If you look at the first line of the example, you can see that datatype is annotated, but why does the error occur?
How should I modify it?
This looks like invalid annotated CVS.
In the csv.from function documentation, you can find examples (as string literals) of both annotated and raw CVS that the cvs.from supports.
I have created 15.4 GB of csv files that I would like to import into fresh new Neo4j graph.db.
After executing neo4j-admin import --delimiter="|" --array-delimiter="&" --nodes="processes.*" command (I have 17229 csv files, that are named "processes_someHash.csv") I get this particular output:
..../pathWithCsvFiles: neo4j-admin import --delimiter="|" --array-delimiter="&" --nodes="processes.*"
WARNING: Max 1024 open files allowed, minimum of 40000 recommended. See the Neo4j manual.
For input string: "10059167292802359779483"
usage: neo4j-admin import [--mode=csv] [--database=<name>]
[--additional-config=<config-file-path>]
[--report-file=<filename>]
[--nodes[:Label1:Label2]=<"file1,file2,...">]
[--relationships[:RELATIONSHIP_TYPE]=<"file1,file2,...">]
[--id-type=<STRING|INTEGER|ACTUAL>]
[--input-encoding=<character-set>]
[--ignore-extra-columns[=<true|false>]]
[--ignore-duplicate-nodes[=<true|false>]]
[--ignore-missing-nodes[=<true|false>]]
[--multiline-fields[=<true|false>]]
[--delimiter=<delimiter-character>]
[--array-delimiter=<array-delimiter-character>]
[--quote=<quotation-character>]
[--max-memory=<max-memory-that-importer-can-use>]
[--f=<File containing all arguments to this import>]
[--high-io=<true/false>]
usage: neo4j-admin import --mode=database [--database=<name>]
[--additional-config=<config-file-path>]
[--from=<source-directory>]
environment variables:
NEO4J_CONF Path to directory which contains neo4j.conf.
NEO4J_DEBUG Set to anything to enable debug output.
NEO4J_HOME Neo4j home directory.
HEAP_SIZE Set JVM maximum heap size during command execution.
Takes a number and a unit, for example 512m.
Import a collection of CSV files with --mode=csv (default), or a database from a
pre-3.0 installation with --mode=database.
options:
--database=<name>
Name of database. [default:graph.db]
--additional-config=<config-file-path>
Configuration file to supply additional configuration in. [default:]
--mode=<database|csv>
Import a collection of CSV files or a pre-3.0 installation. [default:csv]
--from=<source-directory>
The location of the pre-3.0 database (e.g. <neo4j-root>/data/graph.db).
[default:]
--report-file=<filename>
File in which to store the report of the csv-i
... and more of a manual
What does the For input string: "10059167292802359779483" mean?
Have you checked the headers in your CSV files? That's been a problem for me when importing previously.
Any chance your delimiter character is also appearing in data values?
well, I tested neo4j importing with more compact dataset and it worked fine (when there was problem with delimiter for example, then the error message showed me what was the specific problem). I optimized my program for creating these csv files based on this low dataset and used it to make the mentioned bigger csv files, which doesn't work.
I try to import CSV in a Neo4j Database and I have a problem.
On my desktop computer (windows 7, java 1.8.0_40-b25), the LOAD CSV works great.
But on the server (windows 2012 R2, java 1.8.0_65-b17), i have this error message "URI is not hierarchical".
I try to put the data on C:, F: ... no change.
Here's the code :
USING PERIODIC COMMIT 100
LOAD CSV WITH HEADERS FROM
"file:F:/Neo4JData/Destination.csv"
AS line
MERGE (d:Destination {`Code`: line.`Code`});
Thanks for your help.
Are you using 2.3.0 Community Edition?
try:
USING PERIODIC COMMIT 10000 LOAD CSV FROM
'file:///F:\\Neo4JData\\Destination.csv
Create an import folder in the default path of the DB and place the file there that helped me.
For example: C:\Users\XXXXY\Documents\Neo4j\default.graphdb\import and put the csv there. In the query use
USING PERIODIC COMMIT
LOAD CSV WITH HEADERS FROM "file:///customers.csv" AS row
CREATE (:Customer {companyName: row.CompanyName, customerID: row.CustomerID, fax: row.Fax, phone: row.Phone});
I had the same problem. I solved it by putting /// instead of F:/ or F:///.
So if your source is
F:/FolderOne/FolderTwo/file.csv
It becomes
///FolderOne/FolderTwo/file.csv
Remember that in order to add the file you must put file: in front of the source.
So finally
file:///FolderOne/FolderTwo/file.csv
As specified above once you try with
USING PERIODIC COMMIT
LOAD CSV WITH HEADERS FROM "file:///E:/AdventureWorks/adventureworks-neo4j/data/products.csv" as row
CREATE (:Product {productName: row.ProductName, productNumber: row.ProductNumber, productId: row.ProductID, modelName: row.ProductModelName, standardCost: row.StandardCost, listPrice: row.ListPrice});
The "URI is not hierarchical" error disappears. Then most probably you will get an error saying that it couldnt load the resource like
TransientError.Statement.ExternalResourceFailure
In order to solve the same you should find the neo4j.conf file.
Since i'm using a windows 10 machine and community edition of neo4j i could find the same in the below path.
C:\Users\{username}\AppData\Roaming\Neo4j Community Edition
Edit the conf file and comment out the line
dbms.directories.import=import
Doing the above steps enabled me to load the csv file.
The file path seems wrong, can you try with :
"file:F:///Neo4JData/Destination.csv"
Try with:
file:///F:/Neo4JData/Destination.csv
Putting the .csv into the $NEO4JHOME/default.graphdb/import directory worked. You may have to create the folder, or maybe uncommenting the dbms.directories.import=import line in conf might do it. I dunno, did it the hard way :)
However, I found that I still had to include the drive specifier, i.e., file:///c:\csv2import.csv even if it wasn't in the root but in the import directorry
Use the path 'file:///F:/Neo4JData/Destination.csv' and add the Destination.csv file to the neo4jDB\import directory.If the import directory is not there create a new directory named as import and add the file.
Create a folder import under default.graphdb and place your csv files there. Later, you can use file:///fileName.csv in your LOAD CSV query
I was getting this problem as well so what I did was to copy the folder in which the .csv file was present to the following location:
C:\Users\Username\AppData\Roaming\Neo4j Desktop\Application\neo4jDatabases\database-27badd10-8989-482d-871b-cad746091f07\installation-3.3.3\import\
it seems when neo4j is installed and we try to import data from files by giving "file:///c://........" then neo4j starts looking for that file in the location C:\Users\Username\AppData\Roaming\Neo4j Desktop\Application\neo4jDatabases\database-27badd10-8989-482d-871b-cad746091f07\installation-3.3.3\import\
i.e the import folder. So we have to copy all the files needed to be imported by this this type of statement in that import folder.
This one worked at my end in windows
LOAD CSV FROM 'File:///order-details.csv' AS row
RETURN count(row);
How to get source filename as it is from source into HDFS in flume?
Ex: source file /usr/sample.txt hdfs: /tmp/sample.txt not like flumeevetns.23343.tmp
how to stop appending timestamp and .tmp?Ex:flumeevent.12334343.tmp(Here 12334343.tmp) I dont want it.
How to read as a whole file from Flume?
How to read csv file in Flume?
You need to add a parameter for spooldir which adds a header which is false by
default.
agentname.sources.sourcename.fileHeader=true
It will keep the same name of file and push into HDFS.
I am trying to use Flume with hdfs as sink. File is getting exported. But I want to customize the name of the output file. I am using hdfs.filePrefix property for it. It is always creating a file with FlumeData.timestamp.
Please paste your configuration.
I tried it and it did work.
My setting:
agent.sinks.flumeHDFS.hdfs.filePrefix = stackoverflow
and I get the expected result.