Related
I have a production application in which I am using a nodejs client to query the last point of a measurement every second. A separate client is writing a point to the measurement on a 1 second interval as well. After a few minutes, I occasionally start receiving null values for random fields. Some fields have data and others are null.
If I re-query for that exact same point a second later, all the values are there. I have also outputted the data to a csv file, and all the data is there as well.
I have recreated the issue on a standalone example. The issue occurs no matter if I use the npm influx library or make the http requests myself. The more frequent the requests are sent, the more frequent I get points containing null values. With the below example, if I run several instances of this app simultaneously, then many points will contain null values.
Is it possible that I am reading the point before influx has written all field values?
Any help is appreciated.
I'm using influx v 1.8.2 and have recreated the issue on ubuntu and windows.
Code to recreate:
const { InfluxDB } = require("influx");
const influx = new InfluxDB({
host: "localhost",
database: "testDb",
});
const createDb = async () => {
await influx.createDatabase("testDb");
};
const read = async () => {
const res = await influx.query(`
select * from livedata
ORDER BY time desc
limit 1
`);
console.log(res[0]);
};
const write = async () => {
await influx.writeMeasurement("livedata", [
{
tags: {
id: "site",
},
fields: {
site1: Math.random() * 1000,
site2: Math.random() * 1000,
site3: Math.random() * 1000,
site4: Math.random() * 1000,
site5: Math.random() * 1000,
site6: Math.random() * 1000,
site7: Math.random() * 1000,
site8: Math.random() * 1000,
},
},
]);
};
createDb();
setInterval(() => {
write();
read();
}, 300);
Example output:
[
'2021-01-07T21:40:11.4031559Z',
'site',
830.4042230769617,
522.1830877694142,
698.8789904008146,
678.305459618109,
118.82269436309988,
631.6295948279627,
376.3112870744887,
830.4872612882643
]
]
[
[
'2021-01-07T21:40:11.7034901Z',
'site',
null,
null,
null,
65.3968316403697,
680.7946463560837,
330.7338852317838,
872.7936919556367,
145.03057994702618
]
]
[
[
'2021-01-07T21:40:12.0036893Z',
'site',
901.031149970251,
501.1825877093237,
99.38758592260699,
78.79549874505165,
403.8558500935323,
545.085784401504,
969.637642068842,
51.657735620841194
]
]
I'm trying to create a custom world map where countries are merged into regions instead of having individual countries. Unfortunately for some reason something seems to get messed up with the winding order along the process.
As base data I'm using the natural earth 10m_admin_0_countries shape files available here. As criteria for merging countries I have a lookup map that looks like this:
const countryGroups = {
"EUR": ["ALA", "AUT", "BEL"...],
"AFR": ["AGO", "BDI", "BEN"...],
...
}
To merge the shapes I'm using topojson-client. Since I want to have a higher level of control than the CLI commands offer, I wrote a script. It goes through the lookup map and picks out all the topojson features that belong to a group and merges them into one shape and places the resulting merged features into a geojson frame:
const topojsonClient = require("topojson-client");
const topojsonServer = require("topojson-server");
const worldTopo = topojsonServer.topology({
countries: JSON.parse(fs.readFileSync("./world.geojson", "utf-8")),
});
const geoJson = {
type: "FeatureCollection",
features: Object.entries(countryGroups).map(([region, ids]) => {
const relevantCountries = worldTopo.objects.countries.geometries.filter(
(country, i) =>
ids.indexOf(country.properties.ISO_A3) >= 0
);
return {
type: "Feature",
properties: { region, countries: ids },
geometry: topojsonClient.merge(worldTopo, relevantCountries),
};
}),
};
So far everything works well (allegedly). When I try to visualise the map using github gist (or any other visualisation tool like vega lite) the shapes seem to be all messed up. I'm suspecting that I'm doing something wrong during the merging of the features but I can't figure out what it is.
When I try to do the same using the CLI it seems to work fine. But since I need more control over the merging, using just the CLI is not really an option.
The last feature, called "World", should contain all remaining countries, but instead, it contains all countries, period. You can see this in the following showcase.
var w = 900,
h = 300;
var projection = d3.geoMercator().translate([w / 2, h / 2]).scale(100);
var path = d3.geoPath().projection(projection);
var color = d3.scaleOrdinal(d3.schemeCategory10);
var svg = d3.select('svg')
.attr('width', w)
.attr('height', h);
var url = "https://gist.githubusercontent.com/Flave/832ebba5726aeca3518b1356d9d726cb/raw/5957dca433cbf50fe4dea0c3fa94bb4f91c754b7/world-regions-wrong.topojson";
d3.json(url)
.then(data => {
var geojson = topojson.feature(data, data.objects.regions);
geojson.features.forEach(f => {
console.log(f.properties.region, f.properties.countries);
});
svg.selectAll('path')
// Reverse because it's the last feature that is the problem
.data(geojson.features.reverse())
.enter()
.append('path')
.attr('d', path)
.attr('fill', d => color(d.properties.region))
.attr('stroke', d => color(d.properties.region))
.on('mouseenter', function() {
d3.select(this).style('fill-opacity', 1);
})
.on('mouseleave', function() {
d3.select(this).style('fill-opacity', null);
});
});
path {
fill-opacity: 0.3;
stroke-width: 2px;
stroke-opacity: 0.4;
}
<script src="https://cdnjs.cloudflare.com/ajax/libs/d3/5.7.0/d3.js"></script>
<script src="https://d3js.org/topojson.v3.js"></script>
<svg></svg>
To fix this, I'd make sure to always remove all assigned countries from the list. From your data, I can't see where "World" is defined, and if it contains all countries on earth, or if it's a wildcard assignment.
In any case, you should be able to fix it by removing all matches from worldTopo:
const topojsonClient = require("topojson-client");
const topojsonServer = require("topojson-server");
const worldTopo = topojsonServer.topology({
countries: JSON.parse(fs.readFileSync("./world.geojson", "utf-8")),
});
const geoJson = {
type: "FeatureCollection",
features: Object.entries(countryGroups).map(([region, ids]) => {
const relevantCountries = worldTopo.objects.countries.geometries.filter(
(country, i) =>
ids.indexOf(country.properties.ISO_A3) >= 0
);
relevantCountries.forEach(c => {
const index = worldTopo.indexOf(c);
if (index === -1) throw Error(`Expected to find country ${c.properties.ISO_A3} in worldTopo`);
worldTopo.splice(index, 1);
});
return {
type: "Feature",
properties: { region, countries: ids },
geometry: topojsonClient.merge(worldTopo, relevantCountries),
};
}),
};
I'm trying to change header titles by passing an array of titles to options but it does not override the headers. Instead it inserts new headers before the original data. I am passing the same numbers of header titles.
Here is my code:
const ws: XLSX.WorkSheet = XLSX.utils.json_to_sheet(
json,
{header: headerColumns}
);
const wb: XLSX.WorkBook = XLSX.utils.book_new();
XLSX.utils.book_append_sheet(wb, ws, 'Transactions');
const excelBuffer: any = XLSX.write(wb, { bookType: 'xlsx', type: 'array' });
this.saveAsExcelFile(excelBuffer, excelFileName);
And output looks like below:
The basic job of the "header" option is not to override, rather just shift the starting option of the columns.
i.e. any value passed in the header option will be treated as the first column, provided the value should match with existing keys you have in the data.
XLSX.utils.json_to_sheet([{A:1,B:2}, {B:2,C:3}], {header:['C']});
Here column "C" will be the first column in the excel.
For more look out for detailed description here: https://docs.sheetjs.com/#sheetjs-js-xlsx
This is how I have achieved similar behavior:
const XLSX = require('xlsx');
const wb = XLSX.utils.book_new();
const Heading = [
['Sr No', 'User Name', 'Department', 'Bank', 'Country', 'Region', 'Amount']
];
// creating sheet and adding data from 2nd row of column A.
// leaving first row to add Heading
const ws = XLSX.utils.json_to_sheet(data, { origin: 'A2', skipHeader: true });
// adding heading to the first row of the created sheet.
// sheet already have contents from above statement.
XLSX.utils.sheet_add_aoa(ws, Heading, { origin: 'A1' });
// appending sheet with a name
XLSX.utils.book_append_sheet(wb, ws, 'Records');
const fileContent = XLSX.write(wb, { bookType: 'xlsx', type: 'buffer' });
Very traditional approach but working, please see complete code below:
const worksheet: XLSX.WorkSheet = XLSX.utils.json_to_sheet(
this.releaseDateWiseCountList
);
worksheet.A1.v = "Pick Release Date";
worksheet.B1.v = "Task Type";
worksheet.C1.v = "First Shift";
worksheet.D1.v = "Second Shift";
worksheet.E1.v = "Total";
worksheet.F1.v = "Grand Total";
worksheet.G1.v = "Pick %";
const workbook: XLSX.WorkBook = {
Sheets: { 'data': worksheet }, SheetNames: ['data']
};
const excelBuffer: any = XLSX.write(
workbook, { bookType: 'xlsx', type: 'array' }
);
const data: Blob = new Blob([buffer], {type: EXCEL_TYPE});
FileSaver.saveAs(data, 'Result_export_' + new Date().getTime() + EXCEL_EXTENSION);
Using data from database I am trying to simulate the sankey diagram working JSFiddle.
I am assembling my data using the below code
// sdata.php
<?php
$con = sqlsrv_connect($server, $options);
if (!$con){die('Could not connect: ' . sqlsrv_error());}
$sql_query = "select * from test_data";
$result = sqlsrv_query($con, $sql_query);
$series = array();
$series['type'] = 'sankey';
$series['name'] = 'Gendata';
$series['keys'] = '[\'from\',\'to\',\'weight\']';
while($r = sqlsrv_fetch_array($result))
{
$series1 = array();
$series1[] = $r['PARENT'];
$series1[] = $r['CHILD'];
$series1[] = $r['DGEN'];
$series['data'][] = $series1;
}
$result = array();
array_push($result,$series);
print json_encode($result, JSON_NUMERIC_CHECK);
sqlsrv_close($con);
?>
My JSON looks like
[{
"type":"sankey",
"name":"Gendata",
"keys":"['from','to','weight']",
"data":[
["GROUP","COAL",24.46], ["GROUP","GAS",11.96],["GROUP","HYDRO",19.36],
["HYDRO","HYD",19.36], ["COAL","ER2",22.4],["GAS","NR",19]
]
}]
My Chart rending code looks like
$(document).ready(function() {
var options = {
chart: {
renderTo: 'container',
showAxes: true
},
yAxis: [{
lineWidth: 1,
tickPositions: [0, 1, 2, 3]
}],
title: {
text: 'Sankey Diagram'
},
series: []
}
$.getJSON("sdata.php", function(resp) {
console.log(resp);
options.series[0] = resp[1]; //option 1 to assign the data in series
//options.series.push.resp; //option 2 to push the data in series
chart = new Highcharts.Chart(options);
});
});
but I am failing. I am unable to find the error I am missing
Kindly help me.
Let me know if I can be of any further information.
From the code you have posted here, the error is your keys assignment.
You have:
"keys":"['from','to','weight']",
But it needs to be:
"keys": ['from','to','weight'],
That is, the array should not be surrounded by quotes, because then it will be interpreted as a string.
In your PHP that would mean:
$series['keys'] = '[\'from\',\'to\',\'weight\']';
Needs to be:
$series['keys'] = ['from', 'to', 'weight'];
Working example: https://jsfiddle.net/ewolden/aeh02djx/
So in a current app, I have to use a custom Formatter on a couple rows in my jqGrid. All these do is take a few fields from my ajax call, concat them into one, and place that into a row.
EG ( data.toStreet + data.toCity + data.toState + data.toZip ) comes back as "Street City, State Zip" into the "To Address" column. This works fine and the data displays correctly, but when using the filtering toolbar, the filter is only based on the first val (data.street). below is a super simplified version of the pieces of code in question.
$('#grid').jqGrid({
...
colNames:["AddressTo", "AddressFrom"],
colModel:[
{name:"toStreet" formatter: ToAddressFormatter},
{name:"fromStreet" formatter: FromAddressFormatter}
],
...
}),
$('#grid').jqGrid('filterToolbar',
{
stringResult:true,
searchOnenter: true,
defaultSearch: 'cn'
}
});
ToAddressFormatter = function(el, opt, rowObj){
var address = rowObj.toStreet+ " " + rowObj.toCity + ", " + rowObj.toState + " " + rowObj.toZip;
return address;
},
FromAddressFormatter = function(el, opt, rowObj){
var address = rowObj.fromStreet+ " " + rowObj.fromCity + ", " + rowObj.fromState + " " + rowObj.fromZip;
return address;
}
So if the value in the cel says "123 fake st, springfield, Va 22344" after being formatted, the filter toolbar can only search on "123 fake st" and nothing else.
Does anybody have any clue on how to remedy this, or possibly why it's happening and a good workaround??
EDIT:
I have included the beginning of my grid. Also, the property Address of result.d is created in the code below, and not returned from the webservice. My column is mapped to "Address" which displays the formatting properly, but still does not search as intended.
$('#grdDisasters').jqGrid({
datatype: function(postdata) {
var obj = { "showActive": $('#btnFilterActive.pressed').length > 0 ? true : false, "showInactive": $('#btnFilterActive.pressed').length > 0 ? true : false,
'page': postdata.page, 'rows': postdata.rows, 'sortIndex': postdata.sidx, 'sortDirection': postdata.sord, 'search': postdata._search,
'filters': postdata.filters || ''
};
$.ajax({
url: "/GetGrid",
data: JSON.stringify(obj),
success: function(result) {
for (var i = 0, il = result.d.rows.length; i < il; i++) {
LoadedDisasters[i] = result.d.rows[i];
result.d.rows[i].cells.Address = result.d.rows[i].cells.Street + " " + result.d.rows[i].cells.City + ", "+ result.d.rows[i].cells.State+ " "+ result.d.rows[i].cells.Zip;
}
result.d = NET.format(result.d);//just correctly format dates
UpdateJQGridData($('#grdDisasters'), result.d);
},
error: function(result) {
//alert("Test failed");
}
});
jqGrid has a problem filtering rows when data is formatted using custom/predefined formatter.
You will have to filter rows on the server-side.
Add 2 more request parameter in your controller to handle jqgrid search request:
When jqGrid requests for filtered raws it will add a parameter: _search with value: true
and all the search parameter like col1=abc&col4=123 meaning user wanted to filter using column named col1 and column named col4 with values respectively: abc and 123
Use those values and query the database with like operation something as follows:
select id, concat(street1, street2, city, state, zip) as address
where address like "%abc%" and id like "%123%"
return these rows as json to jqGrid and display those in the current page. So basically you will have to have a jqGrid with server-side paging, sorting and searching. You can not use client-side paging, sorting and searching features. Also, make sure you don't have loadonce: true set.
I think that you fill the grid in the wrong way. If your source data has toStreet, toCity, toState, toZip, fromStreet, fromCity, fromState, fromZip properties and you need to have composed addressTo and addressFrom you should do this in another way. Your problem is that toStreet and fromStreet will be saved locally in the internal data parameter in the original format like you get it from the server. The local searching uses the data parameter, so the toStreet and fromStreet like you get there from the server will be used.
You don't posted more full code of jqGrid which you use. So I suppose that you use datatype: 'json', datatype: 'jsonp' or datatype: 'xml' in combination with loadonce: true. You should define colModel
$('#grid').jqGrid({
...
colNames:["AddressTo", "AddressFrom"],
colModel:[
{name: "addressTo", ...},
{name: "addressFrom", ...}
],
beforeProcessing: function (data) {
var i, rows = data.rows, l = rows.length, item;
for (i = 0; i < l; i++) {
item = rows[i];
item.addressTo = item.toStreet + " " + item.toCity + ", " +
item.toState + " " + item.toZip;
item.addressFrom = item.fromStreet+ " " + item.fromCity + ", " +
item.fromState + " " + item.fromZip;
}
}
...
});
The exact code depend on the format of the input data. The advantage of the usage of beforeProcessing is that it will be called before the data will be processed by jqGrid. So you can do any modification in the data or like in the above.
UPDATED: The code of datatype can be easy implemented in another way using standard jqGrid options. So I suggest to use the following settings:
datatype: "json",
url: "/GetGrid",
postData: {
// add and to the list of parameters sent to the web service
showActive: function () {
return $('#btnFilterActive.pressed').length > 0;
},
showInactive: function () {
return $('#btnFilterActive.pressed').length > 0;
}
},
prmNames: {
// rename some parameters sent to the web service
sort: "sortIndex",
order: "sortDirection",
search: "search",
// don't send nd parameter to the server
nd: null
// you leave the nd is you don't set any "Cache-Control" HTTP header
// I would recommend you to set "Cache-Control: private, max-age=0"
// For example
// HttpContext.Current.Response.Cache.SetMaxAge (new TimeSpan(0));
},
serializeGridData: function (postData) {
// control modification of the the data (parameters) which will be sent
// to the web method
if (typeof postData.filters === "undefined") {
postData.filters = "";
}
return JSON.stringify(postData);
},
ajaxGridOptions: { contentType: "application/json" },
jsonReader: {
root: "d.rows",
page: function (obj) { return obj.d.page; },
total: function (obj) { return obj.d.total; },
records: function (obj) { return obj.d.rows.length; },
repeatitems: false
},
loadError: function (jqXHR, textStatus, errorThrown) {
// see an implementation example in the answers
// https://stackoverflow.com/a/6969114/315935
// and
// https://stackoverflow.com/a/5501644/315935
},
colNames:["AddressTo", "AddressFrom"],
colModel:[
{name: "addressTo", ...},
{name: "addressFrom", ...}
],
beforeProcessing: function (data) {
var i, rows, l, item;
data.d = NET.format(data.d); // just correctly format dates
rows = data.d.rows;
l = rows.length;
for (i = 0; i < l; i++) {
item = rows[i];
LoadedDisasters[i] = item;
item.addressTo = item.toStreet + " " + item.toCity + ", " +
item.toState + " " + item.toZip;
item.addressFrom = item.fromStreet+ " " + item.fromCity + ", " +
item.fromState + " " + item.fromZip;
}
}
...
The usage of nd: null with setting of "Cache-Control: private, max-age=0" I described in the answer. You can download the corresponding demo project which use this. In general one needs just include one additional line where you call SetMaxAge
[WebMethod]
[ScriptMethod(UseHttpGet = true, ResponseFormat = ResponseFormat.Json)]
public MyGridData GetGrid(...) {
HttpContext.Current.Response.Cache.SetMaxAge (new TimeSpan(0));
...
}
See more about caching control you can read here.