How to get data in InfluxDB subscription end point? - time-series

Hi I have created a subscription (http) in the InfluxDB using following command :
CREATE SUBSCRIPTION "test-data-update" ON "system_managment"."system_managment2d" DESTINATIONS ALL 'http://localhost:9001/ts/data-update'
And in the ExpressJs app which is running on port 9001 added a route as follows:
app.post("/ts/data-update/write", (req, res) => {
console.log(">>influx data>>");
console.log(req.body);
res.sendStatus(200);
});
I thought I will be getting all the data in the req.body but I am getting nothing in body. How can I get the data through a subscription as mentioned in the influxdb documentation.
I have seen more examples which is using Kapacitor, but as of now we are not planning to us Kapacitor.
Let me know how to get the updated data from InfluxDB subscription endpoint? Or is it supposed to be like this and we have make normal querying to InfluxDB from this endpoint to get data?

Had same issue and the solution is to listen to data event using request object.
app.post("/ts/data-update/write",(req, res) => {
logger.info('influx data>>');
logger.info(req.headers);
logger.info(req.body);
req.on('data', (chunk) => {
console.log(chunk.toString());
});
req.on('end', () => {
res.sendStatus(200);
});
});
Saw header with content length with positive non zero value and did not have value in the request body using express.
Then I listened using netcat on specified port in subscription which hosts a TCP server and had the content at the end.

Related

How do I get my physical device running React Native on Expo Go to communicate with my app's rails backend api?

I am currently trying to run my react-native/rails app on my phone for testing purposes. I can run my sign in and log in screens fine because they do not communicate with my server until the user info is entered. When running my server i use:
$ rails s --binding=0.0.0.0
I do not receive any errors other than knowing my server is not being communicated with. This all works fine on my Android Studio Emulator as well.
// one of my fetch GET requests
export function requestCurrentUser(username, auth_token) {
return function action(dispatch) {
const request = fetch(`'http://10.0.2.2:3000'/users/${username}`, {
method: 'GET',
headers: {
"Authorization": auth_token
}
});
return request.then(
response => response.json(),
err => console.log("user error"),
)
.then(
json => dispatch(receiveCurrentUser({id: json.id, email: json.email, username: json.username})),
err => console.log("user json error"),
);
}
}
I've tried changing my Phone IP settings to a 10.0.2.2 Gateway, and using my phone's IP in my fetch request. I feel like I am missing something conceptually. Thanks in advance.
In fetch request you need to use the IP from the machine that are running the rails server,
probably your notebook and use the same network to connect your app and your rails backend api. In order to test, you can try directly access your api in your phone browser accessing http://IP_FROM_RAILS_MACHINE:3000

How does https://pubsubhubbub.appspot.com/ actually works and what kind of response am I expected to receive?

Good day everyone,
For context purpose, I am trying to monitor a Youtube Channel where, whenever they post a new video, I will get a notification and process the entry.
What I have done:-
a) setup the callback url to receive and reply the hub challenge.
b) https://pubsubhubbub.appspot.com/subscribe
Subscribe to this site as suggested by Youtube where the topic URL is
https://www.youtube.com/feeds/videos.xml?channel_id=UClrEYreVkBee7ZQYei_6Jqg (atom-feed)
and the callback URL is to a Google Cloud functions url.
c) Reply and verified the hub challenge sent by (b).
The questions I have are:
I havent receive any response from pubsubhubbub since the subscription but the Youtube channel already has updated videos since then. Have I missed some steps here or is this not the expectation to have?
I expect to only receive a notification and not the XML feed body from pubsubhubbub and upon notification, I should then process and fetch the XML feed from the channel.
OR
I expect to receive the XML feed body / latest entry from pubsubhubbub.
The reason I am asking this is I see there is a latest entry and not the full feed in the pubsubhubbub site.
Thanks in advance if there are anybody who could help answer these questions.
import express from 'express';
import dotenv from "dotenv";
dotenv.config();
const YouTubeNotifier = require('youtube-notification');
const app = express();
const port = 6050;
const baseUrl = "https://evil-wasp-10.loca.lt"
let channelId = process.env.CHANNEL_ID;
export const notifier = new YouTubeNotifier({
hubCallback: `${baseUrl}/youtube/notifications`,
});
app.use("/youtube/notifications", notifier.listener());
app.listen(port, () => {
console.log(`App listening at http://localhost:${port}`)
})
notifier.subscribe(channelId);
notifier.on('subscribe', (data: any) => {
console.log('Subscribed');
console.log(data);
});
notifier.on('notified', (data: any) => {
console.log('New Video');
console.log(data);
});

Is it possible to send blob from client to server via Actioncable for Rails?

I have searched everywhere for this and haven't come across anyone else with this issue: I am unable to send a blob over a web socket using Actioncable. For whatever reason the channel always receives that data as an empty object. Here is the related code.
A blob:
var blob = new Blob(['something'], {type: 'text/plain'});
Active cable subscription js:
App.messageChannel = App.cable.subscriptions.create(
{
channel: "MessageChannel",
id: some_id
},
{
received: function(data) {
return console.log("The data is " + data);
},
},
);
App.messageChannel.send(blob);
I try to send that from the client to the server and it does send via websocket successfully, but it just sends back data: "{}". It doesn't actually send the blob data. Interestingly if I inspect the network web socket requests, it shows an empty object as being sent (so I don't believe the server is receiving the blob, even though it's being created in the script).
Any ideas? Does ActionCable only support JSON data or am I missing some other obvious element?

How to retrieve Medium stories for a user from the API?

I'm trying to integrate Medium blogging into an app by showing some cards with posts images and links to the original Medium publication.
From Medium API docs I can see how to retrieve publications and create posts, but it doesn't mention retrieving posts. Is retrieving posts/stories for a user currently possible using the Medium's API?
The API is write-only and is not intended to retrieve posts (Medium staff told me)
You can simply use the RSS feed as such:
https://medium.com/feed/#your_profile
You can simply get the RSS feed via GET, then if you need it in JSON format just use a NPM module like rss-to-json and you're good to go.
Edit:
It is possible to make a request to the following URL and you will get the response. Unfortunately, the response is in RSS format which would require some parsing to JSON if needed.
https://medium.com/feed/#yourhandle
⚠️ The following approach is not applicable anymore as it is behind Cloudflare's DDoS protection.
If you planning to get it from the Client-side using JavaScript or jQuery or Angular, etc. then you need to build an API gateway or web service that serves your feed. In the case of PHP, RoR, or any server-side that should not be the case.
You can get it directly in JSON format as given beneath:
https://medium.com/#yourhandle/latest?format=json
In my case, I made a simple web service in the express app and host it over Heroku. React App hits the API exposed over Heroku and gets the data.
const MEDIUM_URL = "https://medium.com/#yourhandle/latest?format=json";
router.get("/posts", (req, res, next) => {
request.get(MEDIUM_URL, (err, apiRes, body) => {
if (!err && apiRes.statusCode === 200) {
let i = body.indexOf("{");
const data = body.substr(i);
res.send(data);
} else {
res.sendStatus(500).json(err);
}
});
});
Nowadays this URL:
https://medium.com/#username/latest?format=json
sits behind Cloudflare's DDoS protection service so instead of consistently being served your feed in JSON format, you will usually receive instead an HTML which is suppose to render a website to complete a reCAPTCHA and leaving you with no data from an API request.
And the following:
https://medium.com/feed/#username
has a limit of the latest 10 posts.
I'd suggest this free Cloudflare Worker that I made for this purpose. It works as a facade so you don't have to worry about neither how the posts are obtained from source, reCAPTCHAs or pagination.
Full article about it.
Live example. To fetch the following items add the query param ?next= with the value of the JSON field next which the API provides.
const MdFetch = async (name) => {
const res = await fetch(
`https://api.rss2json.com/v1/api.json?rss_url=https://medium.com/feed/${name}`
);
return await res.json();
};
const data = await MdFetch('#chawki726');
To get your posts as JSON objects
you can replace your user name instead of #USERNAME.
https://api.rss2json.com/v1/api.json?rss_url=https://medium.com/feed/#USERNAME
With that REST method you would do this: GET https://api.medium.com/v1/users/{{userId}}/publications and this would return the title, image, and the item's URL.
Further details: https://github.com/Medium/medium-api-docs#32-publications .
You can also add "?format=json" to the end of any URL on Medium and get useful data back.
Use this url, this url will give json format of posts
Replace studytact with your feed name
https://api.rss2json.com/v1/api.json?rss_url=https://medium.com/feed/studytact
I have built a basic function using AWS Lambda and AWS API Gateway if anyone is interested. A detailed explanation is found on this blog post here and the repository for the the Lambda function built with Node.js is found here on Github. Hopefully someone here finds it useful.
(Updating the JS Fiddle and the Clay function that explains it as we updated the function syntax to be cleaner)
I wrapped the Github package #mark-fasel was mentioning below into a Clay microservice that enables you to do exactly this:
Simplified Return Format: https://www.clay.run/services/nicoslepicos/medium-get-user-posts-new/code
I put together a little fiddle, since a user was asking how to use the endpoint in HTML to get the titles for their last 3 posts:
https://jsfiddle.net/h405m3ma/3/
You can call the API as:
curl -i -H "Content-Type: application/json" -X POST -d '{"username":"nicolaerusan"}' https://clay.run/services/nicoslepicos/medium-get-users-posts-simple
You can also use it easily in your node code using the clay-client npm package and just write:
Clay.run('nicoslepicos/medium-get-user-posts-new', {"profile":"profileValue"})
.then((result) => {
// Do what you want with returned result
console.log(result);
})
.catch((error) => {
console.log(error);
});
Hope that's helpful!
Check this One you will get all info about your own post........
mediumController.getBlogs = (req, res) => {
parser('https://medium.com/feed/#profileName', function (err, rss) {
if (err) {
console.log(err);
}
var stories = [];
for (var i = rss.length - 1; i >= 0; i--) {
var new_story = {};
new_story.title = rss[i].title;
new_story.description = rss[i].description;
new_story.date = rss[i].date;
new_story.link = rss[i].link;
new_story.author = rss[i].author;
new_story.comments = rss[i].comments;
stories.push(new_story);
}
console.log('stories:');
console.dir(stories);
res.json(200, {
Data: stories
})
});
}
I have created a custom REST API to retrieve the stats of a given post on Medium, all you need is to send a GET request to my custom API and you will retrieve the stats as a Json abject as follows:
Request :
curl https://endpoint/api/stats?story_url=THE_URL_OF_THE_MEDIUM_STORY
Response:
{
"claps": 78,
"comments": 1
}
The API responds within a reasonable response time (< 2 sec), you can find more about it in the following Medium article.

How to connect multiple socket.io clients to different URLs in Node.js?

I am trying to connect many socket.io clients for different URLs in Node.js like so :
app.get('/:id',function(req,res){
io.of('/'+id).on('connection',function(socket){
socket.emit('hello');
})
});
This works however there is a problem :
When a browser refreshs the page http://localhost:3000/xyz for example, the event socket.emit gets fired two times.
If someone accesses the page http://localhost:3000/xyz 10 times, then the event fires 10 times.
This is not good because everytime the user visits that page, the socket events will be fired n+1 times.
What should be done so that I can register sockets to different URLs and at the same time not have this anomaly .
Another thing :
If I do this :
var sock;
io.of('/'+xyz).on('connection',function(socket){
sock=socket;
})
app.get('/:id',function(req,res){
sock.emit('hello');
})
If I use the above code then the socket doesn't get saved succesfully to the sock variable in time. What that means is , I have to do a setInterval of about 1000 .. so that the
sock=socket
line gets fired.
Please help me.
Because with this, in each request to http://localhost:3000/id, you register a new handler, you should be doing that once, not at every request.
app.get('/:id',function(req,res){
io.of('/'+id).on('connection',function(socket){
socket.emit('hello');
})
});
I use below approach to achieve this goal:
client side:
var socket = io.connect('http://localhost:8183/?clientId='+clientId,{"force new connection":true});
server side:
var io = require('socket.io').listen(server);
io.sockets.on('connection', function(socket) {
console.log("url"+socket.handshake.url);
clientId=socket.handshake.query.clientId;
console.log("connected clientId:"+clientId);
});
reference:https://github.com/LearnBoost/socket.io/wiki/Authorizing#global-authorization

Resources