I have a PlaywrightCrawler to scrape Alibaba. But when I add a request to one page like:
https://www.alibaba.com/product-detail/Mono-filament-12-mm-PP-fiber_1600139352513.html?spm=a27aq.industry_category_productlist.dt_3.1.3d733642TkHgZc
This page lasted to loading until timeout and handlePageFunction didn't be called.
Actually, all the content has been loaded completed. I notice that some AJAX runs in the background.
How do I force PlaywrightCrawler to call handlePageFunction even though AJAX didn't complete it?
const crawler = new Apify.PlaywrightCrawler({
requestQueue,
launchContext: {
launchOptions: {
headless: false,
},
},
handlePageFunction,
});
you can change your waitUntil parameter to go to the page as soon as the DOM loads using this:
const crawler = new Apify.PlaywrightCrawler({
requestQueue,
// ...
preNavigationHooks: [async (context, gotoOptions) => {
gotoOptions.waitUntil = 'domcontentloaded';
}],
});
this will fire as soon as the page is ready to be queried by document.querySelectorAll, you may have to wait for certain conditions inside the handlePageFunction before starting to call page methods
Related
In my App I have two windows: mainWindow and actionWindow. On my mainWindow I use the ipcRenderer.on listener to receive as message from the main process when the actionWindow is closed. The message however doesn't come through.
The mainWindow is used to control actions that take place on the actionWindow (e.g. navigate to an URL, remotely close the window, ...). I want to give the user the power to move and close the actionWindow manually as well, which is why its title bar is visible and usable.
I expose ipcRenderer.invoke for two-way communication and ipcRenderer.on to the mainWindow's renderer via contextBridge in a preload file.
This is what the code looks like (based on vite-electron-builder template)
main process
const mainWindow = new BrowserWindow({
show: false, // Use 'ready-to-show' event to show window
webPreferences: {
nativeWindowOpen: true,
webviewTag: false,
preload: join(__dirname, "../../preload/dist/index.cjs"),
},
});
const actionWindow = new BrowserWindow({
// some props
})
actionWindow.on("close", () => {
console.log("window closed")
mainWindow.webContents.send("closed", { message: "window closed" });
});
preload
contextBridge.exposeInMainWorld("ipcRenderer", {
invoke: ipcRenderer.invoke,
on: ipcRenderer.on,
});
renderer (mainWindow)
window.ipcRenderer.on("closed", () => {
console.log("message received")
// do something
});
I know for a fact that
mainWindow has access to the exposed listeners, since invoke works and the actions it fires on the main process are executed on the actionWindow as supposed + the response also comes back to the renderer.
the close listener on the actionWindow works since I can see the log window closed in my console
message received doesn't appear in my dev tools console
To me this means that either
mainWindow.webContents.send doesn't work -> the message is never sent
window.ipcRenderer.on doesn't work -> the message never reaches its destination
So either my code is buggy or Electron has recently put some restrictions on one of these methods which I'm not aware of.
Any ideas?
If there is a smarter way to do this than IPC I'm also open to that.
Ok after hours of searching, trying and suffering I (almost accidentaly) found a solution to my problem. It really seems to be the case that electron simply doesn't do anything anymore when you call the on method from your renderer.
Studying the docs about contextBridge again I saw that the way I exposed invoke and on to the renderer, was considered bad code. The safer way to do this is expose a function for EVERY ipc channel you want to use. In my case using TypeScript it looks like this:
preload
contextBridge.exposeInMainWorld("ipcRenderer", {
invokeOpen: async (optionsString: string) => {
await ipcRenderer.invoke("open", optionsString);
},
onClose: (callback: () => void) => {
ipcRenderer.on("closed", callback);
},
removeOnClose: (callback: () => void) => {
ipcRenderer.removeListener("closed", callback);
},
});
renderer(mainWindow)
window.ipcRenderer.onClose(() => {
// do sth
});
window.ipcRenderer.invokeOpen(JSON.stringify(someData)).then(() => {
// do sth when response came back
});
NOTE: To prevent memory leaks by creating listeners on every render of the mainWindow you also have to use a cleanup function which is provided with removeOnClose (see preload). How to use this function differs depending on the frontend framework. Using React it looks like this:
const doSth= () => {
console.log("doing something")
...
};
useEffect(() => {
window.ipcRenderer.onClose(doSth);
return () => {
window.ipcRenderer.removeOnClose(doSth);
};
}, []);
Not only is this a safer solution, it actually suddenly works :O
Using the cleanup function we also take care of leaks.
I'm working on a SPA with Vue. I'd like to update to a new service-worker when the user navigates to a specific page. A save moment to refresh, because the view of the user already changes (a pattern discussed in this video: https://youtu.be/cElAoxhQz6w)
I have an issue that sometimes (infrequently) the service-worker won't activate while calling skipWaiting. The call is made correctly, and even in Chrome I get a response that the current service-worker stops (see animated GIF), however it the same service-worker starts running again, instead of the waiting one.
After a while (1-2 minutes) the service-worker is suddenly activated. Not a situation you want, because it happens just out of the blue when the user might be in the middle of an activity.
Also when I am in this situation I can't activate the service-worker by calling skipWaiting (by doing multiple navigations) again. It's received by the service-worker but nothing happens. It stays in "waiting to activate". When I press skipWaiting in Chrome itself, it works.
I have no clue what goes wrong. Is this an issue with Chrome, workbox or something else?
Most close comes this topic: self.skipWaiting() not working in Service Worker
I use Vue.js, but I don't depend on the pwa plugin for the service-worker. I use the workbox webpack plugin.
I've edited the example code below, the minimal code probably didn't show the problem well
In main.js:
let sw = await navigator.serviceWorker.register("/service-worker.js", {
updateViaCache: "none",
});
let firstSw = false;
navigator.serviceWorker.addEventListener("controllerchange", () => {
// no need to refresh when the first sw controls the page, we solve this with clientsClaim
// this makes sure when multiple-tabs are open all refresh
if (!firstSw) {
window.location.reload();
}
});
sw.onupdatefound = () => {
const installingWorker = sw.installing;
installingWorker.onstatechange = async () => {
console.log("installing worker state-change: " + installingWorker.state);
if (installingWorker.state === "installed") {
if (navigator.serviceWorker.controller) {
firstSw = false;
// set the waiting service-worker in the store
// so we can update it and refresh the page on navigation
await store.dispatch("setWaitingSW", sw.waiting);
} else {
console.log("First sw available");
firstSw = true;
}
}
};
};
In router.js:
// after navigation to specific routes we check for a waiting service-worker.
router.afterEach(async (to) => {
if (to.name == "specificpage") {
let waitingSw = store.getters["getWaitingSW"];
if (waitingSw) {
waitingSw.postMessage("SKIP_WAITING");
// clean the store, because we might have changed our data model
await store.dispatch("cleanLocalForage");
}
}
});
In service-worker.js:
self.addEventListener("message", event => {
if (event.data === "SKIP_WAITING") {
console.log("sw received skip waiting");
self.skipWaiting();
}
});
skipWaiting() isn't instant. If there are active fetches going through the current service worker, it won't break those. If you're seeing skipWaiting() taking a long time, I'd guess you have some long-running HTTP connections holding the old service worker in place.
I'm not sure that
let sw = await navigator.serviceWorker.register("/service-worker.js", {updateViaCache: "none"});
if (sw.waiting) {
sw.waiting.postMessage("SKIP_WAITING");
}
is the code that you want in this case. Your if (sw.waiting) check is only evaluated once, and the newly registered service worker might still be in the installing state when it's evaluated. If that's the case, then sw.waiting will be false-y at the time of initial evaluation, though it may be true-thy after a small period of time.
Instead, I'd recommend following a pattern like what's demonstrated in this recipe, where you explicitly listen for a service worker to enter waiting on the registration. That example uses the workbox-window library to paper over some of the details.
If you don't want to use workbox-window, you should follow this guidance check to see if sw.installing is set after registration; if it is, listen to the statechange event on sw.installing to detect when it's 'installed'. Once that happens, sw.waiting should be set to the newly installed service worker, and at that point, you could postMessage() to it.
Ok i had a similar issue and it took me two days to find the cause.
There is a scenario where you can cause a race condition between the new service worker and the old if you request a precached asset at the exact same time you call skip waiting.
For me i was prompting the user to update to a new version and upon their confirmation i was showing a loading spinner which was a Vue SFC dynamic import which kicked off a network request to the old service worker to fetch the precached js file which basically caused both to hang and get very confused.
You can check if your having a similar issue by looking at the service worker specific network requests (Network requests button in the image below) that are happening and make sure they aren't happening the instant you're trying to skip waiting on your newer service worker.
I am working on offline support in my PWA app. I am using workbox for that. This is my current code:
const addToFormPlugin = new workbox.backgroundSync.Plugin('addToForm');
workbox.routing.registerRoute(
RegExp('MY_PATH'),
workbox.strategies.networkOnly({
plugins: [addToFormPlugin]
}),
'POST'
);
The code seems to works fine on my computer. However, once I run the app on the phone it takes ages to upload requests stored in IndexedDB. I know that it happens on the SYNC but it seems to take at least 5 minutes. This is not exactly what I need. I wonder if there is an option to access the IndexDB and send all the requests "manually" on click. Another way would be to check if the device is online. Here is how requests are stored:
If you need to force this, the cleanest approach would be to use the workbox.backgroundSync.Queue class (instead of workbox.backgroundSync.Plugin) directly.
The Plugin class takes care of setting up a fetchDidFail callback for you, so if you use the Queue class, you need to do that yourself:
const queue = new workbox.backgroundSync.Queue('addToForm');
workbox.routing.registerRoute(
RegExp('MY_PATH'),
workbox.strategies.networkOnly({
plugins: [{
fetchDidFail: async ({request}) => {
await queue.addRequest(request);
},
}],
}),
'POST'
);
You could then call queue.replayRequests() to trigger the replay, e.g., as a result of a message event:
self.addEventListener('message', (event) => {
if (event.data === 'replayRequests') {
queue.replayRequests();
}
});
But... that all being said, I think your best bet is just to let the browser "do its thing" and figure out when the right time is to replay the queued requests. That will end up being more battery-friendly for mobile devices.
If you're unhappy with the interval that the browser waits before firing a sync event, then the best course of action could be to open a bug against the browser—whether it's Chrome (as appears in your screenshot) or another browser.
<button class="alt-button" ion-item detail-none (click)="goToAbout()" clear small >
<ion-icon name='person' item-left></ion-icon>About us</button>
Button action
goToAbout() {
this.menu.close();
// close the menu and Goto About page
this.app.getRootNav().push(AboutPage);
}
api call
ionViewDidLoad(){
this.loading.present();
this.aboutservice.getPost().then(
response => {
this.items = response
this.loading.dismiss();
},
error=>{
this.error = error
this.showError('There was problem with server');
this.loading.dismiss();
});
}
it loads the api data everytime,but I want to load api data once and same button action i have used for sidemenu,Its working fine.please give any idea.
It seems strange behavior according to the doc here.
ionViewDidLoad
Runs when the page has loaded. This event only happens once per page
being created. If a page leaves but is cached, then this event will
not fire again on a subsequent viewing. The ionViewDidLoad event is
good place to put your setup code for the page.
But you can use it inside the constructor() as shown below.
constructor() {
//your Api call here
}
I've got a Google Searchbar-type input field. When I type in a couple characters and wait for half a second it runs the ajax call to an external website I've set in the "source" function of the autocomplete code and once it has returned the results it returns it to the screen (like it should).
The problem is that while the ajax call is being run to fetch the results it won't allow me to continue typing in the input field until the ajax call has completed.
How can I get it to allow me to continue typing while the ajax call is being made?
Here is my jQuery function:
$('#googleSearchbar').autocomplete({
minLength: 2,
autoFocus: true,
delay: 500,
source: function (request, response) {
results = $.parseJSON($(this).callJson('post', 'http://my_external_url', {
data: request.term
}));
response(results);
},
error: function (err) {
console.error('ERROR : ' + err);
return false;
}
});
I have a hunch you are blocking the browser when making your AJAX request. This line:
results = $.parseJSON($(this).callJson('post', 'http://my_external_url', {
data: request.term
}));
Makes me think that $(this).callJson(...) is a synchronous request, which is going to lock up the entire browser for the duration of the request.
You need to make an asynchronous request and call the response function when that request completes. This should stop the browser from locking up.