Hi I need to create a robot that opens a website, does the sign procedure, then open another form and fill the data before sending the data to a post command by clicking on a button.
What is the best tool for this kind of operation, I tried playwright, selenium and puppetter but they all seem to be unreliable.
Any ideas?
•
Those are exactly the tools likely to be recommended. Can you elaborate on why "they all seem to be unreliable" to you? Have you made an attempt and encountered a problem of some kind?
•
I call the login page, input the credential, click on the sign in button, then it try to get to a specific page on the same website but sometimes it works other times it just get stuck or never gets to the url even with large timeouts. Does it effect the flow if it is not headless?
e.g. I have this call
await page.GotoAsync("https://#######/nies/index?#procedures/import/normal/from-ts", new PageGotoOptions() { Timeout = TimeOutFromSeconds(20) });
sometime it work and sometimes it will just get timed out. I am using playwright in this case.
•
It sounds like the problem may have more to do with the target website than with the libraries used. One thing you could do to test the tools is try using them on different target sites, different target pages on the same site, simpler pages, etc. You may be able to narrow down the problem a bit that way.
Headless vs. not headless shouldn't make a difference for most intents and purposes (though there is a difference in the requests), but that's certainly worth testing too. Honestly, it could be anything at this time. It's not unprecedented for servers to just quietly rate-limit or even block requests they deem to be automated in some way, too.
•
Playwright is the industry standard tool for browser automation, and not the problem here. Asking for different tools is going to run you in circles--instead, ask about the specific issue you're encountering, preferably as a standard Stack Overflow question.
Headless vs headed DOES make a difference by changing the fingerprint dramatically.
Please include in your question:
We can then help with the specific problem you're facing.
•
I think the issue is with the target website, after the first call it seems it is identifying the call as originating from a robot and is blocking my calls for a number of minutes. Is there a way to avoid this?
Many websites attempt to block web scrapers and bots. This might actually be intended behavior from the perspective of the website owner. If you work for the website owner, then you might need those protections turned off, or you might need to target a version of the website that does not sit behind something like Cloudflare. Content delivery networks like Cloudflare often implement anti-bot measures.
If you are not the website owner, then read their terms of use. If they prohibit bot traffic then you would be in violation of their terms of use and subject to whatever craziness they throw at you.
