Response and request.get different for (https://www.watsons.com.sg)

Hi, I have been trying to scrape https://www.watsons.com.sg. If I use scrapy view https://www.watsons.com.sg it will return in 400 bad request. But if I use r=requests.get(“https://www.watsons.com.sg”) i get the content of the webpage in r.text and r.content. However all the xpath operation in my script will not work this way as r.xpath and r.response has nothing.  Is there anyway to get around this? All help and comments appreciated.

@pc2000sg You appear to be having an issue with usage of Scrapy, which is one of many open source projects hosted here on GitHub.  The focus of this forum is to help you with your usage of GitHub itself with your projects, and so it is unlikely that anyone here will have enough experience with Scrapy to be able to provide the assistance you need.  I suggest you engage with the Scrapy community directly and ask your question there! It is likely that you will get a good response if you go to one of the locations listed on the Scrapy community page.

1 Like