How to connect and store the cookie with Scrappy - session-cookies

Question

I want to login to a site with Scrappy and after that call another url.
So far so good I installed Scrappy and made this script:
from scrapy.spider import BaseSpider
from scrapy.selector import HtmlXPathSelector
from scrapy.http import FormRequest
class LoginSpider2(BaseSpider):
name = 'github_login'
start_urls = ['https://github.com/login']
def parse(self, response):
return [FormRequest.from_response(response, formdata={'login': 'username', 'password': 'password'}, callback=self.after_login)]
def after_login(self, response):
if "authentication failed" in response.body:
self.log("Login failed", level=log.ERROR)
else:
self.log("Login succeed", response.body)
After launching this script, I got the log "Login succeed".
Then I added another URL but it didn't work:
To do that I replaced:
start_urls = ['https://github.com/login']
by
start_urls = ['https://github.com/login', 'https://github.com/MyCompany/MyPrivateRepo']
But I got these errors:
2013-06-11 22:23:40+0200 [scrapy] DEBUG: Enabled item pipelines:
Traceback (most recent call last):
File "/usr/local/bin/scrapy", line 4, in <module>
execute()
File "/Library/Python/2.7/site-packages/scrapy/cmdline.py", line 131, in execute
_run_print_help(parser, _run_command, cmd, args, opts)
File "/Library/Python/2.7/site-packages/scrapy/cmdline.py", line 76, in _run_print_help
func(*a, **kw)
File "/Library/Python/2.7/site-packages/scrapy/cmdline.py", line 138, in _run_command
cmd.run(args, opts)
File "/Library/Python/2.7/site-packages/scrapy/commands/crawl.py", line 43, in run
spider = self.crawler.spiders.create(spname, **opts.spargs)
File "/Library/Python/2.7/site-packages/scrapy/spidermanager.py", line 43, in create
raise KeyError("Spider not found: %s" % spider_name)
What I am doing wrong ? I searched on stackoverflow but I didn't find a proper response..
Thanks you

Answer 1

Your error indicates that Scrapy is not able to find the spider. Did you create it in the project/spiders folder?
Anyway, once you get it to run you will find a second issue: the default callback for the start_url requests is self.parse, which will fail for the repo page (there's no login form there). And they will probably run in parallel, so by the time it visits the private repo, it will get an error :P
You should leave only the login url in start_urls, and return a new Request in the after_login method, if it worked. Like this:
def after_login(self, response):
...
else:
return Request('https://github.com/MyCompany/MyPrivateRepo',
callback=self.parse_repo)

Answer 2

Is the name attribute of the spider still set correctly? Incorrect/missing setting of name usually leads to errors like these.