Syntax error while trying to run scrapy crawl on Mac?

Question

I installed Scrapy as suggested on Scrapy.org, installing Xcode, homebrew, and then using the command "pip3 install scrapy".

I also used "sudo easy_install pip" after a few errors with the pip command.

Now I'm getting this SyntaxError whenever I try running my FirstSpider, scrapy crawl FirstSpider:

  File "/usr/local/lib/python3.7/site-packages/twisted/conch/manhole.py", line 154
    def write(self, data, async=False):
                              ^
SyntaxError: invalid syntax

Can anyone explain what's happening or how to solve it?

Full error report:

$ scrapy crawl FirstSpider
2018-08-30 23:14:58 [scrapy.utils.log] INFO: Scrapy 1.5.1 started (bot: Example)
2018-08-30 23:14:58 [scrapy.utils.log] INFO: Versions: lxml 4.2.4.0, libxml2 2.9.8, cssselect 1.0.3, parsel 1.5.0, w3lib 1.19.0, Twisted 18.7.0, Python 3.7.0 (default, Aug 22 2018, 15:22:33) - [Clang 9.1.0 (clang-902.0.39.2)], pyOpenSSL 18.0.0 (OpenSSL 1.1.0i  14 Aug 2018), cryptography 2.3.1, Platform Darwin-17.7.0-x86_64-i386-64bit
2018-08-30 23:14:58 [scrapy.crawler] INFO: Overridden settings: {'BOT_NAME': 'Example', 'NEWSPIDER_MODULE': 'Example.spiders', 'ROBOTSTXT_OBEY': True, 'SPIDER_MODULES': ['Example.spiders']}
Traceback (most recent call last):
  File "/usr/local/lib/scrapy", line 11, in <module>
    sys.exit(execute())
  File "/usr/local/lib/python3.7/site-packages/scrapy/cmdline.py", line 150, in execute
    _run_print_help(parser, _run_command, cmd, args, opts)
  File "/usr/local/lib/python3.7/site-packages/scrapy/cmdline.py", line 90, in _run_print_help
    func(*a, **kw)
  File "/usr/local/lib/python3.7/site-packages/scrapy/cmdline.py", line 157, in _run_command
    cmd.run(args, opts)
  File "/usr/local/lib/python3.7/site-packages/scrapy/commands/crawl.py", line 57, in run
    self.crawler_process.crawl(spname, **opts.spargs)
  File "/usr/local/lib/python3.7/site-packages/scrapy/crawler.py", line 170, in crawl
    crawler = self.create_crawler(crawler_or_spidercls)
  File "/usr/local/lib/python3.7/site-packages/scrapy/crawler.py", line 198, in create_crawler
    return self._create_crawler(crawler_or_spidercls)
  File "/usr/local/lib/python3.7/site-packages/scrapy/crawler.py", line 203, in _create_crawler
    return Crawler(spidercls, self.settings)
  File "/usr/local/lib/python3.7/site-packages/scrapy/crawler.py", line 55, in __init__
    self.extensions = ExtensionManager.from_crawler(self)
  File "/usr/local/lib/python3.7/site-packages/scrapy/middleware.py", line 58, in from_crawler
    return cls.from_settings(crawler.settings, crawler)
  File "/usr/local/lib/python3.7/site-packages/scrapy/middleware.py", line 34, in from_settings
    mwcls = load_object(clspath)
  File "/usr/local/lib/python3.7/site-packages/scrapy/utils/misc.py", line 44, in load_object
    mod = import_module(module)
  File "/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/usr/local/lib/python3.7/site-packages/scrapy/extensions/telnet.py", line 12, in <module>
    from twisted.conch import manhole, telnet
  File "/usr/local/lib/python3.7/site-packages/twisted/conch/manhole.py", line 154
    def write(self, data, async=False):
                              ^
SyntaxError: invalid syntax

Please do not use images for error messages. Now your text can't be indexed into search engines, so no-one else with the same problem can find this post now. — Martijn Pieters, Aug 30 '18 at 21:48
Please copy the text into your question and don't use external images for that. — Michael Butscher, Aug 30 '18 at 21:48

Martijn Pieters · Answer 1 · 2018-08-30T21:53:40.900

0

The code you are using needs to be updated for the newer Python version you are using, as async has become a reserved keyword in Python 3.7.

Ask the project that you are using to fix this. In the meantime, downgrading to Python 3.6 will let you run the software.

The traceback shows you that the error is in the twisted project, which does not yet support Python 3.7. The project is aware of this and actively working towards fixing the problem.

edited Aug 30 '18 at 21:53

answered Aug 30 '18 at 21:49

Martijn Pieters

1,048,767
296
4,058
3,343

[It's fixed in master](https://github.com/twisted/twisted/blob/ee535041258e7ef0b3223d2e12cd9aaa0bc2289f/src/twisted/conch/manhole.py#L156), so now we're waiting for a release. – user2357112 Aug 30 '18 at 21:50
@user2357112: the [support 3.7 milestone](https://twistedmatrix.com/trac/query?group=status&milestone=Python+3.7+support) is still open with more tickets; the most important one that you [can't yet compile twisted on Python 3.7](https://twistedmatrix.com/trac/ticket/9501). – Martijn Pieters Aug 30 '18 at 21:53
Well, at least the `async` argument name is fixed. Looks like there's still a bunch of other stuff to fix. – user2357112 Aug 30 '18 at 21:55

Syntax error while trying to run scrapy crawl on Mac?

1 Answers1