Skip to content

Fix World Grain date parser #83

Description

@mkao006

There appears to be multiple erros when parsing the world grain dates.

This may be considered a minor bug, and the effort may not be justified.

2018-06-22 15:32:25 [scrapy.core.scraper] ERROR: Spider error processing <GET http://www.world-grain.com/articles/news_home/Site_News/2012/08/GRFA_responds_to_IFPRI_report.aspx?ID=%7B4791C417-B261-47A3-9778-DFD017DF1905%7D&p=1> (referer: http://www.world-grain.com/articles/news_home/Site_News/2012/08/GRFA_responds_to_IFPRI_report.aspx?ID=%7B4791C417-B261-47A3-9778-DFD017DF1905%7D)
Traceback (most recent call last):
  File "/home/mk/Git/fao/est/TheReadingMachine/venv/local/lib/python2.7/site-packages/scrapy/utils/defer.py", line 102, in iter_errback
    yield next(it)
  File "/home/mk/Git/fao/est/TheReadingMachine/venv/local/lib/python2.7/site-packages/scrapy/spidermiddlewares/offsite.py", line 30, in process_spider_output
    for x in result:
  File "/home/mk/Git/fao/est/TheReadingMachine/venv/local/lib/python2.7/site-packages/scrapy/spidermiddlewares/referer.py", line 339, in <genexpr>
    return (_set_referer(r) for r in result or ())
  File "/home/mk/Git/fao/est/TheReadingMachine/venv/local/lib/python2.7/site-packages/scrapy/spidermiddlewares/urllength.py", line 37, in <genexpr>
    return (r for r in result or () if _filter(r))
  File "/home/mk/Git/fao/est/TheReadingMachine/venv/local/lib/python2.7/site-packages/scrapy/spidermiddlewares/depth.py", line 58, in <genexpr>
    return (r for r in result or () if _filter(r))
  File "/home/mk/Git/fao/est/TheReadingMachine/venv/local/lib/python2.7/site-packages/scrapy/spiders/crawl.py", line 76, in _parse_response
    cb_res = callback(response, **cb_kwargs) or ()
  File "/home/mk/Git/fao/est/TheReadingMachine/thereadingmachine/scraper/news_scraper/spiders/spiders.py", line 216, in parse_item
    clean_date = self._parse_wg_date(self._clean_date(raw_date))
  File "/home/mk/Git/fao/est/TheReadingMachine/thereadingmachine/scraper/news_scraper/spiders/spiders.py", line 160, in _parse_wg_date
    raise ValueError('no valid date format found for ' + ds)

or date with different format,

2018-06-22 15:34:42 [scrapy.core.scraper] ERROR: Spider error processing <GET http://www.world-grain.com/articles/news_home/World_Grain_News/2013/07/All_US_wheat_production_drops.aspx?ID=%7B3BBC54BE-3C53-4136-8C10-8B621C7E05EC%7D> (referer: http://www.world-grain.com/News/Archive.aspx?page=19&year=2013&month=0)
Traceback (most recent call last):
  File "/home/mk/Git/fao/est/TheReadingMachine/venv/local/lib/python2.7/site-packages/scrapy/utils/defer.py", line 102, in iter_errback
    yield next(it)
  File "/home/mk/Git/fao/est/TheReadingMachine/venv/local/lib/python2.7/site-packages/scrapy/spidermiddlewares/offsite.py", line 30, in process_spider_output
    for x in result:
  File "/home/mk/Git/fao/est/TheReadingMachine/venv/local/lib/python2.7/site-packages/scrapy/spidermiddlewares/referer.py", line 339, in <genexpr>
    return (_set_referer(r) for r in result or ())
  File "/home/mk/Git/fao/est/TheReadingMachine/venv/local/lib/python2.7/site-packages/scrapy/spidermiddlewares/urllength.py", line 37, in <genexpr>
    return (r for r in result or () if _filter(r))
  File "/home/mk/Git/fao/est/TheReadingMachine/venv/local/lib/python2.7/site-packages/scrapy/spidermiddlewares/depth.py", line 58, in <genexpr>
    return (r for r in result or () if _filter(r))
  File "/home/mk/Git/fao/est/TheReadingMachine/venv/local/lib/python2.7/site-packages/scrapy/spiders/crawl.py", line 76, in _parse_response
    cb_res = callback(response, **cb_kwargs) or ()
  File "/home/mk/Git/fao/est/TheReadingMachine/thereadingmachine/scraper/news_scraper/spiders/spiders.py", line 216, in parse_item
    clean_date = self._parse_wg_date(self._clean_date(raw_date))
  File "/home/mk/Git/fao/est/TheReadingMachine/thereadingmachine/scraper/news_scraper/spiders/spiders.py", line 160, in _parse_wg_date
    raise ValueError('no valid date format found for ' + ds)
ValueError: no valid date format found for Jul 12, 2013

In addition, sometimes the extracted date is empty which results in the following error.

2018-06-22 15:32:31 [scrapy.core.scraper] ERROR: Spider error processing <GET http://www.world-grain.com/errors/404.aspx?item=%2farticles%2fnews_home%2fsite_news%2f2012%2f08%2fgrfa_responds_to_ifpri_report&user=extranet%5cAnonymous&site=wg> (referer: http://www.world-grain.com/articles/news_home/Site_News/2012/08/GRFA_responds_to_IFPRI_report.aspx?ID=%7B4791C417-B261-47A3-9778-DFD017DF1905%7D)
Traceback (most recent call last):
  File "/home/mk/Git/fao/est/TheReadingMachine/venv/local/lib/python2.7/site-packages/scrapy/utils/defer.py", line 102, in iter_errback
    yield next(it)
  File "/home/mk/Git/fao/est/TheReadingMachine/venv/local/lib/python2.7/site-packages/scrapy/spidermiddlewares/offsite.py", line 30, in process_spider_output
    for x in result:
  File "/home/mk/Git/fao/est/TheReadingMachine/venv/local/lib/python2.7/site-packages/scrapy/spidermiddlewares/referer.py", line 339, in <genexpr>
    return (_set_referer(r) for r in result or ())
  File "/home/mk/Git/fao/est/TheReadingMachine/venv/local/lib/python2.7/site-packages/scrapy/spidermiddlewares/urllength.py", line 37, in <genexpr>
    return (r for r in result or () if _filter(r))
  File "/home/mk/Git/fao/est/TheReadingMachine/venv/local/lib/python2.7/site-packages/scrapy/spidermiddlewares/depth.py", line 58, in <genexpr>
    return (r for r in result or () if _filter(r))
  File "/home/mk/Git/fao/est/TheReadingMachine/venv/local/lib/python2.7/site-packages/scrapy/spiders/crawl.py", line 76, in _parse_response
    cb_res = callback(response, **cb_kwargs) or ()
  File "/home/mk/Git/fao/est/TheReadingMachine/thereadingmachine/scraper/news_scraper/spiders/spiders.py", line 216, in parse_item
    clean_date = self._parse_wg_date(self._clean_date(raw_date))
  File "/home/mk/Git/fao/est/TheReadingMachine/thereadingmachine/scraper/news_scraper/spiders/spiders.py", line 163, in _clean_date
    clean = (raw.replace('\t', '')
AttributeError: 'NoneType' object has no attribute 'replace'

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions