2025-03-21 04:59:05 [scrapy.utils.log] (PID: 26) INFO: Scrapy 2.12.0 started (bot: catalog_extraction)
2025-03-21 04:59:05 [scrapy.utils.log] (PID: 26) INFO: Versions: lxml 5.3.1.0, libxml2 2.12.9, cssselect 1.3.0, parsel 1.10.0, w3lib 2.3.1, Twisted 24.11.0, Python 3.11.11 (main, Mar 17 2025, 23:23:20) [GCC 12.2.0], pyOpenSSL 25.0.0 (OpenSSL 3.4.1 11 Feb 2025), cryptography 44.0.2, Platform Linux-6.9.12-x86_64-with-glibc2.36
2025-03-21 04:59:05 [fastenal] (PID: 26) INFO: Starting extraction spider fastenal...
2025-03-21 04:59:05 [scrapy.addons] (PID: 26) INFO: Enabled addons:
[]
2025-03-21 04:59:05 [py.warnings] (PID: 26) WARNING: /usr/local/lib/python3.11/site-packages/scrapy/utils/request.py:120: ScrapyDeprecationWarning: 'REQUEST_FINGERPRINTER_IMPLEMENTATION' is a deprecated setting.
It will be removed in a future version of Scrapy.
  return cls(crawler)

2025-03-21 04:59:05 [scrapy.extensions.telnet] (PID: 26) INFO: Telnet Password: d828da147012f67f
2025-03-21 04:59:05 [py.warnings] (PID: 26) WARNING: /var/lib/scrapyd/eggs/catalog_extraction/1742264469.egg/catalog_extraction/extensions/bq_feedstorage.py:33: ScrapyDeprecationWarning: scrapy.extensions.feedexport.build_storage() is deprecated, call the builder directly.

2025-03-21 04:59:06 [scrapy.middleware] (PID: 26) INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
 'scrapy.extensions.telnet.TelnetConsole',
 'scrapy.extensions.closespider.CloseSpider',
 'scrapy.extensions.feedexport.FeedExporter',
 'scrapy.extensions.logstats.LogStats',
 'scrapy_patchright.memusage.ScrapyPlaywrightMemoryUsageExtension',
 'spidermon.contrib.scrapy.extensions.Spidermon']
2025-03-21 04:59:06 [scrapy.crawler] (PID: 26) INFO: Overridden settings:
{'BOT_NAME': 'catalog_extraction',
 'CONCURRENT_ITEMS': 250,
 'CONCURRENT_REQUESTS': 24,
 'FEED_EXPORT_ENCODING': 'utf-8',
 'HTTPPROXY_ENABLED': False,
 'LOG_FILE': '/var/lib/scrapyd/logs/catalog_extraction/fastenal/3345feca061111f0aeca4200a9fe0102.log',
 'LOG_FORMAT': '%(asctime)s [%(name)s] (PID: %(process)d) %(levelname)s: '
               '%(message)s',
 'LOG_LEVEL': 'INFO',
 'NEWSPIDER_MODULE': 'catalog_extraction.spiders',
 'REQUEST_FINGERPRINTER_CLASS': 'scrapy_poet.ScrapyPoetRequestFingerprinter',
 'REQUEST_FINGERPRINTER_IMPLEMENTATION': '2.7',
 'SPIDER_MODULES': ['catalog_extraction.spiders'],
 'TWISTED_REACTOR': 'twisted.internet.asyncioreactor.AsyncioSelectorReactor',
 'USER_AGENT': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, '
               'like Gecko) Chrome/129.0.0.0 Safari/537.36'}
2025-03-21 04:59:07 [scrapy_poet.injection] (PID: 26) INFO: Loading providers:
 [<class 'scrapy_poet.page_input_providers.HttpRequestProvider'>,
 <class 'scrapy_poet.page_input_providers.HttpResponseProvider'>,
 <class 'scrapy_poet.page_input_providers.HttpClientProvider'>,
 <class 'scrapy_poet.page_input_providers.PageParamsProvider'>,
 <class 'scrapy_poet.page_input_providers.RequestUrlProvider'>,
 <class 'scrapy_poet.page_input_providers.ResponseUrlProvider'>,
 <class 'scrapy_poet.page_input_providers.StatsProvider'>]
2025-03-21 04:59:07 [scrapy.middleware] (PID: 26) INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.offsite.OffsiteMiddleware',
 'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
 'scrapy_poet.InjectionMiddleware',
 'scrapy.downloadermiddlewares.retry.RetryMiddleware',
 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
 'scrapy_poet.DownloaderStatsMiddleware']
2025-03-21 04:59:07 [NotFoundHandlerSpiderMiddleware] (PID: 26) INFO: NotFoundHandlerSpiderMiddleware running in DRYRUN mode.
2025-03-21 04:59:07 [scrapy.middleware] (PID: 26) INFO: Enabled spider middlewares:
['catalog_extraction.middlewares.NotFoundHandlerSpiderMiddleware',
 'catalog_extraction.middlewares.FixtureSavingMiddleware',
 'scrapy_poet.RetryMiddleware',
 'scrapy.spidermiddlewares.referer.RefererMiddleware',
 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
 'scrapy.spidermiddlewares.depth.DepthMiddleware']
2025-03-21 04:59:07 [scrapy.middleware] (PID: 26) INFO: Enabled item pipelines:
['catalog_extraction.pipelines.DuplicatedSKUsFilterPipeline',
 'catalog_extraction.pipelines.DiscontinuedProductsAdjustmentPipeline',
 'catalog_extraction.pipelines.PriceRoundingPipeline',
 'scraping_utils.pipelines.AttachSupplierPipeline',
 'spidermon.contrib.scrapy.pipelines.ItemValidationPipeline']
2025-03-21 04:59:07 [scrapy.core.engine] (PID: 26) INFO: Spider opened
2025-03-21 04:59:07 [scrapy.extensions.closespider] (PID: 26) INFO: Spider will stop when no items are produced after 7200 seconds.
2025-03-21 04:59:07 [scrapy.extensions.logstats] (PID: 26) INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2025-03-21 04:59:07 [scrapy.extensions.telnet] (PID: 26) INFO: Telnet console listening on 127.0.0.1:6023
2025-03-21 04:59:07 [scrapy-playwright] (PID: 26) INFO: Starting download handler
2025-03-21 04:59:07 [scrapy-playwright] (PID: 26) INFO: Starting download handler
2025-03-21 04:59:12 [fastenal] (PID: 26) INFO: Batch query job with batch size 100 and offset 0
2025-03-21 04:59:14 [scrapy.spidermiddlewares.httperror] (PID: 26) INFO: Ignoring response <403 https://www.fastenal.com/product/details/0110574>: HTTP status code is not handled or not allowed
2025-03-21 04:59:14 [scrapy.spidermiddlewares.httperror] (PID: 26) INFO: Ignoring response <403 https://www.fastenal.com/product/details/0110030>: HTTP status code is not handled or not allowed
2025-03-21 04:59:14 [scrapy.spidermiddlewares.httperror] (PID: 26) INFO: Ignoring response <403 https://www.fastenal.com/product/details/0110019>: HTTP status code is not handled or not allowed
2025-03-21 04:59:14 [scrapy.spidermiddlewares.httperror] (PID: 26) INFO: Ignoring response <403 https://www.fastenal.com/product/details/0110006>: HTTP status code is not handled or not allowed
2025-03-21 04:59:14 [scrapy.spidermiddlewares.httperror] (PID: 26) INFO: Ignoring response <403 https://www.fastenal.com/product/details/0110237>: HTTP status code is not handled or not allowed
2025-03-21 04:59:14 [scrapy.spidermiddlewares.httperror] (PID: 26) INFO: Ignoring response <403 https://www.fastenal.com/product/details/0110475>: HTTP status code is not handled or not allowed
2025-03-21 04:59:14 [scrapy.spidermiddlewares.httperror] (PID: 26) INFO: Ignoring response <403 https://www.fastenal.com/product/details/0110332>: HTTP status code is not handled or not allowed
2025-03-21 04:59:14 [scrapy.spidermiddlewares.httperror] (PID: 26) INFO: Ignoring response <403 https://www.fastenal.com/product/details/0110689>: HTTP status code is not handled or not allowed
2025-03-21 04:59:14 [scrapy.spidermiddlewares.httperror] (PID: 26) INFO: Ignoring response <403 https://www.fastenal.com/product/details/0110690>: HTTP status code is not handled or not allowed
2025-03-21 04:59:14 [scrapy.spidermiddlewares.httperror] (PID: 26) INFO: Ignoring response <403 https://www.fastenal.com/product/details/0110679>: HTTP status code is not handled or not allowed
2025-03-21 04:59:14 [scrapy.spidermiddlewares.httperror] (PID: 26) INFO: Ignoring response <403 https://www.fastenal.com/product/details/0110579>: HTTP status code is not handled or not allowed
2025-03-21 04:59:14 [scrapy.spidermiddlewares.httperror] (PID: 26) INFO: Ignoring response <403 https://www.fastenal.com/product/details/0110700>: HTTP status code is not handled or not allowed
2025-03-21 04:59:14 [scrapy.spidermiddlewares.httperror] (PID: 26) INFO: Ignoring response <403 https://www.fastenal.com/product/details/0110489>: HTTP status code is not handled or not allowed
2025-03-21 04:59:14 [scrapy.spidermiddlewares.httperror] (PID: 26) INFO: Ignoring response <403 https://www.fastenal.com/product/details/0110711>: HTTP status code is not handled or not allowed
2025-03-21 04:59:14 [scrapy.spidermiddlewares.httperror] (PID: 26) INFO: Ignoring response <403 https://www.fastenal.com/product/details/0110731>: HTTP status code is not handled or not allowed
2025-03-21 04:59:14 [scrapy.spidermiddlewares.httperror] (PID: 26) INFO: Ignoring response <403 https://www.fastenal.com/product/details/0110732>: HTTP status code is not handled or not allowed
2025-03-21 04:59:14 [scrapy.spidermiddlewares.httperror] (PID: 26) INFO: Ignoring response <403 https://www.fastenal.com/product/details/0110733>: HTTP status code is not handled or not allowed
2025-03-21 04:59:14 [scrapy.spidermiddlewares.httperror] (PID: 26) INFO: Ignoring response <403 https://www.fastenal.com/product/details/0110734>: HTTP status code is not handled or not allowed
2025-03-21 04:59:14 [scrapy.spidermiddlewares.httperror] (PID: 26) INFO: Ignoring response <403 https://www.fastenal.com/product/details/0110736>: HTTP status code is not handled or not allowed
2025-03-21 04:59:14 [scrapy.spidermiddlewares.httperror] (PID: 26) INFO: Ignoring response <403 https://www.fastenal.com/product/details/0110735>: HTTP status code is not handled or not allowed
2025-03-21 04:59:14 [scrapy.spidermiddlewares.httperror] (PID: 26) INFO: Ignoring response <403 https://www.fastenal.com/product/details/0110737>: HTTP status code is not handled or not allowed
2025-03-21 04:59:14 [scrapy.spidermiddlewares.httperror] (PID: 26) INFO: Ignoring response <403 https://www.fastenal.com/product/details/0110752>: HTTP status code is not handled or not allowed
2025-03-21 04:59:14 [scrapy.spidermiddlewares.httperror] (PID: 26) INFO: Ignoring response <403 https://www.fastenal.com/product/details/0110804>: HTTP status code is not handled or not allowed
2025-03-21 04:59:14 [scrapy.spidermiddlewares.httperror] (PID: 26) INFO: Ignoring response <403 https://www.fastenal.com/product/details/0110811>: HTTP status code is not handled or not allowed
2025-03-21 04:59:14 [scrapy.spidermiddlewares.httperror] (PID: 26) INFO: Ignoring response <403 https://www.fastenal.com/product/details/0110830>: HTTP status code is not handled or not allowed
2025-03-21 04:59:14 [scrapy.spidermiddlewares.httperror] (PID: 26) INFO: Ignoring response <403 https://www.fastenal.com/product/details/0110833>: HTTP status code is not handled or not allowed
2025-03-21 04:59:14 [scrapy.spidermiddlewares.httperror] (PID: 26) INFO: Ignoring response <403 https://www.fastenal.com/product/details/0110832>: HTTP status code is not handled or not allowed
2025-03-21 04:59:14 [scrapy.spidermiddlewares.httperror] (PID: 26) INFO: Ignoring response <403 https://www.fastenal.com/product/details/0110834>: HTTP status code is not handled or not allowed
2025-03-21 04:59:14 [scrapy.spidermiddlewares.httperror] (PID: 26) INFO: Ignoring response <403 https://www.fastenal.com/product/details/0110844>: HTTP status code is not handled or not allowed
2025-03-21 04:59:14 [scrapy.spidermiddlewares.httperror] (PID: 26) INFO: Ignoring response <403 https://www.fastenal.com/product/details/0110922>: HTTP status code is not handled or not allowed
2025-03-21 04:59:14 [scrapy.spidermiddlewares.httperror] (PID: 26) INFO: Ignoring response <403 https://www.fastenal.com/product/details/0110961>: HTTP status code is not handled or not allowed
2025-03-21 04:59:14 [scrapy.spidermiddlewares.httperror] (PID: 26) INFO: Ignoring response <403 https://www.fastenal.com/product/details/0110963>: HTTP status code is not handled or not allowed
2025-03-21 04:59:14 [scrapy.spidermiddlewares.httperror] (PID: 26) INFO: Ignoring response <403 https://www.fastenal.com/product/details/0111032>: HTTP status code is not handled or not allowed
2025-03-21 04:59:14 [scrapy.spidermiddlewares.httperror] (PID: 26) INFO: Ignoring response <403 https://www.fastenal.com/product/details/0111291>: HTTP status code is not handled or not allowed
2025-03-21 04:59:14 [scrapy.spidermiddlewares.httperror] (PID: 26) INFO: Ignoring response <403 https://www.fastenal.com/product/details/0111037>: HTTP status code is not handled or not allowed
2025-03-21 04:59:14 [scrapy.spidermiddlewares.httperror] (PID: 26) INFO: Ignoring response <403 https://www.fastenal.com/product/details/01120823>: HTTP status code is not handled or not allowed
2025-03-21 04:59:14 [scrapy.spidermiddlewares.httperror] (PID: 26) INFO: Ignoring response <403 https://www.fastenal.com/product/details/01120826>: HTTP status code is not handled or not allowed
2025-03-21 04:59:14 [scrapy.spidermiddlewares.httperror] (PID: 26) INFO: Ignoring response <403 https://www.fastenal.com/product/details/01120838>: HTTP status code is not handled or not allowed
2025-03-21 04:59:14 [scrapy.spidermiddlewares.httperror] (PID: 26) INFO: Ignoring response <403 https://www.fastenal.com/product/details/01120828>: HTTP status code is not handled or not allowed
2025-03-21 04:59:14 [scrapy.spidermiddlewares.httperror] (PID: 26) INFO: Ignoring response <403 https://www.fastenal.com/product/details/01120849>: HTTP status code is not handled or not allowed
2025-03-21 04:59:14 [scrapy.spidermiddlewares.httperror] (PID: 26) INFO: Ignoring response <403 https://www.fastenal.com/product/details/01120872>: HTTP status code is not handled or not allowed
2025-03-21 04:59:14 [scrapy.spidermiddlewares.httperror] (PID: 26) INFO: Ignoring response <403 https://www.fastenal.com/product/details/01120876>: HTTP status code is not handled or not allowed
2025-03-21 04:59:14 [scrapy.spidermiddlewares.httperror] (PID: 26) INFO: Ignoring response <403 https://www.fastenal.com/product/details/01120884>: HTTP status code is not handled or not allowed
2025-03-21 04:59:14 [scrapy.spidermiddlewares.httperror] (PID: 26) INFO: Ignoring response <403 https://www.fastenal.com/product/details/01120926>: HTTP status code is not handled or not allowed
2025-03-21 04:59:14 [scrapy.spidermiddlewares.httperror] (PID: 26) INFO: Ignoring response <403 https://www.fastenal.com/product/details/0115003>: HTTP status code is not handled or not allowed
2025-03-21 04:59:14 [scrapy.spidermiddlewares.httperror] (PID: 26) INFO: Ignoring response <403 https://www.fastenal.com/product/details/0115001>: HTTP status code is not handled or not allowed
2025-03-21 04:59:14 [scrapy.spidermiddlewares.httperror] (PID: 26) INFO: Ignoring response <403 https://www.fastenal.com/product/details/0115005>: HTTP status code is not handled or not allowed
2025-03-21 04:59:14 [scrapy.spidermiddlewares.httperror] (PID: 26) INFO: Ignoring response <403 https://www.fastenal.com/product/details/0115007>: HTTP status code is not handled or not allowed
2025-03-21 04:59:14 [scrapy.spidermiddlewares.httperror] (PID: 26) INFO: Ignoring response <403 https://www.fastenal.com/product/details/0115009>: HTTP status code is not handled or not allowed
2025-03-21 04:59:14 [scrapy.spidermiddlewares.httperror] (PID: 26) INFO: Ignoring response <403 https://www.fastenal.com/product/details/0115012>: HTTP status code is not handled or not allowed
2025-03-21 04:59:14 [scrapy.spidermiddlewares.httperror] (PID: 26) INFO: Ignoring response <403 https://www.fastenal.com/product/details/0115011>: HTTP status code is not handled or not allowed
2025-03-21 04:59:14 [scrapy.spidermiddlewares.httperror] (PID: 26) INFO: Ignoring response <403 https://www.fastenal.com/product/details/0115010>: HTTP status code is not handled or not allowed
2025-03-21 04:59:14 [scrapy.spidermiddlewares.httperror] (PID: 26) INFO: Ignoring response <403 https://www.fastenal.com/product/details/0115013>: HTTP status code is not handled or not allowed
2025-03-21 04:59:14 [scrapy.spidermiddlewares.httperror] (PID: 26) INFO: Ignoring response <403 https://www.fastenal.com/product/details/0115014>: HTTP status code is not handled or not allowed
2025-03-21 04:59:14 [scrapy.spidermiddlewares.httperror] (PID: 26) INFO: Ignoring response <403 https://www.fastenal.com/product/details/0115015>: HTTP status code is not handled or not allowed
2025-03-21 04:59:14 [scrapy.spidermiddlewares.httperror] (PID: 26) INFO: Ignoring response <403 https://www.fastenal.com/product/details/0115016>: HTTP status code is not handled or not allowed
2025-03-21 04:59:14 [scrapy.spidermiddlewares.httperror] (PID: 26) INFO: Ignoring response <403 https://www.fastenal.com/product/details/0115017>: HTTP status code is not handled or not allowed
2025-03-21 04:59:14 [scrapy.spidermiddlewares.httperror] (PID: 26) INFO: Ignoring response <403 https://www.fastenal.com/product/details/0115019>: HTTP status code is not handled or not allowed
2025-03-21 04:59:14 [scrapy.spidermiddlewares.httperror] (PID: 26) INFO: Ignoring response <403 https://www.fastenal.com/product/details/0115021>: HTTP status code is not handled or not allowed
2025-03-21 04:59:14 [scrapy.spidermiddlewares.httperror] (PID: 26) INFO: Ignoring response <403 https://www.fastenal.com/product/details/0115023>: HTTP status code is not handled or not allowed
2025-03-21 04:59:14 [scrapy.spidermiddlewares.httperror] (PID: 26) INFO: Ignoring response <403 https://www.fastenal.com/product/details/0115025>: HTTP status code is not handled or not allowed
2025-03-21 04:59:14 [scrapy.spidermiddlewares.httperror] (PID: 26) INFO: Ignoring response <403 https://www.fastenal.com/product/details/0115024>: HTTP status code is not handled or not allowed
2025-03-21 04:59:14 [scrapy.spidermiddlewares.httperror] (PID: 26) INFO: Ignoring response <403 https://www.fastenal.com/product/details/0115051>: HTTP status code is not handled or not allowed
2025-03-21 04:59:14 [scrapy.spidermiddlewares.httperror] (PID: 26) INFO: Ignoring response <403 https://www.fastenal.com/product/details/0115055>: HTTP status code is not handled or not allowed
2025-03-21 04:59:14 [scrapy.spidermiddlewares.httperror] (PID: 26) INFO: Ignoring response <403 https://www.fastenal.com/product/details/0115053>: HTTP status code is not handled or not allowed
2025-03-21 04:59:14 [scrapy.spidermiddlewares.httperror] (PID: 26) INFO: Ignoring response <403 https://www.fastenal.com/product/details/0115057>: HTTP status code is not handled or not allowed
2025-03-21 04:59:14 [scrapy.spidermiddlewares.httperror] (PID: 26) INFO: Ignoring response <403 https://www.fastenal.com/product/details/0115059>: HTTP status code is not handled or not allowed
2025-03-21 04:59:14 [scrapy.spidermiddlewares.httperror] (PID: 26) INFO: Ignoring response <403 https://www.fastenal.com/product/details/0115060>: HTTP status code is not handled or not allowed
2025-03-21 04:59:14 [scrapy.spidermiddlewares.httperror] (PID: 26) INFO: Ignoring response <403 https://www.fastenal.com/product/details/0115061>: HTTP status code is not handled or not allowed
2025-03-21 04:59:14 [scrapy.spidermiddlewares.httperror] (PID: 26) INFO: Ignoring response <403 https://www.fastenal.com/product/details/0115062>: HTTP status code is not handled or not allowed
2025-03-21 04:59:14 [scrapy.spidermiddlewares.httperror] (PID: 26) INFO: Ignoring response <403 https://www.fastenal.com/product/details/0115063>: HTTP status code is not handled or not allowed
2025-03-21 04:59:14 [scrapy.spidermiddlewares.httperror] (PID: 26) INFO: Ignoring response <403 https://www.fastenal.com/product/details/0115065>: HTTP status code is not handled or not allowed
2025-03-21 04:59:14 [scrapy.spidermiddlewares.httperror] (PID: 26) INFO: Ignoring response <403 https://www.fastenal.com/product/details/0115064>: HTTP status code is not handled or not allowed
2025-03-21 04:59:14 [scrapy.spidermiddlewares.httperror] (PID: 26) INFO: Ignoring response <403 https://www.fastenal.com/product/details/0115067>: HTTP status code is not handled or not allowed
2025-03-21 04:59:14 [scrapy.spidermiddlewares.httperror] (PID: 26) INFO: Ignoring response <403 https://www.fastenal.com/product/details/0115066>: HTTP status code is not handled or not allowed
2025-03-21 04:59:14 [scrapy.spidermiddlewares.httperror] (PID: 26) INFO: Ignoring response <403 https://www.fastenal.com/product/details/0115069>: HTTP status code is not handled or not allowed
2025-03-21 04:59:14 [scrapy.spidermiddlewares.httperror] (PID: 26) INFO: Ignoring response <403 https://www.fastenal.com/product/details/0115071>: HTTP status code is not handled or not allowed
2025-03-21 04:59:14 [scrapy.spidermiddlewares.httperror] (PID: 26) INFO: Ignoring response <403 https://www.fastenal.com/product/details/0115073>: HTTP status code is not handled or not allowed
2025-03-21 04:59:15 [scrapy.spidermiddlewares.httperror] (PID: 26) INFO: Ignoring response <403 https://www.fastenal.com/product/details/0115075>: HTTP status code is not handled or not allowed
2025-03-21 04:59:15 [scrapy.spidermiddlewares.httperror] (PID: 26) INFO: Ignoring response <403 https://www.fastenal.com/product/details/0115074>: HTTP status code is not handled or not allowed
2025-03-21 04:59:15 [scrapy.spidermiddlewares.httperror] (PID: 26) INFO: Ignoring response <403 https://www.fastenal.com/product/details/0115101>: HTTP status code is not handled or not allowed
2025-03-21 04:59:15 [scrapy.spidermiddlewares.httperror] (PID: 26) INFO: Ignoring response <403 https://www.fastenal.com/product/details/0115110>: HTTP status code is not handled or not allowed
2025-03-21 04:59:15 [scrapy.spidermiddlewares.httperror] (PID: 26) INFO: Ignoring response <403 https://www.fastenal.com/product/details/0115103>: HTTP status code is not handled or not allowed
2025-03-21 04:59:15 [scrapy.spidermiddlewares.httperror] (PID: 26) INFO: Ignoring response <403 https://www.fastenal.com/product/details/0115109>: HTTP status code is not handled or not allowed
2025-03-21 04:59:15 [scrapy.spidermiddlewares.httperror] (PID: 26) INFO: Ignoring response <403 https://www.fastenal.com/product/details/0115107>: HTTP status code is not handled or not allowed
2025-03-21 04:59:15 [scrapy.spidermiddlewares.httperror] (PID: 26) INFO: Ignoring response <403 https://www.fastenal.com/product/details/0115105>: HTTP status code is not handled or not allowed
2025-03-21 04:59:15 [scrapy.spidermiddlewares.httperror] (PID: 26) INFO: Ignoring response <403 https://www.fastenal.com/product/details/0115112>: HTTP status code is not handled or not allowed
2025-03-21 04:59:15 [scrapy.spidermiddlewares.httperror] (PID: 26) INFO: Ignoring response <403 https://www.fastenal.com/product/details/0115113>: HTTP status code is not handled or not allowed
2025-03-21 04:59:15 [scrapy.spidermiddlewares.httperror] (PID: 26) INFO: Ignoring response <403 https://www.fastenal.com/product/details/0115111>: HTTP status code is not handled or not allowed
2025-03-21 04:59:15 [scrapy.spidermiddlewares.httperror] (PID: 26) INFO: Ignoring response <403 https://www.fastenal.com/product/details/0115115>: HTTP status code is not handled or not allowed
2025-03-21 04:59:15 [scrapy.spidermiddlewares.httperror] (PID: 26) INFO: Ignoring response <403 https://www.fastenal.com/product/details/0115118>: HTTP status code is not handled or not allowed
2025-03-21 04:59:15 [scrapy.spidermiddlewares.httperror] (PID: 26) INFO: Ignoring response <403 https://www.fastenal.com/product/details/0115117>: HTTP status code is not handled or not allowed
2025-03-21 04:59:15 [scrapy.spidermiddlewares.httperror] (PID: 26) INFO: Ignoring response <403 https://www.fastenal.com/product/details/0115114>: HTTP status code is not handled or not allowed
2025-03-21 04:59:15 [scrapy.spidermiddlewares.httperror] (PID: 26) INFO: Ignoring response <403 https://www.fastenal.com/product/details/0115116>: HTTP status code is not handled or not allowed
2025-03-21 04:59:15 [scrapy.spidermiddlewares.httperror] (PID: 26) INFO: Ignoring response <403 https://www.fastenal.com/product/details/0115119>: HTTP status code is not handled or not allowed
2025-03-21 04:59:15 [scrapy.spidermiddlewares.httperror] (PID: 26) INFO: Ignoring response <403 https://www.fastenal.com/product/details/0115123>: HTTP status code is not handled or not allowed
2025-03-21 04:59:15 [scrapy.spidermiddlewares.httperror] (PID: 26) INFO: Ignoring response <403 https://www.fastenal.com/product/details/0115121>: HTTP status code is not handled or not allowed
2025-03-21 04:59:15 [scrapy.spidermiddlewares.httperror] (PID: 26) INFO: Ignoring response <403 https://www.fastenal.com/product/details/0115124>: HTTP status code is not handled or not allowed
2025-03-21 04:59:15 [scrapy.spidermiddlewares.httperror] (PID: 26) INFO: Ignoring response <403 https://www.fastenal.com/product/details/0115153>: HTTP status code is not handled or not allowed
2025-03-21 04:59:15 [scrapy.spidermiddlewares.httperror] (PID: 26) INFO: Ignoring response <403 https://www.fastenal.com/product/details/0115125>: HTTP status code is not handled or not allowed
2025-03-21 04:59:15 [scrapy.core.engine] (PID: 26) INFO: Closing spider (finished)
2025-03-21 04:59:15 [fastenal] (PID: 26) INFO: [Spidermon] ------------------------------ MONITORS ------------------------------
2025-03-21 04:59:15 [fastenal] (PID: 26) INFO: [Spidermon] Extracted Items Monitor/test_stat_monitor... FAIL
2025-03-21 04:59:15 [fastenal] (PID: 26) INFO: [Spidermon] Item Validation Monitor/test_stat_monitor... SKIPPED (Unable to find 'spidermon/validation/fields/errors' in job stats.)
2025-03-21 04:59:15 [fastenal] (PID: 26) INFO: [Spidermon] Error Count Monitor/test_stat_monitor... SKIPPED (Unable to find 'log_count/ERROR' in job stats.)
2025-03-21 04:59:15 [fastenal] (PID: 26) INFO: [Spidermon] Warning Count Monitor/test_stat_monitor... OK
2025-03-21 04:59:15 [fastenal] (PID: 26) INFO: [Spidermon] Finish Reason Monitor/Should have the expected finished reason(s)... OK
2025-03-21 04:59:15 [fastenal] (PID: 26) INFO: [Spidermon] Unwanted HTTP codes monitor/Should not hit the limit of unwanted http status... OK
2025-03-21 04:59:15 [fastenal] (PID: 26) INFO: [Spidermon] Field Coverage Monitor/test_check_if_field_coverage_rules_are_met... FAIL
2025-03-21 04:59:15 [fastenal] (PID: 26) INFO: [Spidermon] Retry Count monitor/Should not hit the limit of requests that reached the maximum retry amount... OK
2025-03-21 04:59:15 [fastenal] (PID: 26) INFO: [Spidermon] Downloader Exceptions monitor/test_stat_monitor... SKIPPED (Unable to find 'downloader/exception_count' in job stats.)
2025-03-21 04:59:15 [fastenal] (PID: 26) INFO: [Spidermon] Successful Requests monitor/Should have at least the minimum number of successful requests... OK
2025-03-21 04:59:15 [fastenal] (PID: 26) INFO: [Spidermon] Total Requests monitor/Should not hit the total limit of requests... OK
2025-03-21 04:59:15 [fastenal] (PID: 26) INFO: [Spidermon] ----------------------------------------------------------------------
2025-03-21 04:59:15 [fastenal] (PID: 26) ERROR: [Spidermon] 
======================================================================
FAIL: Extracted Items Monitor/test_stat_monitor
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/spidermon/contrib/scrapy/monitors/base.py", line 177, in test_stat_monitor
    self.fail(message)
AssertionError: Unable to find 'item_scraped_count' in job stats.

2025-03-21 04:59:15 [fastenal] (PID: 26) ERROR: [Spidermon] 
======================================================================
FAIL: Field Coverage Monitor/test_check_if_field_coverage_rules_are_met
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/spidermon/contrib/scrapy/monitors/monitors.py", line 477, in test_check_if_field_coverage_rules_are_met
    self.assertTrue(len(failures) == 0, msg=msg)
AssertionError: 
The following items did not meet field coverage rules:
dict/inStock (expected 1.0, got 0)
dict/name (expected 1.0, got 0)
dict/prices (expected 1.0, got 0)
dict/productStatus (expected 1.0, got 0)
dict/supplier (expected 1.0, got 0)
dict/supplierSku (expected 1.0, got 0)
dict/url (expected 1.0, got 0)

2025-03-21 04:59:15 [fastenal] (PID: 26) INFO: [Spidermon] 11 monitors in 0.010s
2025-03-21 04:59:15 [fastenal] (PID: 26) INFO: [Spidermon] FAILED (failures=2, skipped=3)
2025-03-21 04:59:15 [fastenal] (PID: 26) INFO: [Spidermon] -------------------------- FINISHED ACTIONS --------------------------
2025-03-21 04:59:15 [fastenal] (PID: 26) INFO: [Spidermon] ----------------------------------------------------------------------
2025-03-21 04:59:15 [fastenal] (PID: 26) INFO: [Spidermon] 0 actions in 0.000s
2025-03-21 04:59:15 [fastenal] (PID: 26) INFO: [Spidermon] OK
2025-03-21 04:59:15 [fastenal] (PID: 26) INFO: [Spidermon] --------------------------- PASSED ACTIONS ---------------------------
2025-03-21 04:59:15 [fastenal] (PID: 26) INFO: [Spidermon] ----------------------------------------------------------------------
2025-03-21 04:59:15 [fastenal] (PID: 26) INFO: [Spidermon] 0 actions in 0.000s
2025-03-21 04:59:15 [fastenal] (PID: 26) INFO: [Spidermon] OK
2025-03-21 04:59:15 [fastenal] (PID: 26) INFO: [Spidermon] --------------------------- FAILED ACTIONS ---------------------------
2025-03-21 04:59:15 [spidermon.contrib.actions.slack] (PID: 26) INFO:     :skull: `fastenal` *spider finished with errors!* _(errors=2)_

2025-03-21 04:59:15 [spidermon.contrib.actions.slack] (PID: 26) INFO: 
[
                {
        "text": "•  _Extracted Items Monitor/test_stat_monitor_: Unable to find 'item_scraped_count' in job stats.\n•  _Field Coverage Monitor/test_check_if_field_coverage_rules_are_met_: 
The following items did not meet field coverage rules:
dict/inStock (expected 1.0, got 0)
dict/name (expected 1.0, got 0)
dict/prices (expected 1.0, got 0)
dict/productStatus (expected 1.0, got 0)
dict/supplier (expected 1.0, got 0)
dict/supplierSku (expected 1.0, got 0)
dict/url (expected 1.0, got 0)\n",
        "color": "danger",
        "mrkdwn_in": ["text", "pretext"]
    }

,
]
2025-03-21 04:59:15 [fastenal] (PID: 26) INFO: [Spidermon] CustomTemplateSendSlackMessageSpiderFinished... OK
2025-03-21 04:59:15 [fastenal] (PID: 26) INFO: [Spidermon] ----------------------------------------------------------------------
2025-03-21 04:59:15 [fastenal] (PID: 26) INFO: [Spidermon] 1 action in 0.028s
2025-03-21 04:59:15 [fastenal] (PID: 26) INFO: [Spidermon] OK
2025-03-21 04:59:15 [scrapy.utils.signal] (PID: 26) ERROR: Error caught on signal handler: <bound method FeedExporter.close_spider of <scrapy.extensions.feedexport.FeedExporter object at 0x7fbd8801a6d0>>
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 1253, in adapt
    extracted: _SelfResultT | Failure = result.result()
                                        ^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/scrapy/extensions/feedexport.py", line 504, in close_spider
    self._close_slot(slot, spider)
  File "/usr/local/lib/python3.11/site-packages/scrapy/extensions/feedexport.py", line 535, in _close_slot
    d: Deferred[None] = maybeDeferred(slot.storage.store, get_file(slot))  # type: ignore[call-overload]
                                                          ^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/scrapy/extensions/feedexport.py", line 517, in get_file
    assert slot_.file
           ^^^^^^^^^^
AssertionError
2025-03-21 04:59:15 [scrapy.statscollectors] (PID: 26) INFO: Dumping Scrapy stats:
{'NotFoundHandlerSpiderMiddleware/HttpError': 100,
 'big_query/url': 100,
 'downloader/request_bytes': 30609,
 'downloader/request_count': 100,
 'downloader/request_method_count/GET': 100,
 'downloader/response_bytes': 61909,
 'downloader/response_count': 100,
 'downloader/response_status_count/403': 100,
 'elapsed_time_seconds': 7.85363,
 'finish_reason': 'finished',
 'finish_time': datetime.datetime(2025, 3, 21, 4, 59, 15, 198580, tzinfo=datetime.timezone.utc),
 'httperror/response_ignored_count': 100,
 'httperror/response_ignored_status_count/403': 100,
 'items_per_minute': None,
 'log_count/ERROR': 3,
 'log_count/INFO': 146,
 'log_count/WARNING': 2,
 'memusage/max': 122372096,
 'memusage/startup': 122372096,
 'poet/injector/catalog_extraction.pages.fastenal.FastenalProductsPageObject': 100,
 'response_received_count': 100,
 'responses_per_minute': None,
 'scheduler/dequeued': 100,
 'scheduler/dequeued/memory': 100,
 'scheduler/enqueued': 100,
 'scheduler/enqueued/memory': 100,
 'spidermon/validation/validators': 1,
 'spidermon/validation/validators/item/jsonschema': True,
 'start_requests/big_query': 100,
 'start_time': datetime.datetime(2025, 3, 21, 4, 59, 7, 344950, tzinfo=datetime.timezone.utc)}
2025-03-21 04:59:15 [scrapy.core.engine] (PID: 26) INFO: Spider closed (finished)
2025-03-21 04:59:15 [scrapy-playwright] (PID: 26) INFO: Closing download handler
2025-03-21 04:59:15 [scrapy-playwright] (PID: 26) INFO: Closing download handler