Logging Googlebot Crawls for Free with Cloudflare Workers + D1

Introduction When doing SEO work, there are times when you need to investigate whether Googlebot is properly crawling your pages. Google Search Console has a crawl stats feature, but the sample URLs it surfaces are limited to 1,000 entries. For tracking the crawl status of specific pages over time, it falls a bit short. Server access logs are the ideal solution for this kind of investigation. I use this setup on LeapRows, a browser-based CSV tool I built on Vercel. On a self-managed VPS or on-premise server, Googlebot access is automatically recorded in Nginx or Apache logs. However, with serverless PaaS platforms like Vercel, there's no server management interface — which means no direct access to access logs. This is where Cloudflare comes in. By routing your domain's DNS through Cloudflare, you can intercept requests with a Cloudflare Worker before they ever reach Vercel. [Standard Vercel setup] Googlebot → Vercel → Response (no logs) [With Cloudflare] Googlebot → Cloudflare Worker