How I Fixed 86 Unindexed Pages and Made My Site Google-Ready
The Problem
I checked Google Search Console one day and found something alarming: 86 pages were marked as "Discovered — currently not indexed." Google knew my pages existed but refused to index them.
The root cause? My authentication middleware was blocking Google's crawler from seeing any content. Every page required login, so Googlebot hit a login wall and bounced.
The Architecture Change
The fix required rethinking how auth works across the entire app:
Before
- Proxy middleware checked auth on every request
- Unauthenticated users got redirected to
/login - Googlebot got redirected to
/login - Result: 86 invisible pages
After
- Proxy allows unauthenticated page visits (for SEO crawlability)
- Auth gate moved to layout level using a client component
AuthGateWrapperusesusePathname()to check public paths on every navigation- Googlebot sees full page content, users see auth prompts where needed
// Layout checks auth via cookie (zero network cost)
const isAuthenticated = Boolean(getAalFromCookie());
// Client component gates protected content
<AuthGateWrapper isAuthenticated={isAuthenticated}>
{children}
</AuthGateWrapper>
SEO Metadata
Every page now has proper metadata:
- Dynamic
<title>and<meta description> - Open Graph images and descriptions
- JSON-LD breadcrumb structured data
- Sitemap (
/sitemap.xml) listing all public pages withlastModifieddates - robots.txt allowing all crawlers
Content Security Policy
Added CSP headers to protect against XSS while allowing necessary external resources:
| Directive | Allowed Sources |
|---|---|
script-src | self, Google Analytics |
connect-src | self, Supabase, GitHub API, OpenWeather |
img-src | self, GitHub avatars, QR server |
frame-src | none |
The Results
After deploying:
- All public pages are now crawlable
- Google Search Console shows pages moving from "Discovered" to "Indexed"
- Core Web Vitals improved (no unnecessary auth redirects)
- Zero security compromise — protected routes still require login
Key Takeaways
- Auth middleware shouldn't block crawlers — move auth checks to the UI layer
- Every page needs metadata — title, description, OG tags, structured data
- CSP headers matter — protect your users without breaking functionality
- Test with Google's URL Inspection tool — see exactly what Googlebot sees
If you're building a Next.js app with authentication, think about SEO from day one. Retrofitting it later (like I did) works, but it's a lot more effort.
Check out the platform at jayantgoyal.com — portfolio, 99+ dev tools, games, and more.
Related resources
- See the public developer tools sitemap surface that became crawlable after the auth and sitemap updates.
- Read the Jayant Goyal full-stack developer profile that now carries dedicated ProfilePage structured data.
- Browse all blog posts about the portfolio, SEO, and implementation work.