Extract rendered Hacker News listing rows in displayed order, preserving null metadata, page-local ranks, opaque IDs, destinations, discussion URLs, timestamps, and full discussion post text. It supports numbered-page stitching with ID deduplication and reports available versus extracted counts.
Use Cases
- Return the current Hacker News front page in displayed order.
- Retrieve Ask HN, Show HN, newest, or other numbered listing pages.
- Deduplicate rows across a multi-page collection while retaining page-local ranks.
- Separate Ask HN, Show HN, ordinary, and job rows.
- Include discussion timestamps and full post text for each result.
Automation Flow
- Build direct URLs:
https://news.ycombinator.com/{view}for page 1 andhttps://news.ycombinator.com/{view}?p={page}for numbered pages. Use a rendered More-link URL when pagination is cursor-based. - Goto each requested URL and run this evaluator. For each page, record
available_rowsbefore deduplication. Merge page results by numericid, retaining the first occurrence and itspageand page-localrank; settotal_available_rowsto the sum of per-page available counts andtotal_extracted_rowsto the unique merged-row count.
(async()=>{const clean=s=>s==null?null:s.replace(/\s+/g,' ').trim()||null,abs=h=>h?new URL(h,location.href).href:null,isInternal=h=>{if(!h)return true;try{const u=new URL(h,location.href);return u.origin===location.origin&&(/^\/item(?:\?|$)/.test(u.pathname)||u.pathname==='/')}catch{return true}},rows=[...document.querySelectorAll('tr.athing.submission')],page=Number(new URL(location.href).searchParams.get('p')||1),base=rows.map((row,i)=>{const meta=row.nextElementSibling,titleEl=row.querySelector('.titleline>a'),title=clean(titleEl?.textContent),rawHref=titleEl?.getAttribute('href')||null,scoreText=clean(meta?.querySelector('.score')?.textContent),scoreMatch=(scoreText||'').match(/-?\d+/),authorEl=meta?.querySelector('.hnuser'),sub=meta?.querySelector('.subtext'),age=sub?.querySelector('.age'),commentLink=[...(sub?.querySelectorAll('a')||[])].find(a=>/comment|discuss/i.test(clean(a.textContent)||'')),commentMatch=(clean(commentLink?.textContent)||'').match(/\d+/),prefix=title?.match(/^(Ask HN|Show HN):/i)?.[1]?.toLowerCase(),type=prefix==='ask hn'?'ask_hn':prefix==='show hn'?'show_hn':!meta?.querySelector('.score')&&!authorEl?'job':'ordinary_story',id=Number(row.id)||null,discussion_url=id?`https://news.ycombinator.com/item?id=${id}`:null;return{page,rank:i+1,id,type,title,author:clean(authorEl?.textContent),score:scoreMatch?Number(scoreMatch[0]):null,comments:commentMatch?Number(commentMatch[0]):null,timestamp:age?.getAttribute('title')||null,age:clean(age?.textContent),destination_url:isInternal(rawHref)?null:abs(rawHref),raw_href:rawHref,internal_link:isInternal(rawHref),discussion_url,hn_url:discussion_url}}),detailed=await Promise.all(base.map(async r=>{if(!r.discussion_url)return{...r,full_post_text:null};try{const html=await fetch(r.discussion_url).then(x=>x.text()),doc=new DOMParser().parseFromString(html,'text/html');return{...r,full_post_text:doc.querySelector('.toptext')?.innerText?.trim()||null}}catch{return{...r,full_post_text:null}}})),groups={ordinary_story:[],ask_hn:[],show_hn:[],job:[]};for(const r of detailed)groups[r.type].push(r);const more=[...document.querySelectorAll('a')].find(a=>/^More$/i.test(clean(a.textContent)||''));return{page_url:location.href,page,available_rows:rows.length,extracted_rows:detailed.length,empty:rows.length===0,rows:detailed,groups,continuation_url:abs(more?.getAttribute('href'))}})()- For requested pages 1 through 5, navigate directly to
/ask,/ask?p=2,/ask?p=3,/ask?p=4, and/ask?p=5; do not infer rows from Firebase or search results. If a page has no submission rows, retain itsempty: trueresult and count zero available rows. If discussion fetching fails, navigate directly to that row'sdiscussion_urland rerun the detail extraction there.
Params
| Param | What it does | Example value |
|---|---|---|
view | Hacker News listing route | ask |
page | Numbered listing page passed as p | 5 |
pages | Requested page numbers to stitch and deduplicate | 1..5 |
cursor | Cursor from a rendered More link, passed as next | 49729611 |
limit | Cursor-page row limit, passed as n | 31 |
row-limit | Caller-side maximum after extraction | 30 |
Possible Friction Points
| Trigger | Action |
|---|---|
A numbered page contains zero tr.athing.submission rows | Return empty: true, count zero available rows, and do not infer rows from API data. |
The visible More link uses next and n rather than p | Extract continuation_url and navigate to that direct URL instead of clicking. |
A row has no .score and no .hnuser | Classify it as job and preserve null score and author. |
A visible comment link says discuss without a numeric count | Set comments to null rather than zero. |
A link-only discussion has no .toptext | Return full_post_text: null. |
| Discussion-page fetch is blocked or returns incomplete markup | Navigate to the row's discussion_url and run the same detail selectors on the loaded page. |
| A title link is an internal Hacker News item link or missing | Set destination_url to null, retain raw_href, and use discussion_url for the item. |
| Multiple pages contain the same ID | Deduplicate by numeric id while retaining the first page-local rank and page number. |
| More than one page is required | Follow each requested direct URL or returned continuation_url, append page results, and report both total available rows and unique extracted rows. |
Firebase topstories order is compared with the page | Use rendered tr.athing.submission order; it can differ from Firebase order. |