2

This question is NPM specific.

Few years ago I wrote a tool named qnp that downloads entire list of npm packages and then executes local queries very fast, like 0.2 second per query. This allows to perform a very interesting study of the modern programming world, filtering by author names, descriptions, tags, etc, doing hundreds of queries, inspecting results, analyzing, having ideas, doing more queries. An official client is good, but does not allow you to do very fast queries at the speed of thought. Here is my question:

About a year ago the location of the registry metadata DB of NPM was abandoned, now it returns an empty file. How can I download/fetch the entire list of metadata now? I need at least those fields: title/author/description/keywords/date. Optionally downloads count, dependencies list, version.

Here is the code that was working previously:

   var request  =  http.get({
         host: 'registry.npmjs.org',
         path: '/-/all/static/all.json',
         headers: {
            'Accept-Encoding': 'gzip, deflate'
         }
      }, function (a,b,c) {
      var done  =  0 ; var all  =  parseInt(a.headers['content-length'])
      a.on('data', function (a,b,c) {
         done += a.length
         process.stdout.write( '\r' + (done / (all/100)).toFixed(2)+'%  ' )
      })
      console.log('download started') 
      a.pipe(S)
      S.on('finish', function (a,b,c) {
         console.log('download complete') 
         S.close(f)
      })
   })
exebook
  • 32,014
  • 33
  • 141
  • 226
  • In this npm [blog](http://blog.npmjs.org/post/157615772423/deprecating-the-all-registry-endpoint) post it recommends using `/-/v1/search` and provides further details for that endpoint [here](https://github.com/npm/registry/blob/master/docs/REGISTRY-API.md). Also refer to the paragraph in the blog post where it encourages you to write a [registry follower](https://github.com/npm/registry/blob/master/docs/follower.md) as this may be pertinent to your requirement. – RobC Jan 12 '18 at 20:48
  • f.y.i - see this [answer](https://stackoverflow.com/questions/48251633/list-all-public-packages-in-the-npm-registry#answer-48272170) – RobC Jan 16 '18 at 10:22
  • My apologies; I missed this existing question when I posted my [duplicate](https://stackoverflow.com/q/48251633/1709587) yesterday. Since I've now got an answer there - linked to by @RobC - perhaps we should close this question as a duplicate of mine? – Mark Amery Jan 16 '18 at 10:33
  • Possible duplicate of [List all public packages in the npm registry](https://stackoverflow.com/questions/48251633/list-all-public-packages-in-the-npm-registry) – Mark Amery Jan 16 '18 at 10:34
  • See my answer [here](https://stackoverflow.com/a/57545387/5923666) – Raz Luvaton Sep 02 '19 at 12:29

1 Answers1

1

Since this post came up near the top when I searched for the answer, let me point out two packages that might be helpful for people landing on this old question:

Using the first one, I downloaded a list with 2.247.694 entries by using

pnpx all-the-package-names > ~/temp/all-the-package-names.txt

where pnpx is pnpm's equivalent to npx, the npm CLI runner (the latter installed with NodeJS).

John Frazer
  • 1,018
  • 13
  • 18