18

I want to scrape the data from an ArcGIS map. The following map has a popup when we click the red features. How do I access that data programmatically?

Link : https://cslt.maps.arcgis.com/apps/MapSeries/index.html?appid=2c9f3e737cbf4f6faf2eb956fa26cdc5

Gary Sheppard
  • 4,764
  • 3
  • 25
  • 35
curious_nustian
  • 596
  • 2
  • 7
  • 22

1 Answers1

34

Note: Please respect the access and use constraints of any ArcGIS Online item you access. When in doubt, don't save a copy of someone else's data.

The ArcGIS Online REST interface makes it relatively simple to get the data behind ArcGIS Online items. You need to use an environment that can make HTTP requests and parse JSON text. Most current programming languages either have these capabilities built in or have libraries available with these capabilities.

Here's a general workflow that your code could follow.

  1. Use the app ID and the item data endpoint to see the app's JSON text:

    https://www.arcgis.com/sharing/rest/content/items/2c9f3e737cbf4f6faf2eb956fa26cdc5/data

  2. Search that text for webmap and see that the app uses the following web maps:

    • d2b4a98c39fd4587b99ac0878c420125
    • 7b1af1752c3a430184fbf7a530b5ec65
    • c6e9d07e4c2749e4bfe23999778a3153

  3. Look at the item data endpoint for any of those web maps:

    https://www.arcgis.com/sharing/rest/content/items/d2b4a98c39fd4587b99ac0878c420125/data

  4. The list of operationalLayers specifies the feature layer URLs from which you could harvest data. For example:

    https://services2.arcgis.com/gWRYLIS16mKUskSO/arcgis/rest/services/VHR_Areas/FeatureServer/0

  5. Then just run a query with a where of 0=0 (or whatever you want) and an outFields of *:

    https://services2.arcgis.com/gWRYLIS16mKUskSO/arcgis/rest/services/VHR_Areas/FeatureServer/0/query?where=0%3D0&outFields=%2A&f=json

    Use f=html instead if you want to see a human-readable request form and results.

    Note that feature services have a limit of how many features you can get per request, so you will probably want to filter by geometry or attribute values. Read the documentation to learn everything you can do with feature service queries.

Gary Sheppard
  • 4,764
  • 3
  • 25
  • 35
  • yes! i got your point regarding the data rights! Data was required for educational purposes with the permission of the city lake authorities! and thanks alot your answer did helped alot. – curious_nustian May 08 '18 at 05:00
  • 1
    I'm glad my answer helped. – Gary Sheppard May 08 '18 at 08:22
  • 1
    Is this answer still valid? I was looking for data sources [here](https://instnsp.maps.arcgis.com/apps/opsdashboard/index.html#/5eced796595b4ee585bcdba03e30c127). Found no `webmap`, just `mapWidget`, then `datasets.dataSource.id` – which looks like this: _1200f3f1-8f72-4ea6-af16-14f19e9a4517_ – it doesn't seem to work with _/sharing/rest/content/items//data_ requests – pax Mar 14 '20 at 19:28
  • 5
    Yes, this answer is valid. The item you're looking at is a dashboard, which is a different structure than the story map that the OP asked about. In the [dashboard item data](https://www.arcgis.com/sharing/rest/content/items/5eced796595b4ee585bcdba03e30c127/data), there are three `mapWidget` objects, each with an `itemId` that is a web map item ID. You can use any of those items' data ([here](https://www.arcgis.com/sharing/rest/content/items/bbf83b52ae774f2783e8f685d1d43b21/data), for example; the other two seem almost identical to me) and start at step 3 in this answer. – Gary Sheppard Mar 15 '20 at 20:36
  • 1
    This was incredibly helpful! – jlev Jun 10 '20 at 21:44
  • That's great that this answer is helping people, even if it was not marked as the accepted answer yet. – Gary Sheppard Feb 18 '21 at 16:42
  • is it possible to scrape the data of a map like this? - https://sig.sea.gob.cl/mapaLineasBaseEIA/ - it doesn't have an arcgis.com url, thanks – zevloo Mar 20 '21 at 22:58
  • 1
    I'm having some problems to retrieve data from a dashboard but this is REALLY helpful – Nacho R Feb 18 '22 at 19:50