1

I try to use GeoLite2-City.mmdb in spring boot to get location data. To that effect, I created a bean like this in the configuration file:

        @Bean   
        public static synchronized DatabaseReader mmdbReader() throws IOException { 

               if ( reader != null ) {
                   return reader;
               }

                ResourceLoader resourceLoader = new DefaultResourceLoader();

                Resource resource = resourceLoader.getResource("classpath:static/mmdb/GeoLite2-City.mmdb");
                InputStream database = resource.getInputStream();

               reader = new DatabaseReader.Builder(database).fileMode(com.maxmind.db.Reader.FileMode.MEMORY)
                       .withCache(new CHMCache()).build(); 

           return reader;
        }

I also created a util function to retrieve the location like this:

     public static GeoLocation getLocation(HttpServletRequest request, DatabaseReader databaseReader) throws IOException, GeoIp2Exception{

      if(request.getRemoteAddr().equals("0:0:0:0:0:0:0:1") || request.getRemoteAddr().equals("127.0.0.1")) {
          return new GeoLocation("localhost", "Redmond", "USA", 47.673988, -122.121513);
      }

      InetAddress ipAddress = InetAddress.getByName(request.getRemoteAddr());

      CityResponse response = databaseReader.city(ipAddress);

      Country country = response.getCountry();
      City city = response.getCity();
      Location location = response.getLocation();

      return new GeoLocation(ipAddress.toString(), city.getName(), country.getName(), location.getLatitude(), location.getLongitude());
      }

However, every time I deployed the app I get this error


 Caused by: java.lang.OutOfMemoryError: Java heap space
    at java.util.Arrays.copyOf(Arrays.java:3236)
    at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:118)
    at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
    at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153)
    at com.maxmind.db.BufferHolder.<init>(BufferHolder.java:64)
    at com.maxmind.db.Reader.<init>(Reader.java:89)
    at com.maxmind.geoip2.DatabaseReader.<init>(DatabaseReader.java:64)
    at com.maxmind.geoip2.DatabaseReader.<init>(DatabaseReader.java:54)

    at com.maxmind.geoip2.DatabaseReader$Builder.build(DatabaseReader.java:160)
    at com.microsoft.apie.configuration.ApplicationConfiguration.mmdbReader(ApplicationConfiguration.java:104)

My java settings are:

  -Xms512m -Xmx2048m

Can you please assist?

amelongo
  • 93
  • 11
  • 1
    How large is that file now? According to https://stackoverflow.com/questions/37483661/whats-the-size-of-the-maxmind-mmdb-binary-geoip2-databases it was rather small, but that was years ago. – tevemadar Jan 02 '20 at 00:12
  • @mentallurg, the DatabaseReader.java and Reader.java and BufferHolder.java are part of the maxmind library at: https://github.com/maxmind/GeoIP2-java/blob/master/src/main/java/com/maxmind/geoip2/DatabaseReader.java – amelongo Jan 02 '20 at 00:18
  • @tevemadar, the file size is 63,517,885 bytes; so roughly 65MB – amelongo Jan 02 '20 at 00:21
  • 1
    You should use default fileMode which is MEMORY_MAPPED – Strelok Jan 02 '20 at 00:36
  • @Strelok, it didn't work. I get this error: Caused by: java.lang.IllegalArgumentException: Only FileMode.MEMORY is supported when using an InputStream – amelongo Jan 02 '20 at 03:52
  • Well it's pretty obvious, don't use the constructor that takes InputStream, use the one that takes a File and read from file system. – Strelok Jan 02 '20 at 04:28
  • It's a catch 20/20 situation: a.) I can't use resource.getFile() because the app is packaged in a jar and thus throws a FileNotFoundException when using this option b.) And I can't use resource.getInputStream because the filemode MEMORY_MAPPED doesn't support that. You see my situation? – amelongo Jan 02 '20 at 04:41
  • Perhaps that `InputStream` is not clever enough, in terms of positioning or something. A silly workaround to try would be reading it into a `byte[]`, and calling the API with a `ByteArrayInputStream`. – tevemadar Jan 02 '20 at 08:47
  • Just take the file out of the jar – Strelok Jan 02 '20 at 14:42
  • The code provided actually works with GeoLite2-Country.mmdb which is a subset of GeoLite2-City.mmdb. I guess there's something datastructure used by maxmind that has some limitations in the data it can hold. I will investigate this further – amelongo Jan 02 '20 at 17:45

1 Answers1

2

I solved this issue by doing this:

a. I removed the file GeoLite2-City.mmdb from the jar and put in the same folder than the jar

b. I refactored the method and this:

    @Bean  
    public static synchronized DatabaseReader mmdbReader() throws IOException { 

              if ( reader != null ) {
                  return reader;
              }

               /*ResourceLoader resourceLoader = new DefaultResourceLoader();

               Resource resource = resourceLoader.getResource("classpath:static/mmdb/GeoLite2-Country.mmdb");

               InputStream database = resource.getInputStream();
              */
               FileSystemResource resource = new FileSystemResource("/path/to/file/GeoLite2-City.mmdb");

              reader = new DatabaseReader.Builder(resource.getFile()).fileMode(com.maxmind.db.Reader.FileMode.MEMORY_MAPPED)
                      .withCache(new CHMCache()).build(); 

          return reader;
   }

c. Repackaged the app and it works!

Thanks Strelok for this suggestion

amelongo
  • 93
  • 11