How do you Organize / Read a large amount of data?

Question

I have a large amount of data saved as a Data::Dumper output.

How in the world am I suppose to read this data? I would like to reorganize it but I'm completely lost to the approach. the data structures are hashes in arrays that are hashes of hashes...

Here is a (very trimmed down) example. Also the creation wasn't great as a character can have two "attacks" or two "specials" so obviously it's a collision and one will be overwritten.

EDIT: What I'm really asking is this: Is this an ideal way to store data like this? or is there a better way? because to me accessing the hash like $char_hash{Character Name}{abilities}{attack}{tiers}{level 1}{description} seems terrible to write. and iterating through things like @{$char_hash{Character Name}{Equipment}{Equipment Level 1}{Items}} seems crazy difficult

my @char_hash = (
"Character Name" => {
            "description" => "",
            "alignment" => "",
            "categories" => [
                "ex 1",
                "ex 2",
                "ex 4",
                "ex 5"
            ],
            "primaryStat" => "Strength (STR)",
            "baseStats" => {
                "Strength (STR)" => "22",
                "Agility (AGI)" => "15",
                "Intelligence (INT)" => "17",
                "Speed" => "100",
                "Health" => "197",
                "Physical Damage" => "17"
            },
            "abilities" => {
                "attack" => {
                    "name" => "ex 1",
                    "type" => "Physical",
                    "tiers" => {
                        "level 1" => {
                            "description" => ""
                        },
                        "level 2" => {
                            "unlockLevel" => 16,
                            "cost" => {
                                "Money" => 700,
                                "Material" => 3
                            },
                            "fromPrevious" => "+5% Damage",
                            "description" => ""
                        }
                    },
                    "conditions" => {
                    }
                },
                "special" => {
                    "name" => "ex",
                    "cooldown" => 3,
                    "type" => "special",
                    "tiers" => {
                        "level 1" => {
                            "description" => ""
                        },
                        "level 2" => {
                            "unlockLevel" => 18,
                            "cost" => {
                                "Money" => 1300,
                                "Material" => 2
                            },
                            "fromPrevious" => "+5% Damage",
                            "description" => ""
                        }
                    },
                    "conditions" => {
                    }
                },
            "Equipment" => {
                "Equipment Lvl I" => {
                    "cummulatedStats" => {
                        "Strength (STR)" => "+22",
                        "Agility (AGI)" => "+15",
                        "Intelligence (INT)" => "+17",
                        "Speed" => "+100",
                        "Health" => "+197",
                        "Physical Damage" => "+17"
                    },
                    "items" => [
                        {
                            "name" => "",
                            "id" => "",
                            "tier" => 1,
                            "mark" => "",
                            "requiredLevel" => 1,
                            "sellValue" => 10,
                            "stats" => {
                                "Physical Damage" => ""
                            }
                        },
                        {
                            "name" => "",
                            "id" => "",
                            "tier" => 1,
                            "mark" => "",
                            "requiredLevel" => 2,
                            "sellValue" => 20,
                            "stats" => {
                                "Strength (STR)" => "",
                                "Agility (AGI)" => "",
                                "Intelligence (INT)" => ""
                            }
                        },
                        {
                            "name" => "",
                            "id" => "",
                            "tier" => 1,
                            "mark" => "",
                            "requiredLevel" => 2,
                            "sellValue" => 20,
                            "stats" => {
                                "Strength (STR)" => "",
                                "Agility (AGI)" => "",
                                "Intelligence (INT)" => ""
                            }
                        },
                        {
                            "name" => "",
                            "id" => "",
                            "tier" => 1,
                            "mark" => "",
                            "requiredLevel" => 2,
                            "sellValue" => 20,
                            "stats" => {
                                "Speed" => ""
                            }
                        },
                        {
                            "name" => "",
                            "id" => "",
                            "tier" => 1,
                            "mark" => "",
                            "requiredLevel" => 2,
                            "sellValue" => 20,
                            "stats" => {
                                "Strength (STR)" => ""
                            }
                        },
                        {
                            "name" => "",
                            "id" => "",
                            "tier" => 1,
                            "mark" => "",
                            "requiredLevel" => 2,
                            "sellValue" => 20,
                            "stats" => {
                                "Armor" => ""
                            }
                        }
                    ]
                }
            }
        }
}
);

Answer 1

I would say yes, there's a better way.

And the answer is - use object oriented code. OO might sound intimidating if you've not really encountered it - and there's plenty of Java or C++ programmers that like to make it so.

But all it really is, is a data structure that includes code 'built in' to manipulate it. These bits of code are known as "methods" and apply to the object.

So - to take the above. You have "characters" and "equipment" as clear examples of 'objects'.

#!/usr/bin/env perl

package MyStuff::Character;

use strict;
use warnings;

sub new { 
   my ( $class, $name ) = @_;
   my $self = {}; 
   $self -> {name} = $name;
   bless $self, $class; 
   return $self; 
}

sub set_attr { 
   my ( $self, $attr, $value ) = @_; 
   $self -> {attr} -> {$attr} = $value;
}

sub get_attr { 
   my ( $self, $attr ) = @_;
   return $self -> {attr} -> {$attr}; 
}

sub get_name { 
   my ( $self ) = @_;
   return $self -> {name}; 
}

sub add_item {
   my ( $self, $item_ref ) = @_; 
   push ( @{ $self -> {items} }, $item_ref ); 
}

sub inventory {
  my ( $self ) = @_; 
  return @{$self->{items}};
}

package MyStuff::Items;

sub new {
    my ( $class, $name, $type ) = @_; 
    my $self = {};
    $self -> {name} = $name; 
    $self -> _set_type($type); 
    bless $self, $class;
    return $self; 
}

sub get_name {
   my ( $self ) = @_;
   return $self -> {name};
}

sub _set_type {
   my ( $self, $type ) = @_; 
   $self -> {type} = $type;
   if ( $type eq "sword" ) { 
       $self -> {attack_bonus} = "+10"; 
   }
}

package main; 

use strict;
use warnings;

my $character = MyStuff::Character -> new ( "Joe Beefcake" ); 
$character -> set_attr('STR', 9000); 

print $character -> get_name, " has STR ", $character -> get_attr('STR'),"\n";

my $new_sword = MyStuff::Character -> new ( "Hackmaster", "sword"); 
$character -> add_item( $new_sword ); 

print "And is carrying:\n";
foreach my $item ( $character -> inventory ) {
    print $item -> get_name,"\n";
}

This is a very basic example, but hopefully illustrates a new way of tackling complicated data structures?

Specifically - we 'hand off' things we don't care about, to the object to look after, and just use methods to interact with it. Because an item is a self contained object, and it 'knows' it's state. You could take it away, and 'give' it to another character.

The other advantage of doing it like this, is inheritance. I've got a very general 'item' object. Each item will have things you might want to do with any of them - pick them up, carry them, sell them, give them to another person.

But you could then make a 'weapon' class, which inherits the "item" class (and so you can still give someone else your sword) but also adds in extra stuff - like an attack bonus, a proficiency requirement, an attack bonus, etc.

The notion of Object Orientation is not a new one, but it's not too common in perl. There's a perldoc perlobj which has some of the basics.

There's also a few packages which assist the process (the above works standalone) like Moose.

For "saving" and "loading" - there's a number of possible options, and it depends a bit.

I would probably tackle it by serialising 'characters' and 'items' separately to JSON, and reload/validate.

That's covered in a bit more detail here: How to convert Perl objects into JSON and vice versa

Answer 2

Firstly, when I posted your data into a file and used the percent command in vi, I found what I believe is a pasting error. As it was, the 'abilities' hash had three entries including "Equipment". Looking at the indentation this very much looks like a mistake - I believe a closing curly is missing above the letter "E" in "Equipment" and a corresponding closing curly (third last in the file) needs to be removed. Then "Equipment" becomes a characteristic of a Character rather than a type of ability which seems more likely.

After correcting for that, what can you do? I don't think there's an easy way out - you have to comb through the data looking for repeated patterns and identifying groupings of data that can be candidates for objects.

Taking it from the top and ignoring the detail for now, we see a character is made up of seven attributes - perhaps not as bad as it first looks. I'm going to use perl6 as its more concise for class declarations and frankly, with something this big, any solution is a good solution (or more bluntly, I don't want to be here all day);

class Character {
  has $.description ;
  has $.alignment ;
  has $.categories ;
  has $.primaryStat ;
  has $.baseStats ;
  has Abilities $.abilities ;
  has %.equipment of Equipment ;
}

baseStats could be an object in its own right, but its a straight forward hash - so leave it. The idea here is to make sub-objects only when we need to due to the depth of the structure.

Looking at abilities, there are only two; attack abilities and special abilities. It would be better to take them out of their containing hash - its creating another level for the sake of two entries. They also appear to have the same attributes so, we can make an Abilities sub-object and contain two of them in our Character class - one for attack and one for special.

It looks as though tiers is a hash where the keys are "level n" and the values are hashes of level data. Its debatable but, its probably worth creating a class to represent a tier or level. So, that takes us three levels deep - again, perhaps not quite as bad as it first looks.

Equipment looks similar to tiers in that its a hash where the keys are levels and the values are hashes of data. That data is a straight forward hash of stuff under "cummulatedStats" and then a list of items. Again, its a judgement call, but I would say you need an Item sub-object. So we have;

#!/usr/bin/env perl6

class Tiers {
  has $.unlockLevel ;
  has $.cost ;
  has $.fromPrevious ;
  has $.description
}

class Abilities {
  has $.name ;
  has $.cooldown ;
  has $.type ;
  has %.tiers of Tiers ;
  has $.conditions ;
}

class Items {
  has $.name ;
  has $.id ;
  has $.tier ;
  has $.mark ;
  has $.requiredLevel ;
  has $.sellValue ;
  has $.stats ;
}

class Equipment {
  has $.cummulatedStats ;
  has $.items ;
}

class Character {
  has $.description ;
  has $.alignment ;
  has $.categories ;
  has $.primaryStat ;
  has $.baseStats ;
  has Abilities $.attack_abilities ;
  has Abilities $.special_abilities ;
  has %.equipment of Equipment ;
}

my %char_hash = hash.new(
            "description" => "",
            "alignment" => "",
   ... etc from your data ...

Now, we need to pull the Abilities out of their hash to create the sub objects. We also need to do the same with Tiers and Items. This post is already too long so I'll cut to the chase;

my %equipment = %char_hash<Equipment> :delete ;
my %abilities = %char_hash<abilities> :delete ;

my %special = %abilities<special> ;
my %attack  = %abilities<attack>  ;
%abilities = ();

for %special<tiers>.kv -> $level, $data_hash {
  %special<tiers>{ $level } = Tiers.new: |$data_hash
}
for %attack<tiers>.kv -> $level, $data_hash {
  %attack<tiers>{ $level } = Tiers.new: |$data_hash
}

%char_hash<special_abilities> = Abilities.new: |%special ;
%char_hash<attack_abilities>  = Abilities.new: |%attack ;

for %equipment.kv -> $level, $data_hash is rw {
  my @items;
  @items.push: Items.new(|$_) for $data_hash<items>.flat ;
  $data_hash<items> = @items ;
  %equipment{ $level } = Equipment.new: |$data_hash ;
}
%char_hash<equipment> = %equipment ;

my $char = Character.new( |%char_hash );
say $char.perl;

The approach is inside out - we start with tiers as they are nested the deepest. We create our hash of levels and then poke them back into the %special and %attack abilities hashes. From them we can create two Abilities objects and poke them back into the master hash, %char_hash.

Similarly with Items and then Equipment before we're finally ready to create our Character. Once you have the data loaded, you can create stringification methods, .Str, to customize the presentation of each type of object. It should also be easy to create special purpose methods for the manipulation and transformation you wanted to achieve.

... I should never have started on this question ;-)