Get Directory Disk Usage with Golang

Question

I try to learn disk usage of directory with golang but this below app calculate incorrect. I do not want to use filepath.Walk. I tried os.Stat() but that did not get actual directory size. I shared related code statements below, correct directory information and results of app.

const (
    B  = 1
    KB = 1024 * B
    MB = 1024 * KB
    GB = 1024 * MB
)

type DiskStatus struct {
    All  uint64
    Free uint64
    Used uint64
}

func DiskUsage(path string) (ds DiskStatus) {
    fs := syscall.Statfs_t{}
    err := syscall.Statfs(path, &fs)
    if err != nil {
        return
    }
    ds.All = fs.Blocks * uint64(fs.Bsize)
    ds.Free = fs.Bfree * uint64(fs.Bsize)
    ds.Used = ds.All - ds.Free
    fmt.Println(path, ds.Used)
    return ds
}

func GetDir(destination_directory string) *[]model.Directory {
    var dir []model.Directory
    items, err := ioutil.ReadDir(destination_directory)

    if err != nil {
        log.Fatalln(err)
    }

    for _, item := range items {
        size := DiskUsage(destination_directory+item.Name()).Used / GB
        item := model.Directory{Path: item.Name(), TotalSize: size}
        dir = append(dir, item)
    }
    return &dir
}

Correct directory info:

$du -sh *
8.0K containers
2.0G testfolder
3.2M workspace

Results by App:

containers --> 90
testfolder --> 90
workspace --> 90

Thank you for veto question only, Mr. High Reputation!!! – swim Jul 18 '23 at 09:18 — swim, Jul 18 '23 at 09:18

VonC · Accepted Answer · 2023-07-18T12:57:00.147

I believe you are misusing the syscall.Statfs function: it returns file system statistics, not the size of individual directories or files. So, what you are seeing as output is not the size of individual directories, but the overall disk usage for the filesystem that these directories reside on.

If you do not want to use filepath.Walk (which is actually a good way to calculate the directory size by recursively going through all the files), you can write your own recursive function.

package main

import (
    "os"
    "log"
    "fmt"
)

const (
    B  = 1
    KB = 1024 * B
    MB = 1024 * KB
    GB = 1024 * MB
)

func DirSize(path string) (int64, error) {
    var size int64
    entries, err := os.ReadDir(path)
    if err != nil {
        return size, err
    }
    for _, entry := range entries {
        if entry.IsDir() {
            subDirSize, err := DirSize(path + "/" + entry.Name())
            if err != nil {
                log.Printf("failed to calculate size of directory %s: %v\n", entry.Name(), err)
                continue
            }
            size += subDirSize
        } else {
            fileInfo, err := entry.Info()
            if err != nil {
                log.Printf("failed to get info of file %s: %v\n", entry.Name(), err)
                continue
            }
            size += fileInfo.Size()
        }
    }
    return size, nil
}

func main() {
    dirs := []string{"./containers", "./testfolder", "./workspace"}

    for _, dir := range dirs {
        size, err := DirSize(dir)
        if err != nil {
            log.Printf("failed to calculate size of directory %s: %v\n", dir, err)
            continue
        }
        fmt.Printf("%s --> %.2f GB\n", dir, float64(size)/float64(GB))
    }
}

That would go through each directory and file under the given directory and sum up the sizes.
If the entry is a directory, it will recursively calculate the size of the directory.
Although... this code will not handle symbolic links or other special files.

The other option is for the DirSize function uses filepath.Walk to traverse all files under the specified directory, including subdirectories.
The filepath.Walk function takes two arguments: the root directory and a function that will be called for each file or directory in the tree rooted at the root.

Here, for each file it encounters, it adds the size of the file to the total size (ignoring directories).
Since filepath.Walk automatically handles recursive traversal for you, it is generally simpler and more straightforward than manually writing the recursive function.
(playground)

package main

import (
    "os"
    "log"
    "fmt"
    "path/filepath"
)

const (
    B  = 1
    KB = 1024 * B
    MB = 1024 * KB
    GB = 1024 * MB
)

func DirSize(path string) (int64, error) {
    var size int64
    err := filepath.Walk(path, func(_ string, info os.FileInfo, err error) error {
        if err != nil {
            return err
        }
        if !info.IsDir() {
            size += info.Size()
        }
        return err
    })
    return size, err
}

func main() {
    dirs := []string{"./containers", "./testfolder", "./workspace"}

    for _, dir := range dirs {
        size, err := DirSize(dir)
        if err != nil {
            log.Printf("failed to calculate size of directory %s: %v\n", dir, err)
            continue
        }
        fmt.Printf("%s --> %.2f GB\n", dir, float64(size)/float64(GB))
    }
}

Thank you for quick response. I got it. Your opinion is mainly we must to get all info from files for calculation all directory size. If I understood currectly, in additionaly there is not official function about disk usage. If you won't another response for me, firstly I change your code statements with `filestate.Walk()` version in my local and then I will sign to solved. — swim, Jul 18 '23 at 12:31
@swim I am not aware of `filestate.Walk()`, only `filepath.Walk()`. Do you want me to include an example using `filepath.Walk()`? — VonC, Jul 18 '23 at 12:44
Sorry my mistake :) No no problem but if you want to give a example, I'd appreciate it. — swim, Jul 18 '23 at 12:47
Actually, I was expecting that there is golang official function about disk usage. — swim, Jul 18 '23 at 12:54
@swim I have edited the answer with an example using `filepath.Walk()` — VonC, Jul 18 '23 at 12:57
@swim On "disk usage", see "[Get amount of free disk space using Go](https://stackoverflow.com/q/20108520/6309)". — VonC, Jul 18 '23 at 12:58

Get Directory Disk Usage with Golang

1 Answers1