2

I am trying to improve the build speed of one of my projects on CodeBuild. The project uses a Github source provider and source caching of type local is enabled.

The first time I ran the build it took 103 secs. I ran it again immediately after the first one finished expecting it to run in a few seconds due to source caching, but it took 60 secs.

What I am missing here? Is the cache not working? If it is working, why does it take that long on the second run?

Thanks

Project Details:

{
    "projectsNotFound": [],
    "projects": [
        {
            "environment": {
                "computeType": "BUILD_GENERAL1_LARGE",
                "imagePullCredentialsType": "SERVICE_ROLE",
                "privilegedMode": true,
                "image": "111669150171.dkr.ecr.us-east-1.amazonaws.com/***********/ep-build-env:latest",
                "environmentVariables": [
                    {
                        "type": "PLAINTEXT",
                        "name": "NEXUS_URI",
                        "value": "http://***************"
                    },
                    {
                        "type": "PLAINTEXT",
                        "name": "REGISTRY",
                        "value": "111669150171.dkr.ecr.us-east-1.amazonaws.com/*********"
                    }
                ],
                "type": "LINUX_CONTAINER"
            },
            "timeoutInMinutes": 60,
            "name": "StorefrontApi",
            "serviceRole": "arn:aws:iam::111669150171:role/CodeBuild-ECRReadOnly",
            "tags": [],
            "artifacts": {
                "type": "NO_ARTIFACTS"
            },
            "lastModified": 1571227097.581,
            "cache": {
                "type": "LOCAL",
                "modes": [
                    "LOCAL_DOCKER_LAYER_CACHE",
                    "LOCAL_SOURCE_CACHE",
                    "LOCAL_CUSTOM_CACHE"
                ]
            },
            "vpcConfig": {
                "subnets": [
                    "subnet-fd7f958b"
                ],
                "vpcId": "vpc-71e3f414",
                "securityGroupIds": [
                    "sg-19b65e6c",
                    "sg-9e28e9f9"
                ]
            },
            "created": 1571082681.262,
            "sourceVersion": "refs/heads/ep-mysql",
            "source": {
                "buildspec": "version: 0.2\n\nphases:\n  build:\n    commands:\n      - env\n      - cd extensions\n      - mvn --settings $CODEBUILD_SRC_DIR_DEVOPS_WINE/pipelines/storefront/build-war/settings.xml --projects storefront/ext-storefront-webapp -am -DskipAllTests clean install\n\nartifacts:\n  secondary-artifacts:\n    storefront-war:\n      base-directory: $CODEBUILD_SRC_DIR/extensions/storefront/ext-storefront-webapp/target\n      files:\n        - \"*.war\"\n\ncache:\n  paths:\n    - '/root/.m2/**/*'\n    - '/root/.npm/**/*'",
                "insecureSsl": false,
                "gitSubmodulesConfig": {
                    "fetchSubmodules": false
                },
                "location": "https://github.com/*****************.git",
                "gitCloneDepth": 1,
                "type": "GITHUB",
                "reportBuildStatus": false
            },
            "badge": {
                "badgeEnabled": false
            },
            "queuedTimeoutInMinutes": 480,
            "secondaryArtifacts": [],
            "logsConfig": {
                "s3Logs": {
                    "status": "DISABLED",
                    "encryptionDisabled": false
                },
                "cloudWatchLogs": {
                    "status": "ENABLED"
                }
            },
            "secondarySources": [
                {
                    "insecureSsl": false,
                    "gitSubmodulesConfig": {
                        "fetchSubmodules": false
                    },
                    "location": "https://github.com/*****************.git",
                    "sourceIdentifier": "DEVOPS_WINE",
                    "gitCloneDepth": 1,
                    "type": "GITHUB",
                    "reportBuildStatus": false
                }
            ],
            "encryptionKey": "arn:aws:kms:us-east-1:111669150171:alias/aws/s3",
            "arn": "arn:aws:codebuild:us-east-1:111669150171:project/StorefrontApi",
            "secondarySourceVersions": [
                {
                    "sourceVersion": "refs/heads/staging",
                    "sourceIdentifier": "DEVOPS_WINE"
                }
            ]
        }
    ]
}
Rodolfo Picoreti
  • 443
  • 3
  • 15
  • Could you provide more details about your build project? For example, is the privileged mode enabled on it? We'll ideally need your buildARN, which we (CodeBuild service team) can determine when you post on AWS Forums for CodeBuild @ https://forums.aws.amazon.com/forum.jspa?forumID=230. – Subin Mathew Oct 17 '19 at 03:34
  • Hi @SubinMathew, thanks for the answer, I updated the question with my project configuration. Do you also want me to post this question in the AWS forum? – Rodolfo Picoreti Oct 17 '19 at 11:19
  • Yes, posting your question on AWS forum would be helpful in this case, as we will need specific information, which we can send you a private message for. As an experiment, could you please try removing custom_cache, docker_layer_cache, and set privileged mode to false in your CodeBuild project? Meaning, what happens if you just keep source caching on? Also, I see you have secondary sources, we'll need to check if that introduces any delays. – Subin Mathew Oct 17 '19 at 13:00

1 Answers1

1

Apparently, at time of writing, CodeBuild does not use the native git client to fetch the source from GitHub. I understand that the CodeBuild internal teams have an internal feature request to move from whatever they're using to the native git client to improve performance.

Does your repository, by change, have lots of large files in its history? You can use this answer for a command to run to analyze your repository.

If you have lots of large files in your history and you're able to remove them, you can then use a tool like BFG Repo Cleaner to rewrite history. That should speed up the DOWNLOAD_SOURCE phase.

Also, if you have a dedicated support plan with AWS, you should reach out to your TAM to upvote the feature request to move to native git for GitHub source downloads.

blimmer
  • 2,038
  • 20
  • 23