0

We are currently experiencing some connection issues when trying to log to the Google Stackdriver via our Laravel 6 application.

We use the official google/cloud-logging package within a custom Laravel logging channel for the logging with the following setup, which enables us to use the Laravel's native log methods (Log::info('...')):

// Within the `channels` array in `logging.php`
'googlelog' => [
            'driver'               => 'custom',
            'via'                  => CreateStackdriverLogger::class,
            'logName'              => 'api',
            'loggingClientOptions' => [
                'keyFilePath' => resource_path('google-service-account-prod.json'),
            ],
            'level'                => env('LOG_LEVEL', 'info'),
            'username'             => 'Logger'
        ],
use Monolog\Logger;


class CreateStackdriverLogger {

    /**
     * Create a custom Monolog instance.
     *
     * @param array $config
     * @return Logger
     */
    public function __invoke(array $config) {
        $projectId = $config['logName'] ?? '';
        $loggingClientOptions = $config['loggingClientOptions'] ?? [];
        $loggerOptions = $config['loggerOptions'] ?? [];
        $entryOptionsWrapper = $config['entryOptionsWrapper'] ?? 'stackdriver';
        $lineFormat = $config['lineFormat'] ?? '%message%';
        $level = $config['level'] ?? Logger::DEBUG;
        $bubble = $config['bubble'] ?? true;

        $stackdriverHandler = new StackdriverLogger($projectId, $loggingClientOptions, $loggerOptions, $entryOptionsWrapper, $lineFormat, $level, $bubble);

        return new Logger('stackdriver', [$stackdriverHandler]);
    }
}

use Google\Cloud\Logging\LoggingClient;
use Monolog\Formatter\LineFormatter;
use Monolog\Handler\AbstractProcessingHandler;
use Monolog\Logger;

class StackdriverLogger extends AbstractProcessingHandler {

    /**
     * The Stackdriver logger
     *
     * @var \Google\Cloud\Logging\Logger
     */
    private $logger;

    /**
     * A context array key used to take log entry options from
     *
     * @var string
     */
    private $entryOptionsWrapper;

    /**
     * Log entry options (all but severity) as supported by Google\Cloud\Logging\Logger::entry
     *
     * @var array Entry options.
     */
    private $entryOptions = [
        'resource',
        'httpRequest',
        'labels',
        'operation',
        'insertId',
        'timestamp',
    ];

    /**
     * @param string  $logName              Name of your log
     * @param array   $loggingClientOptions Google\Cloud\Logging\LoggingClient valid options
     * @param array   $loggerOptions        Google\Cloud\Logging\LoggingClient::logger valid options
     * @param string  $entryOptionsWrapper  Array key used in the context array to take Google\Cloud\Logging\Entry options from
     * @param string  $lineFormat           Monolog\Formatter\LineFormatter format
     * @param int     $level                The minimum logging level at which this handler will be triggered
     * @param Boolean $bubble               Whether the messages that are handled can bubble up the stack or not
     */
    public function __construct($logName, $loggingClientOptions, $loggerOptions = [], $entryOptionsWrapper = 'stackdriver', $lineFormat = '%message%', $level = Logger::DEBUG,
                                $bubble = true) {
        parent::__construct($level, $bubble);

        $this->logger = (new LoggingClient($loggingClientOptions))->logger($logName, $loggerOptions);
        $this->formatter = new LineFormatter($lineFormat);
        $this->entryOptionsWrapper = $entryOptionsWrapper;
    }

    /**
     * Writes the record down to the log
     *
     * @param array $record
     * @return void
     */
    protected function write(array $record): void {
        $options = $this->getOptionsFromRecord($record);

        $data = [
            'message' => $record['formatted'],
            'data'    => $record['context']
        ];

        $entry = $this->logger->entry($data, $options);

        $this->logger->write($entry);
    }

    /**
     * Get the Google\Cloud\Logging\Entry options
     *
     * @param array $record by reference
     * @return array $options
     */
    private function getOptionsFromRecord(array &$record) {
        $options = [
            'severity' => $record['level_name']
        ];

        if (isset($record['context'][$this->entryOptionsWrapper])) {
            foreach ($this->entryOptions as $entryOption) {
                if ($record['context'][$this->entryOptionsWrapper][$entryOption] ?? false) {
                    $options[$entryOption] = $record['context'][$this->entryOptionsWrapper][$entryOption];
                }
            }
            unset($record['context'][$this->entryOptionsWrapper]);
        }

        return $options;
    }
}

So the logging via this setup seems to work in the most cases, but sometimes we receive the following different errors at irregular intervals all in the context of the logging process :

cURL error 7: Failed to connect to logging.googleapis.com port 443: Connection timed out (see http://curl.haxx.se/libcurl/c/libcurl-errors.html)

cURL error 7: Failed to connect to oauth2.googleapis.com port 443: Connection timed out (see http://curl.haxx.se/libcurl/c/libcurl-errors.html)

cURL error 35: Unknown SSL protocol error in connection to oauth2.googleapis.com:443 (see http://curl.haxx.se/libcurl/c/libcurl-errors.html)

Here's an example for a full stacktrace: https://sentry.io/share/issue/7a25c0a3575e4e2684ad5220cd89b86a/

We already checked, if we are running against any rate limits of Google but that doesn't seem to be the case. Is there anything else that can cause these kind of connection issues?

Ferdinand Frank
  • 345
  • 4
  • 11
  • Can you verify that the service account you are using has the [Logs Writer](https://cloud.google.com/logging/docs/access-control#permissions_and_roles). If the service account is correct can you try the solutions in these two stackoverflow posts [1](https://stackoverflow.com/questions/28366402/failed-to-connect-to-www-googleapis-com-port-443-network-unreachable). [2](https://stackoverflow.com/questions/33451107/failed-to-connect-to-www-googleapis-com-port-443-network-is-unreachable). – Gustavo Mar 10 '20 at 15:46
  • Namely to whitelist the range of Google IP addresses, open port 443 if not open already, and add the appropriate curl option curl_setopt( $curl, CURLOPT_IPRESOLVE, CURL_IPRESOLVE_V4); as done in this [discussion](https://groups.google.com/forum/#!msg/adwords-api/OaJuGuIUg94/Z8AvSQIJAwAJ). as well. – Gustavo Mar 10 '20 at 15:52
  • @Gustavo Thanks for your response! The service account has the Logs Writer. We can't really modify the curl settings since the requests are made by the `google/cloud-logging` package. But the port and firewall settings are a good hint. Maybe a specific range of Google's IP addresses are blocked. I'll check that and get back into touch. – Ferdinand Frank Mar 11 '20 at 08:18
  • @Gustavo So the firewall settings are fine and no IP addresses are blocked. The weird thing is that the errors occur completely random and only in about 1% or even less of the logging requests. – Ferdinand Frank Mar 11 '20 at 09:58
  • If your firewall and ports are alright is it possible that your private key (google-service-account-prod.json) is not valid during those calls. You can also try reinstalling the logging agent. – Gustavo Mar 11 '20 at 17:35
  • @Gustavo The `google-service-account-prod.json` is always the same during the call and it's only happening in less than 1% of the logging requests. We'll close this issue internally now since it's not very urgent due to the low error rate. But we are still open for further ideas on how to solve the issue! :) – Ferdinand Frank Mar 17 '20 at 07:54

0 Answers0