Logging in AWS Lambda with LogFlow

AWS Lambda writes logs to CloudWatch by default. CloudWatch works, but searching across functions is painful, alerting is limited, and costs add up at scale ($0.50/GB ingestion + $0.03/GB storage/month).

This tutorial ships structured logs from Lambda directly to LogFlow via the HTTP API — no log shipper needed.

Prerequisites

• AWS Lambda function (Node.js 18+, Python 3.9+, or Go)
• A LogFlow account (free tier works)
• Your API key from Settings → API Key

Architecture

Lambda functions are ephemeral — they spin up, run, and die. There's no persistent process to batch logs. The approach:

1. Buffer logs during the function invocation
2. Flush to LogFlow at the end of each invocation
3. Include cold start detection and request context

Node.js Lambda

Step 1 — Create the Logger

Add a logger.js file to your Lambda package:

const https = require('https')

const LOGFLOW_API_KEY = process.env.LOGFLOW_API_KEY
const LOGFLOW_URL = 'https://api.getlogflow.com/v1/logs'
const SERVICE = process.env.SERVICE_NAME || 'lambda'

let isColdStart = true
const buffer = []

function log(level, message, extra = {}) {
  const entry = {
    timestamp: new Date().toISOString().replace('T', ' ').replace('Z', ''),
    level,
    message,
    service: SERVICE,
    ...extra,
  }
  buffer.push(entry)
  // Also print to stdout for CloudWatch
  console.log(JSON.stringify(entry))
}

function flush() {
  if (buffer.length === 0 || !LOGFLOW_API_KEY) return Promise.resolve()

  const batch = buffer.splice(0)
  const body = JSON.stringify(batch)

  return new Promise((resolve) => {
    const req = https.request(LOGFLOW_URL, {
      method: 'POST',
      headers: {
        'Authorization': `Bearer ${LOGFLOW_API_KEY}`,
        'Content-Type': 'application/json',
        'Content-Length': Buffer.byteLength(body),
      },
      timeout: 3000,
    }, (res) => {
      res.resume()
      resolve()
    })
    req.on('error', () => resolve()) // Never fail the Lambda
    req.on('timeout', () => { req.destroy(); resolve() })
    req.write(body)
    req.end()
  })
}

module.exports = {
  info: (msg, extra) => log('info', msg, extra),
  warn: (msg, extra) => log('warn', msg, extra),
  error: (msg, extra) => log('error', msg, extra),
  debug: (msg, extra) => log('debug', msg, extra),
  flush,
  isColdStart: () => {
    const was = isColdStart
    isColdStart = false
    return was
  },
}

Step 2 — Use in Your Handler

const logger = require('./logger')

exports.handler = async (event, context) => {
  const startTime = Date.now()
  const coldStart = logger.isColdStart()

  logger.info('invocation.start', {
    function_name: context.functionName,
    request_id: context.awsRequestId,
    cold_start: coldStart,
    memory_mb: context.memoryLimitInMB,
  })

  try {
    // Your business logic
    const result = await processEvent(event)

    logger.info('invocation.success', {
      request_id: context.awsRequestId,
      duration_ms: Date.now() - startTime,
      cold_start: coldStart,
    })

    // Flush logs before Lambda freezes
    await logger.flush()

    return {
      statusCode: 200,
      body: JSON.stringify(result),
    }
  } catch (error) {
    logger.error('invocation.failed', {
      request_id: context.awsRequestId,
      error: error.message,
      stack: error.stack,
      duration_ms: Date.now() - startTime,
    })

    await logger.flush()
    throw error
  }
}

Step 3 — Set Environment Variables

In your Lambda configuration (console, SAM, CDK, or Serverless Framework):

# serverless.yml
provider:
  environment:
    LOGFLOW_API_KEY: ${ssm:/logflow/api-key}
    SERVICE_NAME: my-api

# SAM template.yaml
Environment:
  Variables:
    LOGFLOW_API_KEY: !Sub '{{resolve:ssm:/logflow/api-key}}'
    SERVICE_NAME: my-api

Security: Store your API key in AWS SSM Parameter Store or Secrets Manager, not in plain text.

Python Lambda

import json
import os
import time
import urllib.request

LOGFLOW_API_KEY = os.environ.get("LOGFLOW_API_KEY", "")
LOGFLOW_URL = "https://api.getlogflow.com/v1/logs"
SERVICE = os.environ.get("SERVICE_NAME", "lambda")

_buffer = []
_is_cold_start = True


def log(level, message, **extra):
    entry = {
        "timestamp": time.strftime("%Y-%m-%d %H:%M:%S.000"),
        "level": level,
        "message": message,
        "service": SERVICE,
        **extra,
    }
    _buffer.append(entry)
    print(json.dumps(entry))


def flush():
    global _buffer
    if not _buffer or not LOGFLOW_API_KEY:
        return

    batch = _buffer[:]
    _buffer = []

    try:
        req = urllib.request.Request(
            LOGFLOW_URL,
            data=json.dumps(batch).encode(),
            headers={
                "Authorization": f"Bearer {LOGFLOW_API_KEY}",
                "Content-Type": "application/json",
            },
            method="POST",
        )
        urllib.request.urlopen(req, timeout=3)
    except Exception:
        pass


def handler(event, context):
    global _is_cold_start
    start = time.time()
    cold_start = _is_cold_start
    _is_cold_start = False

    log("info", "invocation.start",
        function_name=context.function_name,
        request_id=context.aws_request_id,
        cold_start=cold_start)

    try:
        result = process_event(event)

        log("info", "invocation.success",
            request_id=context.aws_request_id,
            duration_ms=round((time.time() - start) * 1000),
            cold_start=cold_start)
        flush()

        return {"statusCode": 200, "body": json.dumps(result)}

    except Exception as e:
        log("error", "invocation.failed",
            request_id=context.aws_request_id,
            error=str(e),
            duration_ms=round((time.time() - start) * 1000))
        flush()
        raise

Cold Start Tracking

Cold starts are logged as cold_start: true. In LogFlow, you can:

1. Search: cold_start:true service:my-api to find all cold starts
2. Alert: Set up a keyword alert on cold_start frequency
3. Dashboard: Build a widget showing cold start percentage over time

Best Practices for Lambda Logging

1. Always flush before returning — Lambda freezes the execution environment after the response. Unflushed logs are lost.

2. Use structured JSON — CloudWatch Insights can query JSON fields. LogFlow indexes them automatically.

3. Log cold starts — cold start tracking helps you tune memory allocation and provisioned concurrency.

4. Include request context — awsRequestId, function name, and memory limit should be on every log.

5. Don't log the full event — API Gateway events include headers, cookies, and potentially PII. Log specific fields, not JSON.stringify(event).

6. Keep log volume in check — Lambda at scale generates massive log volume. Use ingestion rules to sample or drop noisy debug logs.

Next Steps

• Set up alerts — get notified when Lambda errors spike
• Deployment markers — track Lambda deploys in your charts
• Monitor cron jobs — track scheduled Lambda invocations
• Docker logging — for containerized workloads
• Log sampling guide — reduce costs at high volume