How to Set Up Vercel Log Drains Into Elasticsearch

Overview

Vercel Log Drains is a relatively new feature to arrive for Vercel users. As somebody who hosts almost all of their projects and websites (including this one) on Vercel, I was curious if I could get an ELK setup to view all of my web logs.

In this article you will learn how to:

• Create a Vercel Log Drain
• Configure Logstash to parse your Log Drain into events
• Configure Nginx to serve your Logstash endpoint
• Configure an Index Template in Elasticsearch to map the fields of Vercel event logs

By using Vercel Log Drains, you will be able to set up logging for all of your Vercel projects and view important logs such as:

• Static request logs - Requests to static assets like HTML and CSS files
• Edge logs - Output from Edge Functions like Middleware
• Lambda - Output from Vercel Functions like API Routes
• External - External rewrites to a different domain
• Build - Output from the Build Step

Prerequisites

There are a couple things you need to have set up already to follow along with this guide:

1. A VPS that you have root access to with:
   • Docker installed
   • Nginx installed
2. An existing Elasticsearch cluster. In this tutorial I assume it is on the same VPS as #1, but you can possibly host it elsewhere. Just know that you will have to adapt my guide to your specific situation.
3. A subdomain to send logs, which we will configure in Nginx later. Make this subdomain point to your VPS’s IP.
4. Finally, you will need to create a username/password combo to protect your Logstash endpoint with Basic Authentication. I used openssl rand -base64 32 to generate the password.

Create a Vercel Log Drain

The first step to this whole process is to create a Log Drain in Vercel for your Team. I use Vercel Pro, so I did this under my Pro team.

Navigate to Log Drains

1. Go to your Vercel dashboard.
2. Select the correct Team using the dropdown in the top-left.
3. Click Settings in the tab bar.
4. Click Log Drains in the left sidebar.

Create a simple Log Drain

Once you are here, you will see a menu to Add a Log Drain (beta).

Here are the settings I used to do this:

1. Sources: Select Static, Edge, Lambda. This will cover your API routes, middleware, and web requests.
2. Delivery Format: Select NDJSON. NDJSON is like normal JSON except you can send multiple objects by separating them with \n
3. Projects: I selected All Team Projects. I absolutely love that you can do this.
4. Custom Secret: Leave this section blank. I did not end up using this for anything.
5. Endpoint: Enter https://<your subdomain>/. This is the web endpoint that Vercel will send its NDJSON-formatted lists of events to.
   • Custom Headers: Check this, and add the following custom header: “Authorization: Basic [encoded username/password]” For the encoded username/password, use a tool such as this one. Input the username/password that you created during step 3 of the Prerequisites section.

You probably have noticed the “Verify URL ownership by responding with status code 200 and the following header” box. Before we can do this, we have to now set up Nginx and Logstash.

Setup Nginx to serve your subdomain

We want Nginx to serve requests sent to your subdomain using SSL and responding with the verification header that Vercel needs. We’ll use a simple site configuration in Nginx and then use certbot to add SSL to it. Then we will configure the verification header.

1. Create a new site configuration in Nginx:

sudo vim /etc/nginx/sites-available/logstash-endpoint.conf

Paste the following simple configuration into it:

server {
  listen 80;
  server_name <your subdomain>;

  location / {
    add_header x-vercel-verify <the verification string here>;
    proxy_pass http://localhost:9201; # This is the port I put Logstash on later in the guide.
  }
}

2. You will need to add this to your sites-enabled too:

sudo ln -s /etc/nginx/sites-available/logstash-endpoint.conf /etc/nginx/sites-enabled/logstash-endpoint.conf

3. Use Certbot to generate a new SSL certificate for your subdomain and apply it to the Nginx config.

sudo certbot

Follow the instructions, choosing your subdomain.

4. Reload Nginx:

sudo nginx -s reload

Setup Logstash using Docker Compose

Logstash is used in this setup to parse the log events sent to your webhook, and then pass them along to your Elasticsearch instance.

Docker compose setup

I set up Logstash using Docker Compose in the same Docker network as my Elastic and Kibana containers.

Create a folder called elk and place my docker-compose.yml in it.

# elk/docker-compose.yml
version: "3.6"
services:

  elastic:
    image: docker.elastic.co/elasticsearch/elasticsearch:8.10.2
    restart: unless-stopped
    container_name: elasticsearch
    ports:
     - 127.0.0.1:9200:9200
    volumes:
     - esdata01:/usr/share/elasticsearch/data
    environment:
      ...

  kibana:
    image: docker.elastic.co/kibana/kibana:8.10.2
    restart: unless-stopped
    container_name: kibana
    depends_on: [elastic]
    ports:
     - 127.0.0.1:5601:5601
    environment:
      ...

  logstash:
    image: docker.elastic.co/logstash/logstash:8.10.2
    restart: unless-stopped
    container_name: logstash
    depends_on: [elastic]
    ports:
     - 127.0.0.1:9201:9201
    volumes:
     - ./vercel-logs.conf:/usr/share/logstash/pipeline/vercel-logs.conf
    environment:
      XPACK_MONITORING_ENABLED: "false"
    command: ["bin/logstash", "-f", "pipeline/vercel-logs.conf"]

Some things to take note of here:

1. Because Logstash’s container is in the same network as the Elasticsearch container, we can refer to it simply with http://elasticsearch:9200 when making our Logstash pipeline.
2. We have mapped Logstash’s port 9201 to our 127.0.0.1:9201. This makes it possible for Nginx to route requests to http://localhost:9201 to serve our Logstash endpoint at our subdomain.
3. We are mapping ./vercel-logs.conf to the Logstash containers /usr/share/logstash/pipeline/vercel-logs.conf. In the next section we will write vercel-logs.conf
4. We run Logstash with XPACK_MONITORING_ENABLED: false, which helped me prevent some odd issues with authentication to the Elasticsearch cluster.
5. We have set the command of the Logstash container to bin/logstash -f pipeline/vercel-logs.conf. This isn’t totally necessary since we already put vercel-logs.conf into /usr/share/logstash/pipline but its nice to make the configuration clear.

Logstash pipeline setup

You will also need to create a Logstash pipeline configuration called vercel-logs.conf in the elk folder.

# elk/vercel-logs.conf
input {
  http {
    host => "0.0.0.0"
    port => "9201"
    codec => json_lines {
      target => "[vercel]"
    }
    # set up Basic Authentication on the pipeline endpoint to prevent
    # random people from sending us logs.
    user => "<username>"
    password => "<password>"
  }
}

filter {
  mutate {
    remove_field => ["event", "http", "user_agent", "host", "url"]
  }
}

output {
  stdout { codec => rubydebug }
  elasticsearch {
    # we are using a data stream for this.
    data_stream => "true"
    data_stream_type => "logs"
    data_stream_dataset => "vercel"

    hosts => ["http://elasticsearch:9200"]
    # used to authenticate to the Elasticsearch cluster
    user => "<elastic username>"
    password => "<elastic password>"
  }
}

Things to note here:

1. Our Logstash pipeline takes input from an http server, which is listening on 0.0.0.0:9201. Remember earlier we mapped this port to our VPS’s 127.0.0.1:9201. This allows Nginx to proxy requests to Logstash.
2. We set the codec of the http input to json_lines which is used to parse NDJSON formatted data. This is what we configured earlier when setting the Vercel Log Drain.
3. We configured the http server to be protected with Basic Authentication to prevent abuse. You should use the username and password combo here that you generated during the Prerequisites section.
4. We added a mutate filter that will remove the following fields that the http input automatically creates: event, http, user_agent, host, url. This is useful because these fields would be describing the request sent by Vercel to Logstash, which is pretty pointless to log.
5. We have two outputs:
   • stdout - You can comment this out once you’re done setting all of this up. This is just to allow you to view the Logstash ouput by using docker compose logs -f logstash
   • elasticsearch - This is what actually sends our events that Logstash parses to the Elasticsearch cluster.
     • We configured it to use a data stream when sending logs to elasticsearch. We will set this up in your Elasticsearch cluster soon.
     • We configured the host variable to be http://elasticsearch:9200. If you’re using a Docker compose file like mine this should work, otherwise you’ll need to engineer your own solution to reach the elasticsearch cluster.
     • user and password are NOT related to the ones we generated in the Prerequisites section. This is the username and password used to authenticate to the cluster and create logs in our data stream. Make sure the user you use here has proper permissions

Now that we have the Logstash pipeline configured, we can finally verify the endpoint back in the Vercel Log Drain interface.

1. Go back to the Vercel Log Drain interface
2. Press Verify. If it errors, verify that the URL is correct, that the response code it returns is 200, and that it returns the x-vercel-verify header.
3. Now test your log drain by clicking Test Log Drain. If you watch your Logstash containers logs using docker compose logs -f logstash you should see a rubydebug formatted event show up with fake data sent by Vercel.

At this point we have everything except for the configuration in Elasticsearch set up. Now let’s do that.

Set up our data stream and Index Template for Vercel logs in Elastic

Once we set this up, Logstash will be able to create logs in our data stream. Additionally, we will be able to see the types of each field in our documents.

1. Log into your Kibana interface as a user with superuser permissions.
2. Click the hamburger icon in the top left, and go to Stack Management.
3. In the sidebar, go to Index Management and then select the Index Templates tab.
4. Create a template using the Create template button to the right.
5. In Logistics, fill out the following:
   • Name: vercel-logs
   • Index patterns: logs-vercel-*
   • Data stream: Check “Create data stream”
   • Priority: 1000
6. Skip Component templates and Index settings
7. In Mappings we will load a JSON object I have prepared for the mappings. These mappings should cover the fields that Vercel will send you in their NDJSON-formatted logs.

{
  "properties": {
    "@version": {
      "type": "keyword"
    },
    "data_stream": {
      "properties": {
        "dataset": {
          "type": "keyword"
        },
        "namespace": {
          "type": "keyword"
        },
        "type": {
          "type": "keyword"
        },
      }
    },
    "vercel": {
      "properties": {
        "branch": {
          "type": "keyword"
        },
        "deploymentId": {
          "type": "keyword"
        },
        "environment": {
          "type": "keyword"
        },
        "host": {
          "type": "keyword"
        },
        "id": {
          "type": "keyword"
        },
        "path": {
          "type": "keyword"
        },
        "projectId": {
          "type": "keyword"
        },
        "projectName": {
          "type": "keyword"
        },
        "proxy": {
          "properties": {
            "cacheId": {
              "type": "keyword"
            },
            "clientIp": {
              "type": "ip"
            },
            "host": {
              "type": "keyword"
            },
            "method": {
              "type": "keyword"
            },
            "path": {
              "type": "keyword"
            },
            "pathType": {
              "type": "keyword"
            },
            "referer": {
              "type": "keyword"
            },
            "region": {
              "type": "keyword"
            },
            "scheme": {
              "type": "keyword"
            },
            "statusCode": {
              "type": "keyword"
            },
            "timestamp": {
              "type": "date"
            },
            "userAgent": {
              "type": "keyword"
            },
            "vercelCache": {
              "type": "keyword"
            },
            "vercelId": {
              "type": "keyword"
            }
          }
        },
        "requestId": {
          "type": "keyword"
        },
        "source": {
          "type": "keyword"
        },
        "timestamp": {
          "type": "date"
        }
      }
    }
  }
}

8. Skip Aliases, and go to Review Template
9. Click Create template.

Summary

Now you should be able to receive logs into your Elasticsearch cluster whenever someone goes to any of your team’s Vercel deployments! Your next step is most likely to create some useful visualizations, dashboards, and alerts.