FunctionBeat Configuration "Gotcha's"

What happened?

So, I am in a pretty... edge case?... scenario. I work in AWS with a full WINDOWS SERVER Platform. Which means, that most anything I want to do, I have to use PowerShell.

THANKFULLY, I convinced them to let me use my Ubuntu Shell to do AWS CLi stuff over PowerShell, I can actually get some work done.

DO NOT USE if you have a Public (non-VPC)

Deploymment unless you want to create the Lambda function to have a Static IP address

Use this method instead: CW Log Stream to ES

Config Information

Okay, I will go through SOME of the functionbeat.yml file to point out a few things that are just not easily understood, or what have you.

MFA. Yes, you SHOULD have it, yes, it makes things a little harder...

Okay, if you are required to use MFA on your account, you all should know by now how to do it. In Linux, it's not hard, but if you look at the instructions given by Amazon, you just need to Export your settings. Which WORKS, WITHOUT MFA. For those of us that use MFA, even at the CLI, which you should, the solution is rather easy.

In your ~/.aws/credentials file, you can have several Profiles set up. Amazon states that it is best to have an MFA profile, however, when working with configuring ES Domain via CLI, make your MFA Profile [default]. Once you do this, there is no need to export your information to the environment and it will pick everything up nice and neat. This is especially important for those deploying the functionbeat tool.

List of CloudWatch Log Groups

In order to get Cloudwatch logs into ELK, you will need to configure FunctionBeat to know which log groups you want to input. To supply multiple log groups, you will format the YAML file as such:

    # List of cloudwatch log group registered to that function.
    triggers:
      - log_group_name: /aws/log-group-01
      - log_group_name: /aws/log-group-02
      - log_group_name: /aws/log-group-03
      - log_group_name: /log-group-01
      - log_group_name: /log-group-02

setup.kibana

You need to add the Kibana Endpoint from AWS ElasticSearch Service and append port 443 to the end. The default ports used in standalone setups will not work.

  # Kibana Host
  host: "https://search-<domain_name>-opcj7u2byguqfu75q6lzmfubju.<region>.amazonaws.com/_plugin/kibana:443"

Elasticsearch output

You need to add the ElasticSearch Domain Endpoint from AWS ElasticSearch Service and append port 443 to the end. The default ports used in standalone setups will not work.

output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["https://search-<domain_name>-opcj7u2byguqfu75q6lzmfubju.<region>.amazonaws.com:443"]