Analyze logs with Dynatrace Davis AI Engine utilizing Amazon Kinesis Information Firehose HTTP endpoint supply

This weblog publish is co-authored with Erick Leon, Sr. Technical Alliance Supervisor from Dynatrace.

Amazon Kinesis Information Firehose is the best approach to reliably load streaming knowledge into knowledge lakes, knowledge shops, and analytics companies. With only a few clicks, you may create fully-managed supply streams that auto scale on demand to match the throughput of your knowledge. Clients already use Kinesis Information Firehose to ingest uncooked knowledge from varied knowledge sources, together with logs from AWS companies. Kinesis Information Firehose now helps delivering streaming knowledge to Dynatrace. Dynatrace begins analyzing incoming knowledge inside minutes of Amazon CloudWatch knowledge era.

Beginning at this time, you should use Kinesis Information Firehose to ship CloudWatch Metrics and Logs on to the Dynatrace observability platform to carry out your explorations and evaluation. Dynatrace, an AWS Accomplice Community (APN) has offered full observability into AWS Providers by ingesting CloudWatch metrics which can be revealed by AWS companies. Dynatrace ingests this knowledge to carry out root-cause evaluation utilizing the Dynatrace Davis AI engine.

On this publish, we describe the Kinesis Information Firehose and associated Dynatrace integration.

Conditions

For this walkthrough, you need to have the next stipulations:

• AWS account.
• Entry to the CloudWatch and Kinesis Information Firehose with permissions to handle HTTP endpoints.
• Dynatrace Clever Observability Platform account, or get a free 15 day trial right here.
• Dynatrace model 1.182+.
• An up to date AWS monitoring coverage to incorporate the extra AWS companies.
  To replace the AWS Id and Entry Administration (IAM) coverage, use the JSON within the hyperlink above, containing the monitoring coverage (permissions) for all supporting companies.
• Dynatrace API token: create token with the next permission and hold available in a notepad.

Determine 1 – Dynatrace API Token

The way it works

Determine 2 – Amazon Kinesis Information Firehose HTTP endpoint supply

Merely create a log stream to your Amazon companies to ship your context wealthy logs to the Amazon CloudWatch Logs service. Subsequent, choose your Dynatrace HTTP endpoint to reinforce your logs streams with the ability of the Dynatrace Intelligence Platform. Lastly, you may as well again up your logs to an Amazon Easy Storage Service (Amazon S3) bucket.

Setup directions

So as to add a service to monitoring, observe these steps:

1. Within the Dynatrace menu, go to Settings > Cloud and virtualization, and choose AWS.
2. On the AWS overview web page, scroll down and choose the specified AWS occasion. Choose the Edit button.
3. Scroll down and choose Add service. Select the service identify from the drop-down, and choose Add service.
4. Choose Save adjustments.

To course of and ship AWS CloudWatch Metrics to Dynatrace, observe these steps.

1. Log in to the AWS console and kind “Kinesis” within the textual content search bar. Choose Kinesis

Determine 3 – AWS Console

2. On the Amazon Kinesis companies web page, choose the radio button for Kinesis Information Firehose and choose the Create supply stream button.

Determine 4 – Amazon Kinesis

3. Select the “Direct PUT” from the drop down, and from Vacation spot drop down, select “Dynatrace”.

Determine 5 – Amazon Kinesis Information Firehose

4. Supply stream identify – Give your stream a brand new identify, for instance: – “KFH-StreamToDynatrace”

Determine 6 – Supply stream identify

5. Within the part “Vacation spot settings”:

Determine 7 – Vacation spot settings

6. HTTP endpoint identify – “Dynatrace”.
7. HTTP endpoint URL – From the drop down, choose “Dynatrace – US”.
8. API token – Enter Dynatrace API TOKEN created within the prerequisite part.
9. API URL – enter the Dynatrace URL to your tenant, for instance: https://xxxxx.reside.dynatrace.com
10. Again Up Settings – Both choose an current S3 bucket or create a brand new one and add particulars and choose the Create supply stream button.

Determine 8 – Backup settings

As soon as profitable, your AWS Console will appear like the next:

Determine 9 – Amazon Kinesis Information Firehose

The Dynatrace Expertise

As soon as the preliminary setups are accomplished in each Dynatrace and the AWS Console, observe these steps to visualise your new KHF stream knowledge within the Dynatrace console.

1. Log in to the Dynatrace Console, and on the left facet menu develop the “infrastructure” part, and choose “AWS”
2. From the display screen, choose the AWS account that you just need to add the KFH stream to.
3. Subsequent, you’ll see a virtualization of your AWS property for the account chosen. Choose the field marked “Supporting Providers”.
4. Subsequent, press the “Configure companies” button.
5. Subsequent, choose “Add service”.
6. From the drop down, choose “Kinesis Information Firehose”.
7. Subsequent, choose the “Add metric” button, and choose the metrics that you just need to see for this stream. Dynatrace has a complete checklist of metrics that may be chosen from the UI. The checklist might be discovered on this hyperlink.

Troubleshooting

1. After configuration, load to the brand new KFH stream no knowledge within the Dynatrace Console.
   1. Verify the Error Logs tab test to guarantee that the Vacation spot URL is right for the Dynatrace Tenant.

Determine 10 – Vacation spot error logs

2. Invalid or misconfigured Dynatrace API token or scope isn’t correctly set.

Determine 11 – Vacation spot error logs

Conclusion

On this publish, we exhibit the Kinesis Information Firehose and associated Dynatrace integration. As well as, engineers can use CloudWatch Metrics to discover their manufacturing methods alongside occasions in Dynatrace. This gives a seamless, present view of your system (from logs to occasions and traces) in a single knowledge retailer.

To study extra about CloudWatch Service, see the Amazon CloudWatch residence web page. When you’ve got any questions, publish them on the AWS CloudWatch service discussion board.

For those who haven’t but signed up for Dynatrace, then you may check out Kinesis Information Firehose with Dynatrace with a free Dynatrace trial.

Concerning the Authors

Erick Leon is a Technical Alliances Sr. Supervisor at Dynatrace, Observability Follow Architect, and Buyer Advocate. He promotes robust technical integrations with a concentrate on AWS. With over 15 years as a Dynatrace buyer, his real-world experiences and classes realized convey helpful insights into the Dynatrace Clever Observability Platform.

Shashiraj Jeripotula (Raj) is a San Francisco-based Sr. Accomplice Options Architect at AWS. He works with varied impartial software program distributors (ISVs), and companions who focus on cloud administration instruments and DevOps to develop joint options and speed up cloud adoption on AWS.