Weird outbound AWS traffic discovery

Summary:

It has come to my attention that AWS sometimes generates weird, unexplained traffic - when we first noticed it, it appeared to be mostly rDNS queries, but further investigation has shown it's far more than that. In fact - rDNS isn't even the typical traffic seen. We're seeing this from accounts/instances that have no resources which should be capable of generating said traffic, and it's frequently (but not always) 40 bytes of source traffic.

TL;DR: AWS generating weird 40-byte packets with no discernible cause/source/reason
EDIT: This appears to be due to a quirk in how AWS logs things. Jump to the end for more details.

Disclaimers:

This is a joint discovery with Alex Heacock, and he's basically done all the work on this, including coming up with the first and second versions of the "steps to reproduce", first and second version of the YAML file, and working with AWS Support to see if we can figure out what's going on. I have written and tested this final version of the "steps to reproduce" in my own personal AWS account and confirmed the behavior observed.

Disclaimer: I am a nobody. (Alex is smart though.) It could be we're missing something obvious, but the fact that it can be reproduced so easily and has no obvious explanation, seems somewhat concerning. Speculation has ranges from worst-case (AWS rootkit?) to the more benign (Athena logs pulling in traffic from other VPC resources on the same hardware?).

Prerequisites:

• AWS account
• Payment setup in AWS account
• Willingness to spend ~$32/month as long as you leave the stack up

Steps to reproduce:

1. Login to AWS and go to CloudFormation
2. Click "Create Stack"
3. Set radio button: "Template is ready"
4. Set radio button: "Upload a template file"
5. Browse to "vpc_flow_log_test2.yml" (provided below)
6. Click "Next"
7. Stack name: "weird-aws-traffic-test" (or your preference)
8. Click "Next"
9. Check the box "I acknowledge that AWS CloudFormation might create IAM resources."
10. Click "Submit"
11. Refresh a few times to watch progress
12. Go to Resources tab; confirm all show CREATE_COMPLETE with green/white checkmark/circle
13. Open Athena
14. Set radio button: "Query your data"
15. Click "Launch query editor"
16. If you don't have an s3 bucket with Athena setup already, follow steps 17-21. Otherwise, you might be able to continue at step 22.
17. On the banner "Before you run your first query, you need to set up a query result location in Amazon S3." Click "Edit settings"
18. Click "Browse S3"
19. Click the radio button for the name that matches step 7 (like: weird-aws-traffic-test-vpcblahblah)
20. Click "Choose"
21. Click "Save"
22. Click "Editor" tab again
23. Paste query (provided below)
24. Click "Run"
25. If you don't see any traffic, wait an hour or two and try again.
26. Repeat steps 24-25 until you see something. Odd traffic is "egress" though there's no resources spun up that should be generating. Appears to be DNS, possibly rDNS, considering it's port 53
27. Done! It's like watching fish in an aquarium.

Steps to cleanup:

1. Login to AWS and go to CloudFormation
2. Click the radio button next to the stack you created
3. Click "Delete"
4. Click "Delete" again.
5. Wait a few minutes; refresh the page to confirm deleted.
6. Done!

Conclusion:

I don't have a conclusion at this point. Hopefully someone can provide some explanation for why this happens.

EDIT: After some time back-and-forth with AWS Support regarding this, we were told these are RST packets. AWS VPC flow logs do not contain TCP flag data when using Version 2 - the version in use that prompted us to notice this. Apparently, another quirk of this version, is that if an unexpected SYN-ACK packet is received, and the VPC responds with an RST packet (as it should, per RFC 793), then the AWS logs the RST packet it sent, but not the SYN-ACK packet that triggered it. Thus, the VPC appears to be sending packets to the source of the SYN-ACK packets entirely unprompted (even though it's not.)

Resources:

YAML file:

Pastebin link: here

Athena query:

SELECT * FROM "vpc_flow_logs_db"."vpc_flow_logs_table"

WHERE srcaddr LIKE '10.0.0.%'

AND action='ACCEPT'

ORDER BY day DESC

limit 10000;