TIL that you need to grant permissions to more than just your elasticbeanstalk-<region>-<account> S3 bucket when deploying to Elastic Beanstalk; you also need permissions on elasticbeanstalk-*.