1

I have an AWS Glue job written in Python that I would like to perform pyunit tests on. However, from reading and experimenting on my own, there seem to be quite a few dependencies on the Glue service, making this task very impractical if not impossible. My only thought would be to separate the non glue dependent functions in the job into a separate module to perform unit testing on.

Is unit testing with AWS Glue possible, and if so, what are the best practices for doing so?

Preston Martin
  • 3,288
  • 4
  • 18
  • 39
  • 1
    Checkout this solution with codepipeline. https://aws.amazon.com/blogs/big-data/implement-continuous-integration-and-delivery-of-serverless-aws-glue-etl-applications-using-aws-developer-tools/ – user17183 Sep 19 '19 at 19:26
  • @user17183 Please add this as an answer by explaining the details rather than only adding a link – 030 Dec 19 '19 at 13:57

2 Answers2

2

Checkout this solution with codepipeline. https://aws.amazon.com/blogs/big-data/implement-continuous-integration-and-delivery-of-serverless-aws-glue-etl-applications-using-aws-developer-tools/

user17183
  • 21
  • 2
1

If you are unit testing, you should be mocking external dependencies like Spark and AWS Glue. Look into Python mocking frameworks like unittest.mock, monkeypatch and pytest-mock. Your unit tests should just test your code and how it interacts with Spark and AWS Glue. Your unit tests should not test Spark and AWS Glue functionality. Do that in your component and integration testing.

David R. Willson
  • 11
  • 1