Working environment
The basic workflow of Serverless projects is this: modify files locally, run locally if you can, deploy to AWS and execute on AWS. Let’s set up a nice working environment to make this flow. Add the following lines to the top of your handler file so you can easily write to the logs: import logging logger= logging. get Logger () logger. set Level( logging. DEBUG) logger.info (“The handler started”)
Now, add the following line at the start of the ‘hello’ function: logger. info (“Let’ s get started ..”) logger. info( event)
Next, open two windows. Run serverless logs -t -f hello in one and serverless invoke -f hello in the other. As the invoke function is called you’ll see the activity in the logs.
I’ve found that having all the logs open and being able to deploy changes quickly really speeds things up. As your project gets more complex you can tie in pylint or other parsers and pre-processors like SASS, React packagers and so forth.
Build a pipeline
For our project, we need two Lambda functions: one for an API to push content to an S3 bucket and the other to scan for interesting lines and push them to DynamoDB. Start by adding the following to serverless.yml under the ‘provider’ and ‘functions’ sections. provider: ... iamRo le Statements: - Effect: “Allow” Action: - “s3:*” Resource :“a rn: aws:s3:::dropbu ck etlxf /*” functions: ... upload: handler: handler.upload events: - http: path: upload method: post In the handler.py file, add ‘import boto3’ to the top of the file and add the following method: def upload(event, context): logger. info( event) j son d at a=js on. loads( event [' body ']) data = jsondata['data'] data_ hash= hash lib.sha1(d at a.encode(' utf -8')). hex digest () s3conn=boto3.r es ource(' s 3') s3conn. Bucket (' drop bu ck etlxf '). put_ object( Key=data_hash + '.json’, Body=js on. dumps( data )) body = { “message": “Uploaded file” } response = { “statusCode": 200, “body": “done” } return response
Now deploy as you did before. Make sure you’re tailing the logs for the ‘upload’ method in one terminal while in the other call your endpoint. You might have to curl a couple of times to get the log started: $ curl -XPOST https://YOUR-ENDPOINT-DOMAIN/dev/ upload -d ‘{"data": “Innocent event"}’ $ curl -XPOST https://YOUR-ENDPOINT-DOMAIN/dev/ upload -d ‘{"data": “BADBAD event"}’
Once this is done, visit the S3 console to see your data: https:// console .aws.amazon.com/s3/home. You’ll find your data placed into a file. Our aim in this is to spot ‘BADBAD’ without having to run a server anywhere.
S3 triggered
Our ‘scan’ method will be triggered by the S3 object being created, utilising the event-driven magic of Serverless computing. To hook our new method up to the S3 events, add the following to the ‘functions’ section of serverless.yml: scan: handler: handler.scan events: - s3: bucket: dropbucketlxf event: s3:Object Created:* This is best read backwards: when objects are created in the bucket called dropbucketlxf call handler.scan. Now
create the method to scan for useful lines and then push it into a database: def scan(event, context): logger. info( event) object_ key= event [' Records' ][0] ['s 3'][' object '][' key '] tmpfilenamein = ‘/tmp/tmpfilein_’ + object_key logger.info("We got file: " + object_key) logger. info (" tmp filename in ="+ tmpfilenamein) s3conn=boto3.r es ource(' s 3') s3conn. Bucket (' drop bu ck etlxf '). download_ file( object_ key, tmpfilenamein) bad_lines = [] for line in open( tmp filename in,‘r '): if ‘badthings’ in line: bad_ lines. append( line) logger.info ("Got bad_lines = {}”. format (bad_lines)) dy na modb=bo to 3. resource (' dy na modb ') table= dy na modb. Table (' bad ones ') for line in bad_lines: table. put_ item( Item ={' line ': line }) response = { “statusCode": 200, “body": “Completed scan” } return response
Now deploy and test the pipeline by firing some more events at your API with some including ‘BADBAD’, as we did with curl above. This data will duly be stored in S3 and then analysed by the simple method above. You can find the result by looking in Dynamo DB: https :// console. aws. amazon.
com/dynamodb/home, which is a highly scalable database. That’s nice if you want to look in the raw DB, but how about if someone wants to act on the interesting lines of data you have found? - http: path: get_bad_lines method: get As before, read this in reverse: when a get call to get_bad_ lines is made, call the handler. Next, we write the handler: def get_bad_lines: dy na modb=bo to 3. resource (' dy na modb ') table= dy na modb. Table (' accounts ') Items = table.scan() return { responseCode: 200, Body: json.dumps (items) }