amazon web services - Advice sought on ways of downloading a large file from a bandwidth-throttled server to a AWS S3 bucket -


for project i'm working on, there need pull down largish text file that's updated daily , made available @ specific customer url, , store in aws s3 triggers downstream processing of file (details unimportant).

i thinking of having download + store in s3 done aws lambda triggered every 24 hours cloudwatch, work, there's catch: file 36mb in size , served host throttles downloads 100kb/s (outside of control). means takes @ least 360s (i.e. 6 mins) download file. aws lambda functions have upper limit of 300s run time, makes impossible use task lambda times-out , exits before file downloaded.

i'm looking suggestions of ways of working around 300s run time limit of aws lambda achieve goal.

as long i'm sticking aws, alternative see set cron job on ec2 instance, seems expensive / overkill, if end not needing always-on ec2 else.

thanks!

i'd have lambda spin small ec2 instance runs copy job. have either custom ami ec2 instance or cloud-init script sets up. let program on ec2 run bit , remember billed hour regardless of how time need. if entire process takes 15 mins (as there no way guarantee against traffic congestion) , you're using t2.nano, got billed usd $0.006 (6 tenths of cent) plus i/o and, likely, ebs space. i'd willing bet you'd spend little.

once job done terminates ec2 instance it's running on.

i realize bit of hassle - cloudwatch triggers lambda triggers ec2. cloudwatch alone isn't going able need ec2.


Comments

Post a Comment

Popular posts from this blog

commonjs - How to write a typescript definition file for a node module that exports a function? -

consider, toml node module can use: // toml.d.ts declare module toml { export function parse(value:string):any; } declare module "toml" { export = toml; } then: /// <reference path="../../../../../defs/toml/toml.d.ts"/> import toml = require('toml'); toml.parse(...); however, node module export single function, such 'glob' ( https://github.com/isaacs/node-glob ). the node usage of module be: var glob = require("glob") glob("*.js", {}, callback(err, files) { ... }); you'd naively expect this: // glob.d.ts declare function globs(paths:string, options:any, callback:{(err:any, files:string[]):void; ...but because typescripts 'import' semantics bit strange, seems can use 'import .. = require()' statement alias modules . trying call: /// <reference path="../../../../../defs/glob/glob.d.ts"/> import blog = require('glob'); results in: error ts2072: mod
Read more

openid - Okta: Failed to get authorization code through API call -

i'm integrating okta own idp server using okta's api. i'm implementing authorization code flow following steps below: in own server, use /api/v1/authn endpoint sessiontoken. use sessiontoken obtain authorization calling endpoint: /oauth2/v1/authorize?client_id=" + clientid + "&sessiontoken=" + sessiontoken + "&response_type=code&response_mode=query&scope=openid&redirect_uri=" + redirecturl + "&state=evanyang&nonce=" it's supposed return response status code 302 , location header containing redirect url code value. however, keep getting response status code 200 , without location header, html body saying "you using unsupported browser." , "javascript disabled on browser." according api documentation: http://developer.okta.com/docs/api/resources/oidc.html#authentication-request , sessiontoken parameter sufficient this: an okta one-time sessiontoken. allows api-based
Read more

thorough guide for profiling racket code -

did googling ("racket profiling", "racket measure performance"), didn't find , there no examples in docs. did google "profile" search on htdp - no luck. (profile (f ...)) output isn't obvious other small snippets. ideally want python's python -m cprofile usage examples. when search "python profiling", both duckduckgo , google give top result: 26.4. python profilers . (although scanned quickly, seems more of reference "thorough usage guide" examples. if had else in mind, perhaps please link that?) the equivalent racket documentation be: profile: statistical profiler . a usage example: #lang racket (require profile) (profile-thunk (thunk (function-to-profile arg0 arg1) )) here (thunk e) convenience (lambda () e) . bigger example: #lang racket (module mod racket (provide f) (define (f) (for/list ([i 10000]) i))) (require (prefix-in mod: 'mod)) (define (f) (for ([i 10000
Read more