php - Receiving "502 Bad Gateway" fron Cloudfront when scraping a website? -


i've built scraping script in php gather information particular website. have tested script thoroughly downloaded html files target website, xpath queries correct. yesterday tried script first time locally targetting actual website , worked. took script, placed on server farm, , turned on.

this morning awoke 90 emails 1 particular server telling me there's been error. server sent 10 emails, while other 2 seem working away fine. i've checked logs keep on database , of errors encountered have been "502 bad gateway". i've tried url through normal web browser , loads fine, , i've tried url via wget method on same server. wget returns error:

error: certificate of http://www.targetwebsite.com not trusted.

error: certificate of http://www.targetwebsite.com hasn't got known issuer.

​using "--no-check-certificate" flag still produces error, downloads html file anyway.

so anyway, in script have following code:

// assign curl options $curloptions = array( curlopt_returntransfer => true, curlopt_header => true, curlopt_followlocation => true, curlopt_encoding => "", curlopt_autoreferer => true, curlopt_connecttimeout => 120, curlopt_timeout => 120, curlopt_maxredirs => 10, curlinfo_header_out => true, curlopt_ssl_verifypeer => false, curlopt_http_version => 'curl_http_version_1_1', curlopt_cookie => $cookiesjar, curlopt_useragent => $useragent, );  // url scrape $url = "https://www.targetsite.com/specific/page/";   // build curl headers $ch = curl_init($url); curl_setopt_array($ch, $curloptions);  $content = curl_exec($ch); $err = curl_errno($ch); $errmsg = curl_error($ch); $header = curl_getinfo($ch); $responsecode = curl_getinfo($ch, curlinfo_http_code);

​ tends work, isn't 100% reliable @ moment due cloudfront returning 502 bad gateway errors. but, @ same time, i've never created web scraping script before, , while i'm sure that's options need make website think i'm legitimate user, i'm missing something!

from reading i've done, there talk how need pass on ciphers through target site. added curl options:

curlopt_ssl_cipher_list => 'ecdhe-rsa-aes128-gcm-sha256, ecdhe-rsa-aes128-sha256, ecdhe-rsa-aes128-sha, ecdhe-rsa-aes256-gcm-sha384, ecdhe-rsa-aes256-sha384, ecdhe-rsa-aes256-sha ,aes128-gcm-sha256, aes256-gcm-sha384, aes128-sha256, aes256-sha, aes128-sha, rc4-md5',

i've noticed improvement, i'm still getting fair number of 502's.

if me amazing.


Comments

Post a Comment

Popular posts from this blog

commonjs - How to write a typescript definition file for a node module that exports a function? -

consider, toml node module can use: // toml.d.ts declare module toml { export function parse(value:string):any; } declare module "toml" { export = toml; } then: /// <reference path="../../../../../defs/toml/toml.d.ts"/> import toml = require('toml'); toml.parse(...); however, node module export single function, such 'glob' ( https://github.com/isaacs/node-glob ). the node usage of module be: var glob = require("glob") glob("*.js", {}, callback(err, files) { ... }); you'd naively expect this: // glob.d.ts declare function globs(paths:string, options:any, callback:{(err:any, files:string[]):void; ...but because typescripts 'import' semantics bit strange, seems can use 'import .. = require()' statement alias modules . trying call: /// <reference path="../../../../../defs/glob/glob.d.ts"/> import blog = require('glob'); results in: error ts2072: mod
Read more

openid - Okta: Failed to get authorization code through API call -

i'm integrating okta own idp server using okta's api. i'm implementing authorization code flow following steps below: in own server, use /api/v1/authn endpoint sessiontoken. use sessiontoken obtain authorization calling endpoint: /oauth2/v1/authorize?client_id=" + clientid + "&sessiontoken=" + sessiontoken + "&response_type=code&response_mode=query&scope=openid&redirect_uri=" + redirecturl + "&state=evanyang&nonce=" it's supposed return response status code 302 , location header containing redirect url code value. however, keep getting response status code 200 , without location header, html body saying "you using unsupported browser." , "javascript disabled on browser." according api documentation: http://developer.okta.com/docs/api/resources/oidc.html#authentication-request , sessiontoken parameter sufficient this: an okta one-time sessiontoken. allows api-based
Read more

thorough guide for profiling racket code -

did googling ("racket profiling", "racket measure performance"), didn't find , there no examples in docs. did google "profile" search on htdp - no luck. (profile (f ...)) output isn't obvious other small snippets. ideally want python's python -m cprofile usage examples. when search "python profiling", both duckduckgo , google give top result: 26.4. python profilers . (although scanned quickly, seems more of reference "thorough usage guide" examples. if had else in mind, perhaps please link that?) the equivalent racket documentation be: profile: statistical profiler . a usage example: #lang racket (require profile) (profile-thunk (thunk (function-to-profile arg0 arg1) )) here (thunk e) convenience (lambda () e) . bigger example: #lang racket (module mod racket (provide f) (define (f) (for/list ([i 10000]) i))) (require (prefix-in mod: 'mod)) (define (f) (for ([i 10000
Read more