java - What is the underlying issue with sporadic connectivity issues using sun.net.www.protocol.http.HttpURLConnection.getInputStream()? -

my question similar this, yet unanswered, stackoverflow question mysterious connectivity issues. (only under conditions in environments, when trying hit 1 particular url aws) http connectivity consistently fails no apparent reason why.

background:

i've been able reproduce in 2 aws ec2 server environments (though cannot reproduce locally), when trying hit 1 particular customer's web services url (all other urls running similar services work fine).

my java version:

# java -version java version "1.6.0_45" java(tm) se runtime environment (build 1.6.0_45-b06) java hotspot(tm) 64-bit server vm (build 20.45-b01, mixed mode) 

the machine i'm trying hit runs restful web service (in tomcat, fronted apache, on windows machine). can curl same endpoints code trying hit instance code running , valid response in ~48-120ms. code, hit configured 10 second timeout. running netstat both machines shows following server (that i'm making request from):

$ netstat -cowtune | grep <remote_ip> tcp        0    389 ::ffff:10.91.184.202:40153  ::ffff:<remote_ip>:<port>   established 501        33146      on (0.08/2/0) tcp        0    389 ::ffff:10.91.184.202:40153  ::ffff:<remote_ip>:<port>   established 501        33146      on (0.22/3/0) tcp        0    389 ::ffff:10.91.184.202:40153  ::ffff:<remote_ip>:<port>   established 501        33146      on (1.50/4/0) tcp        0    389 ::ffff:10.91.184.202:40153  ::ffff:<remote_ip>:<port>   established 501        33146      on (0.48/4/0) tcp        0    389 ::ffff:10.91.184.202:40153  ::ffff:<remote_ip>:<port>   established 501        33146      on (4.07/5/0) tcp        0    389 ::ffff:10.91.184.202:40153  ::ffff:<remote_ip>:<port>   established 501        33146      on (3.05/5/0) tcp        0    389 ::ffff:10.91.184.202:40153  ::ffff:<remote_ip>:<port>   established 501        33146      on (2.03/5/0) tcp        0    389 ::ffff:10.91.184.202:40153  ::ffff:<remote_ip>:<port>   established 501        33146      on (1.00/5/0) tcp        0    389 ::ffff:10.91.184.202:40153  ::ffff:<remote_ip>:<port>   established 501        33146      on (18446744073.69/5/0) tcp        0    390 ::ffff:10.91.184.202:40153  ::ffff:<remote_ip>:<port>   fin_wait1   0          0          on (8.20/6/0) tcp        0    390 ::ffff:10.91.184.202:40153  ::ffff:<remote_ip>:<port>   fin_wait1   0          0          on (7.18/6/0) tcp        0    390 ::ffff:10.91.184.202:40153  ::ffff:<remote_ip>:<port>   fin_wait1   0          0          on (6.15/6/0) tcp        0    390 ::ffff:10.91.184.202:40153  ::ffff:<remote_ip>:<port>   fin_wait1   0          0          on (5.13/6/0) tcp        0    390 ::ffff:10.91.184.202:40153  ::ffff:<remote_ip>:<port>   fin_wait1   0          0          on (4.11/6/0) tcp        0    390 ::ffff:10.91.184.202:40153  ::ffff:<remote_ip>:<port>   fin_wait1   0          0          on (3.09/6/0) tcp        0    390 ::ffff:10.91.184.202:40153  ::ffff:<remote_ip>:<port>   fin_wait1   0          0          on (2.07/6/0) tcp        0    390 ::ffff:10.91.184.202:40153  ::ffff:<remote_ip>:<port>   fin_wait1   0          0          on (1.05/6/0) tcp        0    390 ::ffff:10.91.184.202:40153  ::ffff:<remote_ip>:<port>   fin_wait1   0          0          on (0.03/6/0) tcp        0    390 ::ffff:10.91.184.202:40153  ::ffff:<remote_ip>:<port>   fin_wait1   0          0          on (17.46/7/0) tcp        0    390 ::ffff:10.91.184.202:40153  ::ffff:<remote_ip>:<port>   fin_wait1   0          0          on (16.44/7/0)  tcp        1    390 ::ffff:10.91.184.202:40153  ::ffff:<remote_ip>:<port>   closing     0              0          on (15.42/7/0) tcp        1    390 ::ffff:10.91.184.202:40153  ::ffff:<remote_ip>:<port>   closing     0          0          on (14.39/7/0) tcp        1    390 ::ffff:10.91.184.202:40153  ::ffff:<remote_ip>:<port>   closing     0          0          on (13.37/7/0) tcp        1    390 ::ffff:10.91.184.202:40153  ::ffff:<remote_ip>:<port>   closing     0          0          on (12.35/7/0) tcp        1    390 ::ffff:10.91.184.202:40153  ::ffff:<remote_ip>:<port>   closing     0          0          on (11.33/7/0) tcp        1    390 ::ffff:10.91.184.202:40153  ::ffff:<remote_ip>:<port>   closing     0          0          on (10.31/7/0) tcp        1    390 ::ffff:10.91.184.202:40153  ::ffff:<remote_ip>:<port>   closing     0          0          on (9.29/7/0) tcp        1    390 ::ffff:10.91.184.202:40153  ::ffff:<remote_ip>:<port>   closing     0          0          on (8.27/7/0) tcp        1    390 ::ffff:10.91.184.202:40153  ::ffff:<remote_ip>:<port>   closing     0          0          on (7.25/7/0) tcp        1    390 ::ffff:10.91.184.202:40153  ::ffff:<remote_ip>:<port>   closing     0          0          on (6.23/7/0) tcp        1    390 ::ffff:10.91.184.202:40153  ::ffff:<remote_ip>:<port>   closing     0          0          on (5.21/7/0) tcp        1    390 ::ffff:10.91.184.202:40153  ::ffff:<remote_ip>:<port>   closing     0          0          on (4.19/7/0) tcp        1    390 ::ffff:10.91.184.202:40153  ::ffff:<remote_ip>:<port>   closing     0          0          on (3.17/7/0) tcp        1    390 ::ffff:10.91.184.202:40153  ::ffff:<remote_ip>:<port>   closing     0          0          on (2.15/7/0) tcp        1    390 ::ffff:10.91.184.202:40153  ::ffff:<remote_ip>:<port>   closing     0          0          on (1.13/7/0) tcp        1    390 ::ffff:10.91.184.202:40153  ::ffff:<remote_ip>:<port>   closing     0          0          on (0.11/7/0) 

and remote server (that i'm making requests to):

d:\cygwin>netstat -ant 1 | grep 54.81.126.17 tcp    <ip_address>:<port>    54.81.126.17:40153    syn_received    inhost tcp    <ip_address>:<port>    54.81.126.17:40153    established     inhost tcp    <ip_address>:<port>    54.81.126.17:40153    established     inhost tcp    <ip_address>:<port>    54.81.126.17:40153    established     inhost tcp    <ip_address>:<port>    54.81.126.17:40153    established     inhost tcp    <ip_address>:<port>    54.81.126.17:40153    established     inhost tcp    <ip_address>:<port>    54.81.126.17:40153    established     inhost tcp    <ip_address>:<port>    54.81.126.17:40153    established     inhost tcp    <ip_address>:<port>    54.81.126.17:40153    established     inhost tcp    <ip_address>:<port>    54.81.126.17:40153    established     inhost  tcp    <ip_address>:<port>    54.81.126.17:40153    fin_wait_2      inhost tcp    <ip_address>:<port>    54.81.126.17:40153    fin_wait_2      inhost tcp    <ip_address>:<port>    54.81.126.17:40153    fin_wait_2      inhost tcp    <ip_address>:<port>    54.81.126.17:40153    fin_wait_2      inhost tcp    <ip_address>:<port>    54.81.126.17:40153    fin_wait_2      inhost tcp    <ip_address>:<port>    54.81.126.17:40153    fin_wait_2      inhost tcp    <ip_address>:<port>    54.81.126.17:40153    fin_wait_2      inhost tcp    <ip_address>:<port>    54.81.126.17:40153    fin_wait_2      inhost tcp    <ip_address>:<port>    54.81.126.17:40153    fin_wait_2      inhost tcp    <ip_address>:<port>    54.81.126.17:40153    fin_wait_2      inhost tcp    <ip_address>:<port>    54.81.126.17:40153    fin_wait_2      inhost tcp    <ip_address>:<port>    54.81.126.17:40153    fin_wait_2      inhost 

at configured 10 second timeout, server shows transition established fin_wait_1. time later server show transition fin_wait_1 closing, , @ same moment remote server transitions established fin_wait_2. remote tomcat never registers receiving request. tshark shows:

0.000000 10.182.160.132 -> <remote_ip> tcp 74 49486 > http-alt [syn] seq=0 win=14600 len=0 mss=1460 sack_perm=1 tsval=1814494 tsecr=0 ws=128 0.035580 <remote_ip> -> 10.182.160.132 tcp 70 http-alt > 49486 [syn, ack] seq=0 ack=1 win=8192 len=0 mss=1380 sack_perm=1 tsval=101011325 tsecr=1814494 0.035601 10.182.160.132 -> <remote_ip> tcp 66 49486 > http-alt [ack] seq=1 ack=1 win=14600 len=0 tsval=1814503 tsecr=101011325 0.035935 10.182.160.132 -> <remote_ip> http 457 post /service/rest/security/myendpoint http/1.1 0.171137 10.182.160.132 -> <remote_ip> http 457 [tcp retransmission] post /service/rest/security/myendpoint http/1.1 0.443125 10.182.160.132 -> <remote_ip> http 457 [tcp retransmission] post /service/rest/security/myendpoint http/1.1 0.987118 10.182.160.132 -> <remote_ip> http 457 [tcp retransmission] post /service/rest/security/myendpoint http/1.1 2.079144 10.182.160.132 -> <remote_ip> http 457 [tcp retransmission] post /service/rest/security/myendpoint http/1.1 4.263141 10.182.160.132 -> <remote_ip> http 457 [tcp retransmission] post /service/rest/security/myendpoint http/1.1 8.631153 10.182.160.132 -> <remote_ip> http 457 [tcp retransmission] post /service/rest/security/myendpoint http/1.1 10.036939 10.182.160.132 -> <remote_ip> tcp 66 49486 > http-alt [fin, ack] seq=392 ack=1 win=14600 len=0 tsval=1817003 tsecr=101011325 10.072638 <remote_ip> -> 10.182.160.132 tcp 66 [tcp window update] http-alt > 49486 [ack] seq=1 ack=1 win=64296 len=0 tsval=101012329 tsecr=1814503 17.351131 10.182.160.132 -> <remote_ip> http 457 [tcp retransmission] post /service/rest/security/myendpoint http/1.1 20.584358 <remote_ip> -> 10.182.160.132 tcp 66 http-alt > 49486 [fin, ack] seq=1 ack=1 win=64296 len=0 tsval=101013380 tsecr=1814503 20.584421 10.182.160.132 -> <remote_ip> tcp 66 49486 > http-alt [ack] seq=393 ack=2 win=14600 len=0 tsval=1819640 tsecr=1 

my old code:

inputstream getresponsestream(string webserviceurl) {   url server = new url(webserviceurl);   httpurlconnection connection = (httpurlconnection) server.openconnection();   connection.setdoinput(true);   connection.setdooutput(true);   connection.setrequestmethod("get");   return connection.getinputstream(); // timeout happens here } 

my better code (this , below):

private object getresponse(httpurlconnection connection,         sdrestresponsetype resptype) throws ioexception, jaxbexception,         protocolexception {     inputstream = null;     try {         // check if valid response         int responsecode = connection.getresponsecode(); // timeout happens here         if (responsecode == httpurlconnection.http_ok) {             = connection.getinputstream();             switch (resptype) {             case boolean:                 return boolean.valueof(readinput(is));             case string:                 return readinput(is);             case xml:                 unmarshaller unmarshaller = context.createunmarshaller();                 return unmarshaller.unmarshal(is);             default:                 return null;             }         }          = connection.geterrorstream();         unmarshaller unmarshaller = context.createunmarshaller();         object response = unmarshaller.unmarshal(is);          if (response instanceof fault) {             throw new sdfaultexception((fault) response);         }         throw new protocolexception(connection.getresponsemessage());      } {         if (is != null) {             is.close();         }     } } 

the bit of code creates httpurlconnection object executes request:

private httpurlconnection getconnection(string operation, boolean xmlcontent)         throws ioexception {      url server = new url(baseurl + operation);     httpurlconnection connection = (httpurlconnection) server             .openconnection();     connection.setdoinput(true);     connection.setdooutput(true);     connection.setreadtimeout(10000);     connection.setrequestmethod(post); // remote endpoint accepts request either or post fine, except code     connection.setrequestproperty(content_type, (xmlcontent ? xml_encoded             : url_encoded));     // set header values     connection.addrequestproperty(client_id, header.getclientid());     if (header.getlocale() != null) {         connection.addrequestproperty(locale, header.getlocale());     }     if (header.getsessiontoken() != null) {         connection.addrequestproperty(session, header.getsessiontoken());     }     if (this.passthrough != null) {         connection.addrequestproperty(passthru, this.passthrough);     }      return connection; } 

my server (the box) running linux, apache, , app in tomcat. dns lookups have turned nothing unexpected. connectivity between boxes seems normal in other respects (i have not exhaustively gone through iptables config). when step through code seems normal until execution disappears compiled code of sun.net.www.protocol.http.httpurlconnection.getinputstream().

in grepcode's openjdk source, line 710 shows ioexception getting swallowed, since oracle version source proprietary (and therefore not available anywhere i've found) i'm wondering if knows (or point me to) might happening under hood, since i've not been able rule out possibility of issues in server environment.

thanks in advance insight!

answering own ?:

never trust staff.

after double/triple checking, turns out remote server was fronted active intrusion detection system blocking unknown ip addresses. since aws instances can change ip when cycled, if white-listed known ip, work until instance bounced. lesson learned: nauseatingly specific when asking "could blocking us?"

why allowed curl through still mystery until reply email update answer with...