Join the Community
and take part in the story

Gateway: Account HEAD failed: <endpoint_url> 503 Service Unavailable


#1

Hello,

I re-installed everything, sds nodes and gateway, but I still got the same error. I’m pretty sure no dirty operations introduced during installation. I also checked cluster list and gridinit_cmd status and there are no broken or down types. I didn’t find anything unusual in proxy-server.conf on gateway.

Pre-env has been exported. I ran swift cmd for the first time after gateway installation:

root@free-24:~# swift stat
Account HEAD failed: http://192.168.2.24:6007/v1.0/AUTH_1884a2a14df646f2a86a32abc86dc4e2 503 Service Unavailable
Failed Transaction ID: tx4ec455fded7d489c9e293-005abd9859

journal log

Mar 30 09:51:53 free-24 OIO,OPENIO,oioswift,0[1653]: - - 30/Mar/2018/01/51/53 HEAD /v1.0/AUTH_1884a2a14df646f2a86a32abc86dc4e2 HTTP/1.0 204 - Swift - - - - tx94e0e625f6834ae39f478-005abd9839 - 0.0122 RL - 1522374713.746169090 1522374713.758320093 -
Mar 30 09:51:53 free-24 OIO,OPENIO,oioswift,0[1653]: 192.168.2.24 192.168.2.24 30/Mar/2018/01/51/53 HEAD /v1.0/AUTH_1884a2a14df646f2a86a32abc86dc4e2 HTTP/1.0 503 - python-swiftclient-3.4.0 gAAAAABavZg5QPkV... - - - tx94e0e625f6834ae39f478-005abd9839 - 0.0650 - - 1522374713.744107962 1522374713.809098005 -
Mar 30 09:51:54 free-24 OIO,OPENIO,oioswift,0[1653]: 192.168.2.24 192.168.2.24 30/Mar/2018/01/51/54 HEAD /v1.0/AUTH_1884a2a14df646f2a86a32abc86dc4e2 HTTP/1.0 503 - python-swiftclient-3.4.0 gAAAAABavZg5QPkV... - - - tx8bf8910726fe4bf6bc501-005abd983a - 0.0484 - - 1522374714.815087080 1522374714.863527060 -
Mar 30 09:51:56 free-24 OIO,OPENIO,oioswift,0[1653]: 192.168.2.24 192.168.2.24 30/Mar/2018/01/51/56 HEAD /v1.0/AUTH_1884a2a14df646f2a86a32abc86dc4e2 HTTP/1.0 503 - python-swiftclient-3.4.0 gAAAAABavZg5QPkV... - - - tx39497f7fbaf94be0b7730-005abd983c - 0.0453 - - 1522374716.870274067 1522374716.915565014 -
Mar 30 09:52:00 free-24 OIO,OPENIO,oioswift,0[1653]: 192.168.2.24 192.168.2.24 30/Mar/2018/01/52/00 HEAD /v1.0/AUTH_1884a2a14df646f2a86a32abc86dc4e2 HTTP/1.0 503 - python-swiftclient-3.4.0 gAAAAABavZg5QPkV... - - - tx311462a021ad46ce9e37e-005abd9840 - 0.0486 - - 1522374720.924176931 1522374720.972748995 -
Mar 30 09:52:09 free-24 OIO,OPENIO,oioswift,0[1653]: 192.168.2.24 192.168.2.24 30/Mar/2018/01/52/09 HEAD /v1.0/AUTH_1884a2a14df646f2a86a32abc86dc4e2 HTTP/1.0 503 - python-swiftclient-3.4.0 gAAAAABavZg5QPkV... - - - tx5e489ce507554fff94814-005abd9848 - 0.0452 - - 1522374728.984013081 1522374729.029166937 -
Mar 30 09:52:25 free-24 OIO,OPENIO,oioswift,0[1653]: 192.168.2.24 192.168.2.24 30/Mar/2018/01/52/25 HEAD /v1.0/AUTH_1884a2a14df646f2a86a32abc86dc4e2 HTTP/1.0 503 - python-swiftclient-3.4.0 gAAAAABavZg5QPkV... - - - tx4ec455fded7d489c9e293-005abd9859 - 0.0487 - - 1522374745.044442892 1522374745.093168974 -
Mar 30 09:53:41 free-24 puppet-agent[1431]: Could not request certificate: Failed to open TCP connection to puppet:8140 (getaddrinfo: Name or service not known)

–debug

root@free-24:~# swift stat --debug
DEBUG:keystoneclient.auth.identity.v2:Making authentication request to http://192.168.2.24:5000/v2.0/tokens
DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): 192.168.2.24
DEBUG:urllib3.connectionpool:http://192.168.2.24:5000 "POST /v2.0/tokens HTTP/1.1" 200 712
DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): 192.168.2.24
DEBUG:urllib3.connectionpool:http://192.168.2.24:6007 "HEAD /v1.0/AUTH_1884a2a14df646f2a86a32abc86dc4e2 HTTP/1.1" 503 0
INFO:swiftclient:REQ: curl -i http://192.168.2.24:6007/v1.0/AUTH_1884a2a14df646f2a86a32abc86dc4e2 -I -H "X-Auth-Token: gAAAAABavZ-eOXrdsysQpdIopqBJf9jCyA1h3Rc7-weHfb7aUTP6x2TW_d-sj8qjx8jQQSBbK2zrsuP69E6Dsgh5q-hlaK9pZdw4xtqNGLC2s9q_32mfHsWZ_EyEbni_AuxWOtbVA0GfqquyHnqo3RIB-5d41k9IiEUzqqok6cfDnVtKJOTMN94"
INFO:swiftclient:RESP STATUS: 503 Service Unavailable
INFO:swiftclient:RESP HEADERS: {u'Date': u'Fri, 30 Mar 2018 02:23:26 GMT', u'Content-Length': u'0', u'Content-Type': u'text/html; charset=UTF-8', u'X-Openstack-Request-Id': u'txc9bc21b394fc4e2e92887-005abd9f9e', u'X-Trans-Id': u'txc9bc21b394fc4e2e92887-005abd9f9e'}
DEBUG:urllib3.connectionpool:http://192.168.2.24:6007 "HEAD /v1.0/AUTH_1884a2a14df646f2a86a32abc86dc4e2 HTTP/1.1" 503 0
INFO:swiftclient:REQ: curl -i http://192.168.2.24:6007/v1.0/AUTH_1884a2a14df646f2a86a32abc86dc4e2 -I -H "X-Auth-Token: gAAAAABavZ-eOXrdsysQpdIopqBJf9jCyA1h3Rc7-weHfb7aUTP6x2TW_d-sj8qjx8jQQSBbK2zrsuP69E6Dsgh5q-hlaK9pZdw4xtqNGLC2s9q_32mfHsWZ_EyEbni_AuxWOtbVA0GfqquyHnqo3RIB-5d41k9IiEUzqqok6cfDnVtKJOTMN94"
INFO:swiftclient:RESP STATUS: 503 Service Unavailable
INFO:swiftclient:RESP HEADERS: {u'Date': u'Fri, 30 Mar 2018 02:23:27 GMT', u'Content-Length': u'0', u'Content-Type': u'text/html; charset=UTF-8', u'X-Openstack-Request-Id': u'txcfd41b97e79242b38ec27-005abd9f9f', u'X-Trans-Id': u'txcfd41b97e79242b38ec27-005abd9f9f'}
DEBUG:urllib3.connectionpool:http://192.168.2.24:6007 "HEAD /v1.0/AUTH_1884a2a14df646f2a86a32abc86dc4e2 HTTP/1.1" 503 0
INFO:swiftclient:REQ: curl -i http://192.168.2.24:6007/v1.0/AUTH_1884a2a14df646f2a86a32abc86dc4e2 -I -H "X-Auth-Token: gAAAAABavZ-eOXrdsysQpdIopqBJf9jCyA1h3Rc7-weHfb7aUTP6x2TW_d-sj8qjx8jQQSBbK2zrsuP69E6Dsgh5q-hlaK9pZdw4xtqNGLC2s9q_32mfHsWZ_EyEbni_AuxWOtbVA0GfqquyHnqo3RIB-5d41k9IiEUzqqok6cfDnVtKJOTMN94"
INFO:swiftclient:RESP STATUS: 503 Service Unavailable
INFO:swiftclient:RESP HEADERS: {u'Date': u'Fri, 30 Mar 2018 02:23:29 GMT', u'Content-Length': u'0', u'Content-Type': u'text/html; charset=UTF-8', u'X-Openstack-Request-Id': u'tx6126c35ccafe44fea8c7c-005abd9fa1', u'X-Trans-Id': u'tx6126c35ccafe44fea8c7c-005abd9fa1'}
DEBUG:urllib3.connectionpool:http://192.168.2.24:6007 "HEAD /v1.0/AUTH_1884a2a14df646f2a86a32abc86dc4e2 HTTP/1.1" 503 0
INFO:swiftclient:REQ: curl -i http://192.168.2.24:6007/v1.0/AUTH_1884a2a14df646f2a86a32abc86dc4e2 -I -H "X-Auth-Token: gAAAAABavZ-eOXrdsysQpdIopqBJf9jCyA1h3Rc7-weHfb7aUTP6x2TW_d-sj8qjx8jQQSBbK2zrsuP69E6Dsgh5q-hlaK9pZdw4xtqNGLC2s9q_32mfHsWZ_EyEbni_AuxWOtbVA0GfqquyHnqo3RIB-5d41k9IiEUzqqok6cfDnVtKJOTMN94"
INFO:swiftclient:RESP STATUS: 503 Service Unavailable
INFO:swiftclient:RESP HEADERS: {u'Date': u'Fri, 30 Mar 2018 02:23:33 GMT', u'Content-Length': u'0', u'Content-Type': u'text/html; charset=UTF-8', u'X-Openstack-Request-Id': u'txff6af0cea4934802acdd7-005abd9fa5', u'X-Trans-Id': u'txff6af0cea4934802acdd7-005abd9fa5'}
DEBUG:urllib3.connectionpool:http://192.168.2.24:6007 "HEAD /v1.0/AUTH_1884a2a14df646f2a86a32abc86dc4e2 HTTP/1.1" 503 0
INFO:swiftclient:REQ: curl -i http://192.168.2.24:6007/v1.0/AUTH_1884a2a14df646f2a86a32abc86dc4e2 -I -H "X-Auth-Token: gAAAAABavZ-eOXrdsysQpdIopqBJf9jCyA1h3Rc7-weHfb7aUTP6x2TW_d-sj8qjx8jQQSBbK2zrsuP69E6Dsgh5q-hlaK9pZdw4xtqNGLC2s9q_32mfHsWZ_EyEbni_AuxWOtbVA0GfqquyHnqo3RIB-5d41k9IiEUzqqok6cfDnVtKJOTMN94"
INFO:swiftclient:RESP STATUS: 503 Service Unavailable
INFO:swiftclient:RESP HEADERS: {u'Date': u'Fri, 30 Mar 2018 02:23:41 GMT', u'Content-Length': u'0', u'Content-Type': u'text/html; charset=UTF-8', u'X-Openstack-Request-Id': u'tx6f38c2db71de4594ae59f-005abd9fad', u'X-Trans-Id': u'tx6f38c2db71de4594ae59f-005abd9fad'}
DEBUG:urllib3.connectionpool:http://192.168.2.24:6007 "HEAD /v1.0/AUTH_1884a2a14df646f2a86a32abc86dc4e2 HTTP/1.1" 503 0
INFO:swiftclient:REQ: curl -i http://192.168.2.24:6007/v1.0/AUTH_1884a2a14df646f2a86a32abc86dc4e2 -I -H "X-Auth-Token: gAAAAABavZ-eOXrdsysQpdIopqBJf9jCyA1h3Rc7-weHfb7aUTP6x2TW_d-sj8qjx8jQQSBbK2zrsuP69E6Dsgh5q-hlaK9pZdw4xtqNGLC2s9q_32mfHsWZ_EyEbni_AuxWOtbVA0GfqquyHnqo3RIB-5d41k9IiEUzqqok6cfDnVtKJOTMN94"
INFO:swiftclient:RESP STATUS: 503 Service Unavailable
INFO:swiftclient:RESP HEADERS: {u'Date': u'Fri, 30 Mar 2018 02:23:57 GMT', u'Content-Length': u'0', u'Content-Type': u'text/html; charset=UTF-8', u'X-Openstack-Request-Id': u'txb93d40d5c8d44ea2a01dd-005abd9fbd', u'X-Trans-Id': u'txb93d40d5c8d44ea2a01dd-005abd9fbd'}
ERROR:swiftclient.service:Account HEAD failed: http://192.168.2.24:6007/v1.0/AUTH_1884a2a14df646f2a86a32abc86dc4e2 503 Service Unavailable
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/swiftclient/service.py", line 514, in stat
    items, headers = get_future_result(stats_future)
  File "/usr/lib/python2.7/dist-packages/swiftclient/service.py", line 230, in get_future_result
    res = f.result(timeout=timeout)
  File "/usr/lib/python2.7/dist-packages/concurrent/futures/_base.py", line 405, in result
    return self.__get_result()
  File "/usr/lib/python2.7/dist-packages/concurrent/futures/thread.py", line 55, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/lib/python2.7/dist-packages/swiftclient/multithreading.py", line 187, in conn_fn
    return fn(*conn_args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/swiftclient/command_helpers.py", line 24, in stat_account
    headers = conn.head_account(headers=req_headers)
  File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1719, in head_account
    return self._retry(None, head_account, headers=headers)
  File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1678, in _retry
    service_token=self.service_token, **kwargs)
  File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 806, in head_account
    raise ClientException.from_response(resp, 'Account HEAD failed', body)
ClientException: Account HEAD failed: http://192.168.2.24:6007/v1.0/AUTH_1884a2a14df646f2a86a32abc86dc4e2 503 Service Unavailable
Account HEAD failed: http://192.168.2.24:6007/v1.0/AUTH_1884a2a14df646f2a86a32abc86dc4e2 503 Service Unavailable
Failed Transaction ID: txb93d40d5c8d44ea2a01dd-005abd9fbd

is there any way to test the comm between gateway and oioproxy ?

Best regards,
Yongsheng


#2

btw, I installed gateway on Ubuntu 16.04 and the same for sds nodes.


#3

i just find that on gateway

  UNIT                LOAD   ACTIVE SUB    DESCRIPTION
● swift-proxy.service loaded failed failed LSB: Swift proxy server

I can’t start it. Is this a problem? Is swift-proxy server needed ?


#4

Any idea please? Btw, just give you an update that S3 examples work on this gateway but i’m using Keystone authentication. It doesn;t work if I use tempauth and it gives error:

root@free-24:~# aws --endpoint-url http://localhost:6007 --no-verify-ssl s3api create-bucket --bucket bucket-901

An error occurred (SignatureDoesNotMatch) when calling the CreateBucket operation: The request signature we calculated does not match the signature you provided. Check your key and signing method.

even if i replaced localhost with an ip.


#5

Hello @yongsheng, thank you for reporting this. There was indeed an erratum in the documentation which we have noticed recently, and have now corrected.

openiosds::oioswift {'oioswift-0':
  ns                 => 'OPENIO',
  ipaddress          => '0.0.0.0',
  sds_proxy_url      => $openio_proxy_url,
  password     => $swift_passwd,
  memcache_servers   => "${ipaddress}:6019",
  region_name        => $region,
  middleware_swift3 => {'location' => $region},
  project_name => 'services',
}

You need to add the correct project_name to the oioswift declaration in the puppet file, and re-apply the puppet file. The documentation has been updated aswell, with some basic tests added in order to make sure everything works as intended: http://docs.openio.io/17.04/install-swift-s3-ubuntu/installation.html#repositories-configuration.


#6

Yes it works now. Appreciate!


#7