Join the Community
and take part in the story

No META0 available in the namespace


#1

Hello,

I am installing open source version on CentOS 7, but I can’t get success on the step of
Package Installation and Service Configuration. I followed this page: http://docs.openio.io/17.04/install-guide-centos/installation.html

After applying the manifest, puppet apply --no-stringify_facts /root/openio.pp

[root@io-x01 ~]# gridinit_cmd status
KEY                       STATUS      PID GROUP
OPENIO-account-0          UP         2572 OPENIO,account,account-0
OPENIO-beanstalkd-0       UP         2519 OPENIO,beanstalkd,beanstalkd-0
OPENIO-conscience-0       UP         2561 OPENIO,conscience,conscience-0
OPENIO-conscienceagent-0  UP         2353 OPENIO,conscienceagent,conscienceagent-0
OPENIO-meta0-0            BROKEN       -1 OPENIO,meta0,meta0-0
OPENIO-meta1-0            UP         2495 OPENIO,meta1,meta1-0
OPENIO-meta2-0            UP         2573 OPENIO,meta2,meta2-0
OPENIO-oio-blob-indexer-0 UP         2362 OPENIO,oio-blob-indexer,oio-blob-indexer-0
OPENIO-oio-event-agent-0  UP         2587 OPENIO,oio-event-agent,oio-event-agent-0
OPENIO-oioproxy-0         UP         2600 OPENIO,oioproxy,oioproxy-0
OPENIO-rawx-0             UP         2341 OPENIO,rawx,rawx-0
OPENIO-rdir-0             UP         2601 OPENIO,rdir,rdir-0
OPENIO-redis-0            UP         2463 OPENIO,redis,redis-0
OPENIO-redissentinel-0    UP         2544 OPENIO,redissentinel,redissentinel-0
OPENIO-zookeeper-0        UP         2547 OPENIO,zookeeper,zookeeper-0
[root@io-x01 ~]# 

In /var/log/oio/sds/OPENIO/meta0-0/meta0-0.log

2017-12-21T23:56:13.612418-05:00 io-x01 OIO,OPENIO,meta1,0: warning  2140 D884 log WRN oio.m1v2 PREFIXES reload failure : (453) Reload error: NS loading error : META0 locate error : Proxy error (404): (code=453) Unexpected service type
2017-12-21T23:56:14.603210-05:00 io-x01 OIO,OPENIO,meta1,0: warning  2140 D884 log WRN oio.m1v2 PREFIXES reload failure : (0) Reload error: NS loading error : No META0 available in the namespace
2017-12-21T23:56:15.068375-05:00 io-x01 OIO,OPENIO,meta1,0: notice  2495 EAE6 log NOT oio.sqlx TFO[1] UDP[1] OUT[0]
2017-12-21T23:56:15.068679-05:00 io-x01 OIO,OPENIO,meta1,0: info  2495 EAE6 log INF oio.sqlx FD limits set to ACTIVES[21835] PASSIVES[21834] BASES[21835/32768] SYS[65536]
2017-12-21T23:56:15.069545-05:00 io-x01 OIO,OPENIO,meta1,0: info  2495 EAE6 log INF oio.sqlite Election delays:
2017-12-21T23:56:15.069802-05:00 io-x01 OIO,OPENIO,meta1,0: info  2495 EAE6 log INF oio.sqlite - get_status=5000ms but nowait after 900000ms
2017-12-21T23:56:15.070035-05:00 io-x01 OIO,OPENIO,meta1,0: info  2495 EAE6 log INF oio.sqlite - expire_SLAVE=900000ms, expire_MASTER=1500000ms, expire_NONE=300000ms

What’s wrong here?

Best regards,
Yongsheng


#2

Hi Yongsheng!

Meta0 service fails to start, could you post your openio.pp file?


#3

Hello

[root@io-x01 ~]# cat openio.pp 
$ipaddr = "192.168.2.91"
class {'openiosds':}
openiosds::conscience {'conscience-0':
  ns        => 'OPENIO',
  ipaddress => $ipaddr,
}
openiosds::namespace {'OPENIO':
  ns                       => 'OPENIO',
  conscience_url           => "192.168.2.91:6000",
  zookeeper_url            => "192.168.2.91:6005,192.168.2.92:6005,192.168.2.93:6005",
  oioproxy_url             => "${ipaddr}:6006",
  eventagent_url           => "beanstalk://${ipaddr}:6014",
  ns_service_update_policy => {'meta2'=>'KEEP|3|1|','sqlx'=>'KEEP|3|1|','rdir'=>'KEEP|1|1|user_is_a_service=rawx'},
  ns_storage_policy        => 'THREECOPIES',
}
openiosds::account {'account-0':
  ns                   => 'OPENIO',
  ipaddress            => $ipaddr,
  sentinel_hosts       => '192.168.2.91:6012,192.168.2.92:6012,192.168.2.93:6012',
  sentinel_master_name => 'OPENIO-master-1',
}
openiosds::meta0 {'meta0-0':
  ns        => 'OPENIO',
  ipaddress => $ipaddr,
}
openiosds::meta1 {'meta1-0':
  ns        => 'OPENIO',
  ipaddress => $ipaddr,
}
openiosds::meta2 {'meta2-0':
  ns        => 'OPENIO',
  ipaddress => $ipaddr,
}
openiosds::rawx {'rawx-0':
  ns        => 'OPENIO',
  ipaddress => $ipaddr,
}
openiosds::oioeventagent {'oio-event-agent-0':
  ns        => 'OPENIO',
  ipaddress => $ipaddr,
}
openiosds::oioproxy {'oioproxy-0':
  ns        => 'OPENIO',
  ipaddress => $ipaddr,
}
openiosds::zookeeper {'zookeeper-0':
  ns        => 'OPENIO',
  ipaddress => $ipaddr,
  servers   => ['192.168.2.91:2888:3888','192.168.2.92:2888:3888','192.168.2.93:2888:3888'],
}
openiosds::redissentinel {'redissentinel-0':
  ns          => 'OPENIO',
  ipaddress   => $ipaddr,
  master_name => 'OPENIO-master-1',
  redis_host  => "192.168.2.91",
}
openiosds::redis {'redis-0':
  ns        => 'OPENIO',
  ipaddress => $ipaddr,
}
openiosds::conscienceagent {'conscienceagent-0':
  ns => 'OPENIO',
}
openiosds::beanstalkd {'beanstalkd-0':
  ns        => 'OPENIO',
  ipaddress => $ipaddr,
}
openiosds::rdir {'rdir-0':
  ns        => 'OPENIO',
  ipaddress => $ipaddr,
}
openiosds::oioblobindexer {'oio-blob-indexer-rawx-0':
  ns => 'OPENIO',
}

I also wondering

openiosds::zookeeper {'zookeeper-0':
  ns        => 'OPENIO',
  ipaddress => $ipaddr,
  servers   => ['SERVER1:2888:3888','SERVER2:2888:3888','SERVER3:2888:3888'],
}

What does ‘SERVER1:2888:3888’ mean here?


#4

SERVER1 refers to the address IP of the server1.

Replace SERVER1, SERVER2 and SERVER3 with the corresponding IP addresses.

Make sure to replace with the actual IP addresses of your servers.


#5

Yes, i did replace them.


#6

There’s another question on Installation page.

On the server 2 and 3, add slaveof => ‘SERVER1 6011’, in the redis block.

Are you sure it is a white space between SERVER1 and 6011 ? Not a colon ?


#7

This is the pp on server 2

[root@io-x02 ~]# cat openio.pp 
$ipaddr = "192.168.2.92"
class {'openiosds':}
openiosds::conscience {'conscience-0':
  ns        => 'OPENIO',
  ipaddress => $ipaddr,
}
openiosds::namespace {'OPENIO':
  ns                       => 'OPENIO',
  conscience_url           => "192.168.2.91:6000",
  zookeeper_url            => "192.168.2.91:6005,192.168.2.92:6005,192.168.2.93:6005",
  oioproxy_url             => "${ipaddr}:6006",
  eventagent_url           => "beanstalk://${ipaddr}:6014",
  ns_service_update_policy => {'meta2'=>'KEEP|3|1|','sqlx'=>'KEEP|3|1|','rdir'=>'KEEP|1|1|user_is_a_service=rawx'},
  ns_storage_policy        => 'THREECOPIES',
}
openiosds::account {'account-0':
  ns                   => 'OPENIO',
  ipaddress            => $ipaddr,
  sentinel_hosts       => '192.168.2.91:6012,192.168.2.92:6012,192.168.2.93:6012',
  sentinel_master_name => 'OPENIO-master-1',
}
openiosds::meta0 {'meta0-0':
  ns        => 'OPENIO',
  ipaddress => $ipaddr,
}
openiosds::meta1 {'meta1-0':
  ns        => 'OPENIO',
  ipaddress => $ipaddr,
}
openiosds::meta2 {'meta2-0':
  ns        => 'OPENIO',
  ipaddress => $ipaddr,
}
openiosds::rawx {'rawx-0':
  ns        => 'OPENIO',
  ipaddress => $ipaddr,
}
openiosds::oioeventagent {'oio-event-agent-0':
  ns        => 'OPENIO',
  ipaddress => $ipaddr,
}
openiosds::oioproxy {'oioproxy-0':
  ns        => 'OPENIO',
  ipaddress => $ipaddr,
}
openiosds::zookeeper {'zookeeper-0':
  ns        => 'OPENIO',
  ipaddress => $ipaddr,
  servers   => ['192.168.2.91:2888:3888','192.168.2.92:2888:3888','192.168.2.93:2888:3888'],
}
openiosds::redissentinel {'redissentinel-0':
  ns          => 'OPENIO',
  ipaddress   => $ipaddr,
  master_name => 'OPENIO-master-1',
  redis_host  => "192.168.2.91",
}
openiosds::redis {'redis-0':
  ns        => 'OPENIO',
  ipaddress => $ipaddr,
  slaveof   => '192.168.2.91:6011',
}
openiosds::conscienceagent {'conscienceagent-0':
  ns => 'OPENIO',
}
openiosds::beanstalkd {'beanstalkd-0':
  ns        => 'OPENIO',
  ipaddress => $ipaddr,
}
openiosds::rdir {'rdir-0':
  ns        => 'OPENIO',
  ipaddress => $ipaddr,
}
openiosds::oioblobindexer {'oio-blob-indexer-rawx-0':
  ns => 'OPENIO',
}

#8

The log you gave is for the meta1 not the meta0.


#9

Sorry, I thought they met the same issue. Here’s meta0.

2017-12-21T23:56:13.675060-05:00 io-x01 OIO,OPENIO,meta0,0: warning  2143 7B05 log WRN oio.m0v2 Failed to connect to zookeeper, zk code (-4)
2017-12-21T23:56:13.677823-05:00 io-x01 OIO,OPENIO,meta0,0: warning  2143 7B05 log WRN oio.m0v2 Failed to register meta0 [192.168.2.91:6001] to zookeeper: Failed to create Zk node [/hc/ns/OPENIO/srv/meta0/BAAC62E64514AE1C1CCA60C10F610CBC24BE7299EF117DCD65C617CF1DE962E9]: (-4) connection loss
2017-12-21T23:56:14.680392-05:00 io-x01 OIO,OPENIO,meta0,0: warning  2143 7B05 log WRN oio.m0v2 Failed to register meta0 [192.168.2.91:6001] to zookeeper: Failed to create Zk node [/hc/ns/OPENIO/srv/meta0/BAAC62E64514AE1C1CCA60C10F610CBC24BE7299EF117DCD65C617CF1DE962E9]: (-4) connection loss
2017-12-21T23:56:15.106782-05:00 io-x01 OIO,OPENIO,meta0,0: notice  2506 6A19 log NOT oio.sqlx TFO[1] UDP[1] OUT[0]
2017-12-21T23:56:15.107202-05:00 io-x01 OIO,OPENIO,meta0,0: info  2506 6A19 log INF oio.sqlx FD limits set to ACTIVES[21835] PASSIVES[21834] BASES[21835/32768] SYS[65536]
2017-12-21T23:56:15.115443-05:00 io-x01 OIO,OPENIO,meta0,0: info  2506 6A19 log INF oio.sqlite Election delays:
2017-12-21T23:56:15.115646-05:00 io-x01 OIO,OPENIO,meta0,0: info  2506 6A19 log INF oio.sqlite - get_status=5000ms but nowait after 900000ms
2017-12-21T23:56:15.115822-05:00 io-x01 OIO,OPENIO,meta0,0: info  2506 6A19 log INF oio.sqlite - expire_SLAVE=900000ms, expire_MASTER=1500000ms, expire_NONE=300000ms
2017-12-21T23:56:15.115993-05:00 io-x01 OIO,OPENIO,meta0,0: info  2506 6A19 log INF oio.sqlite - retry_failed=2000ms
2017-12-21T23:56:15.116160-05:00 io-x01 OIO,OPENIO,meta0,0: info  2506 6A19 log INF oio.sqlite - ping_final=900000ms
2017-12-21T23:56:15.116352-05:00 io-x01 OIO,OPENIO,meta0,0: info  2506 6A19 log INF oio.sqlite Schema configured for type [meta0]
2017-12-21T23:56:15.335560-05:00 io-x01 OIO,OPENIO,meta0,0: warning  2506 6A19 log WRN oio.m0v2 Failed to connect to zookeeper, zk code (-4)
2017-12-21T23:56:15.336053-05:00 io-x01 OIO,OPENIO,meta0,0: warning  2506 6A19 log WRN oio.m0v2 Failed to register meta0 [192.168.2.91:6001] to zookeeper: Failed to create Zk node [/hc/ns/OPENIO/srv/meta0/BAAC62E64514AE1C1CCA60C10F610CBC24BE7299EF117DCD65C617CF1DE962E9]: (-4) connection loss
2017-12-21T23:56:16.338217-05:00 io-x01 OIO,OPENIO,meta0,0: warning  2506 6A19 log WRN oio.m0v2 Failed to register meta0 [192.168.2.91:6001] to zookeeper: Failed to create Zk node [/hc/ns/OPENIO/srv/meta0/BAAC62E64514AE1C1CCA60C10F610CBC24BE7299EF117DCD65C617CF1DE962E9]: (-4) connection loss

#10

Try to run on server1: gridinit_cmd start OPENIO-meta0-0


#11
[root@io-x01 log]# gridinit_cmd start OPENIO-meta0-0
ALREADY 	OPENIO-meta0-0	Operation already in progress
[root@io-x01 log]# 
[root@io-x01 log]# gridinit_cmd status
KEY                       STATUS      PID GROUP
OPENIO-account-0          UP         2572 OPENIO,account,account-0
OPENIO-beanstalkd-0       UP         2519 OPENIO,beanstalkd,beanstalkd-0
OPENIO-conscience-0       UP         2561 OPENIO,conscience,conscience-0
OPENIO-conscienceagent-0  UP         2353 OPENIO,conscienceagent,conscienceagent-0
OPENIO-meta0-0            UP        12509 OPENIO,meta0,meta0-0
OPENIO-meta1-0            UP         2495 OPENIO,meta1,meta1-0
OPENIO-meta2-0            UP         2573 OPENIO,meta2,meta2-0
OPENIO-oio-blob-indexer-0 UP         2362 OPENIO,oio-blob-indexer,oio-blob-indexer-0
OPENIO-oio-event-agent-0  UP         2587 OPENIO,oio-event-agent,oio-event-agent-0
OPENIO-oioproxy-0         UP         2600 OPENIO,oioproxy,oioproxy-0
OPENIO-rawx-0             UP         2341 OPENIO,rawx,rawx-0
OPENIO-rdir-0             UP         2601 OPENIO,rdir,rdir-0
OPENIO-redis-0            UP         2463 OPENIO,redis,redis-0
OPENIO-redissentinel-0    UP         2544 OPENIO,redissentinel,redissentinel-0
OPENIO-zookeeper-0        UP         2547 OPENIO,zookeeper,zookeeper-0

OK, it looks different now. How about server 2 and 3 ?


#12

If server2 and server3 have all services as UP you can continue the installation process. Otherwise run the same command on these servers as well.


#13

Hi Julien,

Thanks for help! It looks cluster is well done.

[root@io-x01 ~]# openio --oio-ns OPENIO cluster list
+---------+-------------------+---------------------------------+----------+-------+------+-------+
| Type    | Id                | Volume                          | Location | Slots | Up   | Score |
+---------+-------------------+---------------------------------+----------+-------+------+-------+
| rdir    | 192.168.2.92:6010 | /var/lib/oio/sds/OPENIO/rdir-0  | io-x02   | n/a   | True |   100 |
| rdir    | 192.168.2.93:6010 | /var/lib/oio/sds/OPENIO/rdir-0  | io-x03   | n/a   | True |   100 |
| rdir    | 192.168.2.91:6010 | /var/lib/oio/sds/OPENIO/rdir-0  | io-x01   | n/a   | True |   100 |
| account | 192.168.2.92:6009 | n/a                             | io-x02   | n/a   | True |   100 |
| account | 192.168.2.93:6009 | n/a                             | io-x03   | n/a   | True |    99 |
| account | 192.168.2.91:6009 | n/a                             | io-x01   | n/a   | True |   100 |
| rawx    | 192.168.2.92:6004 | /var/lib/oio/sds/OPENIO/rawx-0  | io-x02   | n/a   | True |    94 |
| rawx    | 192.168.2.93:6004 | /var/lib/oio/sds/OPENIO/rawx-0  | io-x03   | n/a   | True |    94 |
| rawx    | 192.168.2.91:6004 | /var/lib/oio/sds/OPENIO/rawx-0  | io-x01   | n/a   | True |    93 |
| meta2   | 192.168.2.92:6003 | /var/lib/oio/sds/OPENIO/meta2-0 | io-x02   | n/a   | True |    94 |
| meta2   | 192.168.2.93:6003 | /var/lib/oio/sds/OPENIO/meta2-0 | io-x03   | n/a   | True |    94 |
| meta2   | 192.168.2.91:6003 | /var/lib/oio/sds/OPENIO/meta2-0 | io-x01   | n/a   | True |    93 |
| meta1   | 192.168.2.92:6002 | /var/lib/oio/sds/OPENIO/meta1-0 | io-x02   | n/a   | True |    94 |
| meta1   | 192.168.2.93:6002 | /var/lib/oio/sds/OPENIO/meta1-0 | io-x03   | n/a   | True |    94 |
| meta1   | 192.168.2.91:6002 | /var/lib/oio/sds/OPENIO/meta1-0 | io-x01   | n/a   | True |    93 |
| meta0   | 192.168.2.92:6001 | /var/lib/oio/sds/OPENIO/meta0-0 | io-x02   | n/a   | True |   100 |
| meta0   | 192.168.2.93:6001 | /var/lib/oio/sds/OPENIO/meta0-0 | io-x03   | n/a   | True |   100 |
| meta0   | 192.168.2.91:6001 | /var/lib/oio/sds/OPENIO/meta0-0 | io-x01   | n/a   | True |    99 |
+---------+-------------------+---------------------------------+----------+-------+------+-------+

But one more question comes. When I accessed the proxy at http://192.168.2.91:6006
, it returned 404:

{“status”:404,“message”:“No handler found”}


#14

Hi all,

I got the same issue ( {“status”:404,“message”:“No handler found”} ) when i tried to access the proxy url http://ip_proxy:6006.

What are i missed ?

Any help would be appreciate.

Cheers.

JSZ


#15

Hello jsz.

The oio-proxy isn’t meant to be interacted with via the HTTP API externally, nor does it provide any kind of web user interface. Its purpose is in fact mostly internal, which seems counter intuitive considering the name of the service.

In order to interact with the store, please use either one of the available SDKs (the OpenIO CLI uses the python SDK internally), or a connector (Swift/S3).


Error 404 message when connecting to the Proxy - install process - Centos 7
#16

Ok.
Thanks for your reply.

I’ll take a look at S3 connector to interact with.

Have a nice day.


#17

Everytime after I ran zk-bootstrap.py OPENIO, I always met
OPENIO-meta0-0 BROKEN -1 OPENIO,meta0,meta0-0

Is this an issue on installation? Is it possible to fix it?


#19

Hello,

Just found this message keeps on showing:

Mar 19 03:56:19 oio-27 OIO,OPENIO,meta0,0[13886]: 13886 1A35 log WRN oio.sqlite TIMEOUT! (waiting for election status) [OPENIO.meta0] step=0/NONE
Mar 19 03:56:19 oio-27 OIO,OPENIO,meta0,0[13886]: 13886 1A35 log WRN oio.m0v2 META0 reload failed : (503) Reload error: Loading error: Open/Lock error: Election failed [OPENIO][meta0]
Mar 19 03:56:19 oio-27 OIO,OPENIO,meta0,0[13886]: 13886 1A35 access INF 192.168.2.27:6001 192.168.2.27:45516 M0_ALL 200 20651326 67 - 5E35B51517DE485A84BC9888C509C230 t=20651231

Any idea?

Best regards,
Yongsheng


#20

I had the same issue exactly.
Centos7.
Following the documentation http://docs.openio.io/17.04/install-guide-centos/installation.html#initialize

paragraph : Package Installation and Service Configuration
when I check that everything went well by using “sudo gridinit_cmd status” everything is up except for OPENIO-meta0-0 which is reported as “broken” on all my 3 nodes.

At that stage I did reboot all the node before moving forward
After the reboot everything appears “UP” and I was able to finish the installation process.


#21

The steps meta0 service initialization and meta0 and meta1 restart may do the work to finish initialization, but OPENIO-meta0-0 BROKEN ahead still causes people upset.