Join the Community
and take part in the story

Migrate rdir data from one location to another?


#1

Hello,

I have rdir data from different volumes on the same location:

~# openio volume assignation
+--------------------+--------------------+---------------+---------------+
| Rdir               | Rawx               | Rdir location | Rawx location |
+--------------------+--------------------+---------------+---------------+
| 192.168.33.11:6010 | 192.168.33.13:6004 | openio-1      | openio-3      |
| 192.168.33.11:6010 | 192.168.33.14:6004 | openio-1      | openio-4      |
| 192.168.33.12:6010 | 192.168.33.11:6004 | openio-2      | openio-1      |
| 192.168.33.13:6010 | 192.168.33.12:6004 | openio-3      | openio-2      |
+--------------------+--------------------+---------------+---------------+

How can I migrate, say, rdir associated with the openio-3 rawx volume from openio-1 to openio-4?

Thanks.


#2

Hello @olc.

In order to migrate the rdir associated with the rawx over at 192.168.33.13:6004, you need to do the following:

  1. Deploy an rdir on the 4th node
  2. Link all the rawx service to the new rdir
  3. Restart oioproxy*, oio-event-agent, and all oio-blob-indexer services linked to the affected rawx in order to trigger an rdir rebuild.
  4. (Optional) decomission the old rdir

* oio-proxy needs to be restarted due to this issue: https://github.com/open-io/oio-sds/issues/1192

Example where the rdir is linked to a single rawx:

  1. Using puppet, add an rdir to the manifest:
$ cat /root/openio.pp
[...]
openiosds::rdir {'rdir-1':
  num       => 1,
  ns        => 'OPENIO',
  ipaddress => "192.168.33.14",
  port      => 6301,
  db_path   => "/var/lib/oio/sds/OPENIO/rdir-1",
}
[...]

(Adapt num, port, and path to available ones)

Then apply the puppet manifest, restart the conscienceagent on the machine, and unlock the score of the new rdir:

$ puppet apply --no-stringify_facts /root/openio.pp
$ gridinit_cmd restart @conscienceagent
$ openio cluster unlock rdir 192.168.33.14:6301 --oio-ns OPENIO
  1. Update the reference between rawx/rdir
$ export OIO_NS=OPENIO
$ openio reference unlink 192.168.33.13:6004 rdir --oio-account _RDIR
$ openio reference force 192.168.33.13:6004 192.168.33.14:6301 rdir --oio-account _RDIR
$ # Check the volume assignation
$ openio volume assignation
  1. On the machine hosting the rawx, restart the proxy, the event-agent, and the blob-indexer of the affected rawx (it should have the same id):
$ # On node 192.168.33.13 (Adapt the ids to the ones given by gridinit_cmd status
$ sudo gridinit_cmd restart OPENIO-oioproxy-1 OPENIO-oio-event-agent-0 OPENIO-oio-blob-indexer-1
$ # Wait for the blob indexer to reindex the chunks (up to 5 mins)
$ tail -f /var/log/oio/sds/OPENIO/oio-blob-indexer-1/oio-blob-indexer-1.log

Check that the new rdir has data:

$ # On node 192.168.33.14
$ find /var/lib/oio/sds/OPENIO/rdir-1/ -name *.log
/var/lib/oio/sds/OPENIO/rdir-1/192.168.33.13:6004/000003.log
  1. Decomission the old rdir (be careful before running some of the following steps)
$ # On machine 192.168.33.11 (adapt rdir id from gridinit_cmd status)
$ gridinit_cmd stop OPENIO-rdir-1
$ rm /etc/gridinit.d/OPENIO-rdir-1
$ rm /etc/oio/sds/OPENIO/watch/rdir-1.yml
$ gridinit_cmd reload && gridinir_cmd restart @conscienceagent
$ openio cluster flush rdir --oio-ns OPENIO && sleep 5 && openio cluster list --oio-ns OPENIO
$ # Remove logs, data, config
$ rm -rf /var/lib/oio/sds/OPENIO/rdir-1
$ rm -rf /var/log/oio/sds/OPENIO/rdir-1
$ rm -rf /etc/oio/sds/OPENIO/rdir-1

Hope this helps :wink:


#3

Thanks a lot for the detailed instructions, @vladimir-openio. It worked without a glitch! Guess that it will be of interest for lots of people though! :slight_smile:

BTW, each time I add a new rawx server in the cluster, it starts with no rdir service associated with. Issuing an openio --oio-ns=OPENIO volume admin bootstrap does this trick, but the new rawx is always assigned to an rdir which is already assigned. That’s not surprising because no free rdir service is available except the one which has been installed/started on the new node.

What could be the best way to deal with this situation?

Tks and best regards,

Olivier