Join the Community
and take part in the story

Download limit oio_sds_download = 10MB?


#1

Hello,

Am writing an application that would upload content into OpenIO SDS. I have been able to upload content of all sizes to the OpenIO SDS cluster. However the download via oio_sds_download (sequential, buffer, or file) all of them seem to stop the download at 10MB exactly (10485760) bytes.

My OpenIO Server is running on Ubuntu xenial (3.2) and client also on Ubuntu Xenial (different host) with 3.2 libraries (compiled from source).

>     #0  _download (dl=0x7ffc3c511c40) at ...deps/oio-sds/core/sds.c:919
>     #1  0x00007f886412b266 in _download_to_hook (sds=0xc7b590, src=0x7ffc3c511db0, dst=0x7ffc3c511d10) at .../oio-sds/core/sds.c:1010
>     #2  0x00007f886412b410 in _download_to_file (sds=0xc7b590, src=0x7ffc3c511db0, dst=0x7ffc3c511dc0) at .../oio-sds/core/sds.c:1046
>     #3  0x00007f886412b7a0 in oio_sds_download (sds=0xc7b590, dl=0x7ffc3c511db0, snk=0x7ffc3c511dc0) at .../io-sds/core/sds.c:1132
>     #4  0x00007f886412b89b in oio_sds_download_to_file (sds=0xc7b590, url=0xc7cac0, local=0xbcdc18 "/tmp/b6d0c620.tmp .../oio-sds/core/sds.c:1153

The value of total in _download after computing metachunks always results in 10MB while my file size is around 20 mb.

Is this a bug or another issue.

My snippet looks like this -

oio_url_set(url, OIOURL_PATH, key);
outfile = key;
err = oio_sds_download_to_file(sds_handle, url, outfile);
gerror = (GError*)err;
if (gerror != NULL) {
    log_error(...)
}

Why is there a cut-off at 10MB for the file download.

Thank you.


#2

Hello,

I just reproduced your problem. The bug is during the upload: only the first chunk is registered in meta2 database. I’m investigating.

Notice that there is no problem when using erasure coding.


#3

Hello,

I have fixed the bug with this commit. It will be included in openio-sds 3.2.4 et 3.3.1. The packages for Ubuntu should be available soon (the packages for CentOS are already available).

Regards,

Florent


#4

Hi @linuxmonk,

packages are available in our repositories, thank you for you help !


#5

Thank you @fvennetier and @racciari. I will run a git update and test them here.


#6

Hello @fvennetier, @racciari

I ran a test today after an apt-get update on both my client (where my mail server is running) and the OpenIO cluster. I can see openio --version gives 3.2.4 on both machines. I still see the issue where the file gets cut off at exactly 10 MB.

Is there any log you can see to confirm if the issue is fixed / is the same as am reporting and so on.


#7

Hello @linuxmonk.

As explained in my first message, the bug was during upload. So all 10MB+ objects you uploaded before the update will be incompletely downloaded.

Diagnosis

You can check if an object is affected by running openio object locate <CONTAINER> <OBJECT>. This will print the list of chunks of the object. For an object bigger than 10MB, if you see only one position (first column), the object is affected.
Here is an example of an incomplete object (notice that the chunk size on my cluster is 1MB, not 10MB, and I have 3 copies of each chunk):

> openio object locate FVE0 myobject
+-----+----------------------------------------------------------------------------------------+---------+----------------------------------+
| Pos | Id                                                                                     |    Size | Hash                             |
+-----+----------------------------------------------------------------------------------------+---------+----------------------------------+
| 0.0 | http://127.0.0.1:6024/9B5724CFBD3273C34F8E5C6E7D5B0800F9726517DBB4CB3B0D71FAD9E6606BFF | 1048576 | 50F5DF222D92CD558E6014CCB91DFF94 |
| 0.0 | http://127.0.0.1:6021/6F8D72ED3D07D4A14839F5B4091208ABA4DFBC88D1E7F0BFF914AF750378D1C2 | 1048576 | 50F5DF222D92CD558E6014CCB91DFF94 |
| 0.0 | http://127.0.0.1:6014/FFE888C31B8BF338E0E99424C87E8B67FDE2BDB25E790ED76526D1A9A557DB1E | 1048576 | 50F5DF222D92CD558E6014CCB91DFF94 |
+-----+----------------------------------------------------------------------------------------+---------+----------------------------------+

Normally you should see one position per 10MB (per 1MB in my case):

> openio object locate FVE0 myobject2
+-----+----------------------------------------------------------------------------------------+---------+----------------------------------+
| Pos | Id                                                                                     |    Size | Hash                             |
+-----+----------------------------------------------------------------------------------------+---------+----------------------------------+
| 0.0 | http://127.0.0.1:6022/3A172F6014252841F3D5742EB8BCE0436B732123C0ECFC4E73B487F3C1EF1F6A | 1048576 | 50F5DF222D92CD558E6014CCB91DFF94 |
| 0.0 | http://127.0.0.1:6024/4A7BD20FE9794E15BF85DF1310BDC0707522D70B33820B02F38211E191CAD53E | 1048576 | 50F5DF222D92CD558E6014CCB91DFF94 |
| 0.0 | http://127.0.0.1:6015/27ED6075D06F6155EDE510F186099E582B02E4739ABEACBDFCB5956DE617B509 | 1048576 | 50F5DF222D92CD558E6014CCB91DFF94 |
| 1.0 | http://127.0.0.1:6019/47DD2ED52B1EB67114006C6E1044F374B1690350A3D99533A7C8A5486DFE3EF4 |  524288 | 194C0D761A700658C081D4037E9BCF0E |
| 1.0 | http://127.0.0.1:6021/4903BF37B9886DCD9E9BB4F9183DC17963EDC1B3F1FCF8BD7BDFE7D4B3A681B5 |  524288 | 194C0D761A700658C081D4037E9BCF0E |
| 1.0 | http://127.0.0.1:6024/6CB8740CAB164C1510D79B7309AA136C21797B9FEB9A96248EA33A91E7EDEBA0 |  524288 | 194C0D761A700658C081D4037E9BCF0E |
+-----+----------------------------------------------------------------------------------------+---------+----------------------------------+

Solution

To fix this, there is a tool called oio-blob-registrator, that registers the orphan chunks into the meta2 databases. You will have to run it on each rawx volume (on the machine hosting the volume). To get the path to the rawx volumes, you can run:

openio cluster list rawx

Then run:

oio-blob-registrator <NAMESPACE> <VOLUME PATH> --insert

If you run openio object locate again, you should see new chunks appear.

> openio object locate FVE0 myobject
+-----+----------------------------------------------------------------------------------------+---------+----------------------------------+
| Pos | Id                                                                                     |    Size | Hash                             |
+-----+----------------------------------------------------------------------------------------+---------+----------------------------------+
| 0.0 | http://127.0.0.1:6014/2DFEE2161D1EEC66CDE23D129D3FD1581F82710FAAA9C981E48153043D092AA4 | 1048576 | 50F5DF222D92CD558E6014CCB91DFF94 |
| 0.0 | http://127.0.0.1:6023/B93AF1438E2FD54AAB29251BC3A7A4D46A0864EBDBA99747CFE82B082D2D6819 | 1048576 | 50F5DF222D92CD558E6014CCB91DFF94 |
| 0.0 | http://127.0.0.1:6022/3344CF7F4BD1CE7CCA5A117D4695C0665FFFCD8D87E6836E264CAA44C1255F65 | 1048576 | 50F5DF222D92CD558E6014CCB91DFF94 |
| 1.0 | http://127.0.0.1:6024/5FA4DBB8C6BA43AFD910A8B4E94D4E3C110C06391CC90A4010F246E040D63AC1 |  524288 | 194C0D761A700658C081D4037E9BCF0E |
+-----+----------------------------------------------------------------------------------------+---------+----------------------------------+

Once there is at least one chunk per position, you can download your object, it should be the correct size.


#8

Hello @fvennetier

Thank you very much for the quick reply. Running oio-blob-registrator fixes the issue.

However, I’d like to bring to your attention that I created a new container after updating to 3.2.4 and had initiated an upload from my tool which uses oio_sds_* APIs to upload the files. The files by themselves were not replicated to meta-2; The upload still appears to have the issue in 3.2.4 ?


#9

Hello @linuxmonk,

did you update the package on your client ? The problem is located on the client side, to solve this issue, you have to update on the client side and restart your application (if it’s a daemon).


#10

Hello @racciari

Yes, I have updated openio on client and on the server. The upload size on the object list shows ~14 MB and the MD5 Sum of the object compared to my local machine with the one on OIO are the same. It’s just that the replication didn’t happen.


#11

When you talk about replication, I think you mean registration.

I checked the .tar.gz source of the openio-sds-3.2.4 deb packages, and it contains the fix.
Earlier you talked about git. Have you tried to run oio-sds from sources before trying the packages? If so, you may have an old liboiosds.so.0 somewhere in /usr/local/lib (or $CMAKE_INSTALL_PREFIX/lib).

Can you run

ldd <path_to_your_application> | grep liboio

and paste the result here?