Mailing List Archive

openssh-based file transfers (e.g. rsync, scp, ...) are running 40 (!!) times faster via IPv4 than IPv6
openssh 7.6p1-lp150.7.4 on OpenSuse Leap 15 (both server and client)



Hi all,


first post to list, hopefully on-topic. Haven't found anything on the net, tried to ask at first in OpenSuse forums a
while ago (https://forums.opensuse.org/showthread.php/533588-rsnapshot-rsync-massive-performance-decrease) and today
opened a bug in OpenSuse's Bugzilla (https://bugzilla.opensuse.org/show_bug.cgi?id=1115075)

As written in bug report, as general network issues could be excluded, any hints where to start, how to narrow down if
this is a specific openssh issue or something else?


Thanks in advance,
Michael
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
Re: openssh-based file transfers (e.g. rsync, scp, ...) are running 40 (!!) times faster via IPv4 than IPv6 [ In reply to ]
Il giorno mer 7 nov 2018 alle ore 17:58 michaelof@rocketmail.com
<michaelof@rocketmail.com> ha scritto:
>
> openssh 7.6p1-lp150.7.4 on OpenSuse Leap 15 (both server and client)
>
>
>
> Hi all,
>
>
> first post to list, hopefully on-topic. Haven't found anything on the net, tried to ask at first in OpenSuse forums a
> while ago (https://forums.opensuse.org/showthread.php/533588-rsnapshot-rsync-massive-performance-decrease) and today
> opened a bug in OpenSuse's Bugzilla (https://bugzilla.opensuse.org/show_bug.cgi?id=1115075)
>
> As written in bug report, as general network issues could be excluded, any hints where to start, how to narrow down if
> this is a specific openssh issue or something else?
>
>
> Thanks in advance,
> Michael


Can you try the same transfer with a (totally) different protocol,
like FTP, NFS, CIFS?
This is to exclude any network-related slowdown.

--
Vincenzo Romano - NotOrAnd.IT
Information Technologies
--
NON QVIETIS MARIBVS NAVTA PERITVS
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
Re: openssh-based file transfers (e.g. rsync, scp, ...) are running 40 (!!) times faster via IPv4 than IPv6 [ In reply to ]
Vincenzo,


thanks for answering !!!

As I found out before that this slow down only happens for encrypted transmissions, I've followed your suggestion and
tested with a https-based download, from my Nextcloud instance on my VPS.

Same slow speed when connecting via IPv6, and as fast as expected when connecting via IPv4.

Just to confirm: There's NO dependency/relation between openssh and https, on Linux, correct? (Web Server is Apache)
If NO, sorry for wasting your time, and again THANK YOU for leading me into another direction!


Regards,
Michael


Am 07.11.18 um 19:05 schrieb Vincenzo Romano:
> Il giorno mer 7 nov 2018 alle ore 17:58 michaelof@rocketmail.com
> <michaelof@rocketmail.com> ha scritto:
>>
>> openssh 7.6p1-lp150.7.4 on OpenSuse Leap 15 (both server and client)
>>
>>
>>
>> Hi all,
>>
>>
>> first post to list, hopefully on-topic. Haven't found anything on the net, tried to ask at first in OpenSuse forums a
>> while ago (https://forums.opensuse.org/showthread.php/533588-rsnapshot-rsync-massive-performance-decrease) and today
>> opened a bug in OpenSuse's Bugzilla (https://bugzilla.opensuse.org/show_bug.cgi?id=1115075)
>>
>> As written in bug report, as general network issues could be excluded, any hints where to start, how to narrow down if
>> this is a specific openssh issue or something else?
>>
>>
>> Thanks in advance,
>> Michael
>
>
> Can you try the same transfer with a (totally) different protocol,
> like FTP, NFS, CIFS?
> This is to exclude any network-related slowdown.
>
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
Re: openssh-based file transfers (e.g. rsync, scp, ...) are running 40 (!!) times faster via IPv4 than IPv6 [ In reply to ]
Il giorno mer 7 nov 2018 alle ore 19:30 michaelof@rocketmail.com
<michaelof@rocketmail.com> ha scritto:
>
> Vincenzo,
>
>
> thanks for answering !!!
>
> As I found out before that this slow down only happens for encrypted transmissions, I've followed your suggestion and
> tested with a https-based download, from my Nextcloud instance on my VPS.
>
> Same slow speed when connecting via IPv6, and as fast as expected when connecting via IPv4.
>
> Just to confirm: There's NO dependency/relation between openssh and https, on Linux, correct? (Web Server is Apache)
> If NO, sorry for wasting your time, and again THANK YOU for leading me into another direction!
>
>
> Regards,
> Michael

libssl could be a common factor.
Try FTP, please.

>
>
> Am 07.11.18 um 19:05 schrieb Vincenzo Romano:
> > Il giorno mer 7 nov 2018 alle ore 17:58 michaelof@rocketmail.com
> > <michaelof@rocketmail.com> ha scritto:
> >>
> >> openssh 7.6p1-lp150.7.4 on OpenSuse Leap 15 (both server and client)
> >>
> >>
> >>
> >> Hi all,
> >>
> >>
> >> first post to list, hopefully on-topic. Haven't found anything on the net, tried to ask at first in OpenSuse forums a
> >> while ago (https://forums.opensuse.org/showthread.php/533588-rsnapshot-rsync-massive-performance-decrease) and today
> >> opened a bug in OpenSuse's Bugzilla (https://bugzilla.opensuse.org/show_bug.cgi?id=1115075)
> >>
> >> As written in bug report, as general network issues could be excluded, any hints where to start, how to narrow down if
> >> this is a specific openssh issue or something else?
> >>
> >>
> >> Thanks in advance,
> >> Michael
> >
> >
> > Can you try the same transfer with a (totally) different protocol,
> > like FTP, NFS, CIFS?
> > This is to exclude any network-related slowdown.
> >



--
Vincenzo Romano - NotOrAnd.IT
Information Technologies
--
NON QVIETIS MARIBVS NAVTA PERITVS
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
Re: openssh-based file transfers (e.g. rsync, scp, ...) are running 40 (!!) times faster via IPv4 than IPv6 [ In reply to ]
Il giorno mer 7 nov 2018 alle ore 19:38 Vincenzo Romano
<vincenzo.romano@notorand.it> ha scritto:>
> Il giorno mer 7 nov 2018 alle ore 19:30 michaelof@rocketmail.com
> <michaelof@rocketmail.com> ha scritto:
> >
> > Vincenzo,
> >
> >
> > thanks for answering !!!
> >
> > As I found out before that this slow down only happens for encrypted transmissions, I've followed your suggestion and
> > tested with a https-based download, from my Nextcloud instance on my VPS.
> >
> > Same slow speed when connecting via IPv6, and as fast as expected when connecting via IPv4.
> >
> > Just to confirm: There's NO dependency/relation between openssh and https, on Linux, correct? (Web Server is Apache)
> > If NO, sorry for wasting your time, and again THANK YOU for leading me into another direction!
> >
> >
> > Regards,
> > Michael
>
> libssl could be a common factor.
> Try FTP, please.

Nope, libssl isn't a common library.
Again , try FTP.

> > Am 07.11.18 um 19:05 schrieb Vincenzo Romano:
> > > Il giorno mer 7 nov 2018 alle ore 17:58 michaelof@rocketmail.com
> > > <michaelof@rocketmail.com> ha scritto:
> > >>
> > >> openssh 7.6p1-lp150.7.4 on OpenSuse Leap 15 (both server and client)
> > >>
> > >>
> > >>
> > >> Hi all,
> > >>
> > >>
> > >> first post to list, hopefully on-topic. Haven't found anything on the net, tried to ask at first in OpenSuse forums a
> > >> while ago (https://forums.opensuse.org/showthread.php/533588-rsnapshot-rsync-massive-performance-decrease) and today
> > >> opened a bug in OpenSuse's Bugzilla (https://bugzilla.opensuse.org/show_bug.cgi?id=1115075)
> > >>
> > >> As written in bug report, as general network issues could be excluded, any hints where to start, how to narrow down if
> > >> this is a specific openssh issue or something else?
> > >>
> > >>
> > >> Thanks in advance,
> > >> Michael
> > >
> > >
> > > Can you try the same transfer with a (totally) different protocol,
> > > like FTP, NFS, CIFS?
> > > This is to exclude any network-related slowdown.
> > >
>
>
>
> --
> Vincenzo Romano - NotOrAnd.IT
> Information Technologies
> --
> NON QVIETIS MARIBVS NAVTA PERITVS



--
Vincenzo Romano - NotOrAnd.IT
Information Technologies
--
NON QVIETIS MARIBVS NAVTA PERITVS
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
Re: openssh-based file transfers (e.g. rsync, scp, ...) are running 40 (!!) times faster via IPv4 than IPv6 [ In reply to ]
> first post to list, hopefully on-topic. Haven't found anything on the
> net, tried to ask at first in OpenSuse forums a
> while ago
> (https://forums.opensuse.org/showthread.php/533588-rsnapshot-rsync-massive-performance-decrease)
> and today
> opened a bug in OpenSuse's Bugzilla
> (https://bugzilla.opensuse.org/show_bug.cgi?id=1115075)
>
> As written in bug report, as general network issues could be excluded,
> any hints where to start, how to narrow down if
> this is a specific openssh issue or something else?

Your IPv6 _route_ has 3 times the latency:

$ time /usr/sbin/traceroute vserver.domain.tld
...
7 domain.tld (IPv4) 26.002 ms !X 24.859 ms !X 25.230 ms !X

$ time /usr/sbin/traceroute6 vserver.domain.tld
...
8 domain.tld (IPv6) 80.430 ms !X 79.301 ms !X 80.444 ms !X

The big time waster seems to be this link:
6 nug-d-i40-v6.telia.net (2001:2000:3018:8d::1) 32.046 ms 23.850
ms 24.097 ms
7 contabo-ic-305268-ffm-b11.c.telia.net (2001:2000:3080:953::2)
74.829 ms 77.704 ms 78.066 ms

I'll now speculate that you might as well have some MTU mismatches,
meaning that packages have to get split up along the way, further
slowing down your connection.

Do you have native IPv6 all the way, or is there an 6-in-4 tunnel
inbetween (from the fritzbox on)?

Please try
# ping -M do -s <size> <ip>
for sizes between 1300 and 1500 and find the largest (even) packet size
that still works, and then do the same with ping6 as well.

Also note whether some packetsize works, but just larger ones vanish
along the way.
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
Re: openssh-based file transfers (e.g. rsync, scp, ...) are running 40 (!!) times faster via IPv4 than IPv6 [ In reply to ]
Servus Philipp,


Unfortunately the traceroute(6) results are both more or less random. Sometimes traceroute "hangs" a while, wherever,
sometimes traceroute6. Sometimes traceroute is faster, sometimes traceroute6. Not reliable.

Your MTU question, tried as adviced:

Maximum size for IPv4 is 1466, and max size for IPv6 is 1444. Exceeding these values leads to a "ping: local error:
Message too long, mtu=1492" in both cases.

My Fritzbox says "Native IPv6", no tunnel, no underlying "IPv4 via DS-Lite". These are the default settings for ISP
"Deutsche Telekom", VDSL-50.


VINCENZO: Tried as adviced to test with FTP, installed pure-ftpd and opened VPS's ftp port in firewall. Maximum IPv4
speed was much slower (1.5-2 MB/sec) than other tests, maybe temporary issue or virtualization. But again: IPv6 was MUCH
slower than IPv4.





Am 07.11.18 um 20:57 schrieb Philipp Marek:
>
>> first post to list, hopefully on-topic. Haven't found anything on the
>> net, tried to ask at first in OpenSuse forums a
>> while ago
>> (https://forums.opensuse.org/showthread.php/533588-rsnapshot-rsync-massive-performance-decrease)
>> and today
>> opened a bug in OpenSuse's Bugzilla
>> (https://bugzilla.opensuse.org/show_bug.cgi?id=1115075)
>>
>> As written in bug report, as general network issues could be excluded,
>> any hints where to start, how to narrow down if
>> this is a specific openssh issue or something else?
>
> Your IPv6 _route_ has 3 times the latency:
>
>   $ time /usr/sbin/traceroute vserver.domain.tld
>   ...
>    7  domain.tld (IPv4)  26.002 ms !X  24.859 ms !X  25.230 ms !X
>
>   $ time /usr/sbin/traceroute6 vserver.domain.tld
>   ...
>    8  domain.tld (IPv6)  80.430 ms !X  79.301 ms !X  80.444 ms !X
>
> The big time waster seems to be this link:
>    6  nug-d-i40-v6.telia.net (2001:2000:3018:8d::1)  32.046 ms  23.850 ms  24.097 ms
>    7  contabo-ic-305268-ffm-b11.c.telia.net (2001:2000:3080:953::2)  74.829 ms  77.704 ms  78.066 ms
>
> I'll now speculate that you might as well have some MTU mismatches,
> meaning that packages have to get split up along the way, further
> slowing down your connection.
>
> Do you have native IPv6 all the way, or is there an 6-in-4 tunnel
> inbetween (from the fritzbox on)?
>
> Please try
>    # ping -M do -s <size> <ip>
> for sizes between 1300 and 1500 and find the largest (even) packet size
> that still works, and then do the same with ping6 as well.
>
> Also note whether some packetsize works, but just larger ones vanish
> along the way.
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
Re: openssh-based file transfers (e.g. rsync, scp, ...) are running 40 (!!) times faster via IPv4 than IPv6 [ In reply to ]
> Unfortunately the traceroute(6) results are both more or less random.
> Sometimes traceroute "hangs" a while, wherever,
> sometimes traceroute6. Sometimes traceroute is faster, sometimes
> traceroute6. Not reliable.
That might just as well be DNS issue; the reported latency
values are the interesting data.


> Your MTU question, tried as adviced:
>
> Maximum size for IPv4 is 1466, and max size for IPv6 is 1444.
> Exceeding these values leads to a "ping: local error:
> Message too long, mtu=1492" in both cases.
Hmmm, the IPv6 header is 20 bytes longer than IPv4 -- so there
are still 2 bytes missing.

Please try to set the MTU manually smaller, eg. to 1400,
and test once more.
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
Re: openssh-based file transfers (e.g. rsync, scp, ...) are running 40 (!!) times faster via IPv4 than IPv6 [ In reply to ]
Philipp,


thank you so much for your valuable hints!!

BUT as this list's topic is OpenSSH, and as I now know that my issue has NOTHING to do with OpenSSH, I would like to
stop here, not wasting anyone's time.

Thanks to Vincenco's FTP hint I've now tested with a Debian based "rescue live system" instead of my OpenSuse based VPS,
on server side, offered by my VPS's hosting company. And on the client side with my Android smartphone and the (FOSS)
"Ghost Commander" instead of my OpenSuse box@home. To be sure that my issue has also nothing to do with OpenSuse, as my
preferred Linux flavor.

And in fact it has nothing to do with either OpenSSH or OpenSuse....


Thanks again,
Michael


Am 07.11.18 um 22:18 schrieb Philipp Marek:
>
>> Unfortunately the traceroute(6) results are both more or less random.
>> Sometimes traceroute "hangs" a while, wherever,
>> sometimes traceroute6. Sometimes traceroute is faster, sometimes
>> traceroute6. Not reliable.
> That might just as well be DNS issue; the reported latency
> values are the interesting data.
>
>
>> Your MTU question, tried as adviced:
>>
>> Maximum size for IPv4 is 1466, and max size for IPv6 is 1444.
>> Exceeding these values leads to a "ping: local error:
>> Message too long, mtu=1492" in both cases.
> Hmmm, the IPv6 header is 20 bytes longer than IPv4 -- so there
> are still 2 bytes missing.
>
> Please try to set the MTU manually smaller, eg. to 1400,
> and test once more.
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev