FTP Repository archive connection reset

[valentinene]

Hello,

I have an ftp server on which I hold several bigger p2 respositories, eclipse stuff mostly, cdt, updates etc as .zip archives.
In my project I set the tags with the p2 type and the URL with the schema: jar:ftp://MY_USERNAME:MY_PASSWORD@MY_FTP_SERVER_IP/eclipse.org/cdt-10.3.zip!/
I am using tycho 3.0.1 and I am behind a corporate proxy, I have added the proxy settings in my settings.xml, and I also specify these variables on the command line: -Dftp.proxyHost=MY_HOST -Dftp.proxyPort=8080 but when it's time to actually add my repositories ( [INFO] Adding repository jar:ftp://..) I get the following error:

org.eclipse.equinox.p2.core.ProvisionException: Communication with repository at jar:ftp:// etc... failed.
at org.eclipse.equinox.internal.p2.repository.CacheManager.createCache (CacheManager.java:206)
    at org.eclipse.tycho.p2.remote.RemoteRepositoryCacheManager.createCache (RemoteRepositoryCacheManager.java:69)
    at org.eclipse.equinox.internal.p2.metadata.repository.SimpleMetadataRepositoryFactory.getLocalFile 
   .
   .
Caused by: java.net.SocketTimeoutException: Read timed out
    at sun.nio.ch.NioSocketImpl.timedRead (NioSocketImpl.java:273)
    at sun.nio.ch.NioSocketImpl.implRead (NioSocketImpl.java:299)
    at sun.nio.ch.NioSocketImpl.read (NioSocketImpl.java:340)
    at sun.nio.ch.NioSocketImpl$1.read (NioSocketImpl.java:789)
    at java.net.Socket$SocketInputStream.read (Socket.java:1025)
    at java.io.BufferedInputStream.read1 (BufferedInputStream.java:308)
    at java.io.BufferedInputStream.implRead (BufferedInputStream.java:382)
    at java.io.BufferedInputStream.read (BufferedInputStream.java:361)
    at java.io.FilterInputStream.read (FilterInputStream.java:119)
    at java.io.InputStream.transferTo (InputStream.java:790)
    at java.nio.file.Files.copy (Files.java:3170)
    at sun.net.www.protocol.jar.URLJarFile$1.run (URLJarFile.java:219)
    at sun.net.www.protocol.jar.URLJarFile$1.run (URLJarFile.java:215)
    at java.security.AccessController.doPrivileged (AccessController.java:569)
    at sun.net.www.protocol.jar.URLJarFile.retrieve (URLJarFile.java:214)
    at sun.net.www.protocol.jar.URLJarFile.getJarFile (URLJarFile.java:74)
    .
    .

If I also add -Dsun.net.client.defaultReadTimeout=3000000 on the command line then the error message becomes:

org.apache.maven.InternalErrorException: Internal error: java.lang.RuntimeException: 
Failed to load p2 repository with ID 'cdt-10.3' from location 
jar:ftp://MY_USERNAME:MY_PASSWORD@MY_FTP_SERVER_IP/eclipse.org/cdt-10.3.zip!/
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:120)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:960)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:293)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:196)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
    at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:566)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:282)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:225)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:406)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:347)
Caused by: java.lang.RuntimeException: Failed to load p2 repository with ID 'cdt-10.3' from location 
jar:ftp://MY_USERNAME:MY_PASSWORD@MY_FTP_SERVER_IP/eclipse.org/cdt-10.3.zip!/
    at org.eclipse.tycho.p2.target.TargetPlatformFactoryImpl.loadMetadataRepository (TargetPlatformFactoryImpl.java:308)
    at org.eclipse.tycho.p2.target.TargetPlatformFactoryImpl.gatherExternalInstallableUnits (TargetPlatformFactoryImpl.java:276)
    at org.eclipse.tycho.p2.target.TargetPlatformFactoryImpl.createTargetPlatform (TargetPlatformFactoryImpl.java:178)
    at org.eclipse.tycho.p2.target.TargetPlatformFactoryImpl.createTargetPlatform (TargetPlatformFactoryImpl.java:130)
    .
    .
Caused by: java.net.SocketException: Connection reset
    at java.net.SocketInputStream.read (SocketInputStream.java:186)
    at java.net.SocketInputStream.read (SocketInputStream.java:140)
    at java.io.BufferedInputStream.read1 (BufferedInputStream.java:290)
    at java.io.BufferedInputStream.read (BufferedInputStream.java:351)
    at sun.net.www.MeteredStream.read (MeteredStream.java:134)
    at java.io.FilterInputStream.read (FilterInputStream.java:133)
    at sun.net.www.protocol.http.HttpURLConnection$HttpInputStream.read (HttpURLConnection.java:3510)
    at java.io.InputStream.transferTo (InputStream.java:704)
    at java.nio.file.Files.copy (Files.java:3078)
    at sun.net.www.protocol.jar.URLJarFile$1.run (URLJarFile.java:218)
    at sun.net.www.protocol.jar.URLJarFile$1.run (URLJarFile.java:214)
    at java.security.AccessController.doPrivileged (Native Method)
    at sun.net.www.protocol.jar.URLJarFile.retrieve (URLJarFile.java:213)
    at sun.net.www.protocol.jar.URLJarFile.getJarFile (URLJarFile.java:74) 
    .
    .

Downloading over FTP in this way works for smaller archives and also I am not on a very slow network.
One thing I noticed which I am not sure it affects anything is that these p2 repositories don't have the XZ archived artifact and content xml's.
If I unzip the files on the FTP server and then point the URL to that folder instead everything works correctly.
Is there any other timeout flag that I need to set for archive repositories over FTP?
Could there be anything else affecting this repository download over FTP?

[laeubi]

I have an ftp server on which I hold several bigger p2 respositories, eclipse stuff mostly, cdt, updates etc as .zip archives.

Is there any reason to not explode them into individual files? Can you probably even make the FTP folder accessible via http?

In my project I set the tags with the p2 type and the URL with the schema: jar:ftp://MY_USERNAME:MY_PASSWORD@MY_FTP_SERVER_IP/eclipse.org/cdt-10.3.zip!/

While this seems to work in the first place, this is actually really bad and should be avoided. This will lead to P2 downloading the whole file multiple times and can really slow down things a lot.

That's mostly the reason for the problems you see.

Is there any other timeout flag that I need to set for archive repositories over FTP?
Could there be anything else affecting this repository download over FTP?

FTP servers are not very common today (for different reasons) and most people use http instead, so this is the best supported protocol. Tycho has only limited support for FTP but if this is crucial to your business and likes improve that you can consider a sponsoring to allow me to assign more time-slots particular issue, or you can contact me directly for an individual contract e.g. if you need first class FTP support.

[valentinene]

It's not that crucial. I think we will move away from this approach since it is not doing us any favours. Thanks!