6 Replies Latest reply on Dec 4, 2018 2:27 AM by Intel Corporation

    Trouble SSH into AI Devcloud Cluster

    pleasehelpme101

      I am having trouble SSHing into the cluster and I know it is because of my company firewall, but all of the prescribed fixes on Colfax & these forums have not seem=ed to help me. Are there any workarounds to this or way to work on establishing a secure VPN between us and the cluster?

       

      Thanks in advance for the help

        • 1. Re: Trouble SSH into AI Devcloud Cluster
          Intel Corporation
          This message was posted on behalf of Intel Corporation

          Hi Michael Madigan,

          You would have obtained a username and a private key to your DevCloud account. Following are few of the suggestions that you could make use of -

          1. After downloading and configuring your private SSH key, add the following to your account's ~/.ssh/config file - 

              Host colfax
                  User <user_ID>
                  HostName localhost
                  Port 4022
                  IdentityFile <path_to_colfax-access-key-<user_ID>>
                  ProxyCommand ssh -T colfax-via-proxy

              Host colfax-via-proxy
                  User guest
                  HostName cluster.colfaxresearch.com
                  IdentityFile <path_to_colfax-access-key-<user_ID>>
                  LocalForward 4022 c009:22
                  # (Windows) ProxyCommand connect.exe -S <proxy_name>:<port> %h %p
                  # (Mac) ProxyCommand nc -X 5 -x <proxy_name>:<port> %h %p
                  # (Unix) ProxyCommand nc -x <proxy_name>:<port> %h %p
              Usage: ssh colfax

          2. If 1. doesn't work, you can connect to your system using an open network such as a mobile network.

          3. Another option is to connect to DevCloud using Jupyter notebook. The details for connecting via Jupyter notebook can be found in the Welcome mail that you would have received for account activation.

          You would also find some helpful documents for using the Jupyter notebook at - https://communities.intel.com/thread/127653

          Please let us know if this helped.

          Regards,
          Astha

          • 2. Re: Trouble SSH into AI Devcloud Cluster
            pleasehelpme101

            1. I have previously attempted this (and tried again after seeing this post) and am not able to successfully SSH into it from behind the company firewall. Have been told on my end that I would need to set up a secure link between your environment and ours and gain approval for that... We have a lot of security measure in place.

             

            2. The mobile net work will not work for what we are trying to do and we will not be able to provide enough resources for everyone we are planning on having use this.

             

            3. I have already worked with your Jupyter notebooks, but the compute power is not much better than running things on my local machine, and the internet speeds are not the problem. My internet speed is currently listed as having 289 Mbit/s.

            • 3. Re: Trouble SSH into AI Devcloud Cluster
              Intel Corporation
              This message was posted on behalf of Intel Corporation

              Hi Michael,

              We are afraid that we may not be able to help with the first two points. A good open network or gaining approval from your company is the way to go about it.

              For the third point, there are two things that could be happening - 
              1. The local on which you are working already is a high end machine. 
              2. You are not utilizing the resources to give an optimized performance. For instance, workloads using TensorFlow framework could make use of setting certain environment variables such as OMP_NUM_THREADS, KMP_BLOCKTIME etc. to tune the performance. This depends on the kind of workload that you have, the framework you are using etc. 

              In either case, it would be good if you share the type of workload, framework and any other point that would help us identify the work you are executing. This will help us suggest optimizations specific to the workload. 

              Also, please note that the constraint in Jupyter notebook is only on the time for which the notebook can remain active which should not affect the compute power.

              Regards,
              Astha

              • 4. Re: Trouble SSH into AI Devcloud Cluster
                Intel Corporation
                This message was posted on behalf of Intel Corporation

                Hi Michael,

                Could you please let us know how we can help you further? This will help us in streamlining further actions on this issue.

                Regards,
                Astha

                • 5. Re: Trouble SSH into AI Devcloud Cluster
                  pleasehelpme101

                  Hi Astha,

                   

                  I am using Python Pandas with data that is a bit large, which would be ideal to use Spark instead, but that is not possible with your current offerings.

                   

                  As for the other questions I previously had, it seems that you are correct and this is something that con not necessarily be solved. We will plan on just using the Jupyter notebooks and not try to SSH in.

                   

                  Thank you for your help!

                  • 6. Re: Trouble SSH into AI Devcloud Cluster
                    Intel Corporation
                    This message was posted on behalf of Intel Corporation

                    Hi,

                    I hope the Intel SME interaction has helped with the issue.

                    We will be closing the thread now. Please open another thread in case you are facing any other issue.

                    Regards,
                    Astha