5 Replies Latest reply on Dec 4, 2018 2:27 AM by Intel Corporation

    Tips on improving Jupyter notebooks performance?

    pleasehelpme101

      I know it is not recommended to use the Jupyter notebooks, but rather sting into the cluster. However, the jupyter notebooks resources performance has been subpar. Any recommendations on ways to

       

      Also, uploading a .csv to the jupyter notebooks takes a loooong time. Any suggestions on how to speed this process up? We have 200 GB of free storage, but it is a struggle to even get a few GB's uploaded and in a clean, organized, and normal format.

        • 1. Re: Tips on improving Jupyter notebooks performance?
          Intel Corporation
          This message was posted on behalf of Intel Corporation

          Hi Michael Madigan,

          Thank you for contacting us with your query.

          1. An ssh terminal is preferred over a Jupyter terminal in situations where you need to run a workload that might require long hours to complete. The maximum timeout for the ssh terminal is 24 hours compared to the maximum of 4 hours timeout for a Jupyter notebook. This should not have any impact on the performance otherwise.

          Having said that, could you please elaborate on your first question? It seems to be incomplete.

          2. Uploading a .csv file to the notebook depends a lot on the speed offered by your Internet connection. 
          You can try doing a secure copy (scp) using WinSCP in Windows and scp in Linux. 
          Also, could you please give us the size of the .csv files that you are trying to upload?

          Please let us know if the solutions provided here were helpful.

          Regards,
          Astha

          • 2. Re: Tips on improving Jupyter notebooks performance?
            pleasehelpme101

            Hi Astha,

             

            1. My apologies. It meant to say something along the lines of... "Any recommendations on ways to increase compute power or resource allocation while using the Jupyter notebooks?"

             

            2. I do nothing speed of the internet is the issue here, as my internet is currently listed as having a 289 Mbit/s link speed. I am unable to ssh into the cluster or any external environment from my local machine due to a company firewall. As for the size, they are not small files... I was working on loading a 10GB .csv file, but we were hoping to upload much larger files than that in the near future.

             

            Thank you for the help.

            • 3. Re: Tips on improving Jupyter notebooks performance?
              Intel Corporation
              This message was posted on behalf of Intel Corporation

              Hi Michael,

               

              1. I could suggest optimizations specific to the framework. This would go both for ssh terminal and Jupyter terminal -
              Please follow the given link for optimizations - https://software.intel.com/en-us/articles/tips-to-improve-performance-for-popular-deep-learning-frameworks-on-multi-core-cpus

               

              From the link, you could try a few variations as well such as setting KMP_BLOCK_TIME to 30 or 0 and NUM_PARALLEL_EXEC_UNITS to 6 or 12

               

              2. We will check and get back to you if there are better ways to perform file uploads. In the meanwhile, could you please be a bit more specific on how much time it took for you to upload the .csv, the size of the file and the time taken to upload it.

               


              Regards,
              Astha

              • 4. Re: Tips on improving Jupyter notebooks performance?
                pleasehelpme101

                Hi Astha,

                 

                Thank you for the link, I am hoping that will help us going forward.

                 

                As for the file upload, it timed out before the 4 hours that we are limited to with the Jupyter Notebooks and it was 12GB at the time.

                 

                Thank you for your help!

                • 5. Re: Tips on improving Jupyter notebooks performance?
                  Intel Corporation
                  This message was posted on behalf of Intel Corporation

                  Hi,

                  I hope the Intel SME interaction has helped with the issue.

                  We will be closing the thread now. Please open another thread in case you are facing any other issue.

                  Regards,
                  Astha