5 Replies Latest reply on Dec 7, 2018 5:12 PM by Intel Corporation

    SSD 540s compresses incompressible data?

    yurybx

      Good for you!

      Recently, I bought this drive and decided to test it. I was most interested in his ability to compress data, because in this thread I read about it. In order to evaluate the effectiveness of the compression algorithm, I first wrote compressible data on the SSD, and then wrote incompressible data. In each case, I recorded changes in two SMART indicators: "Total LBA's written" and "Total NAND writes". I expected that in the first case the difference between these indicators will be much greater than in the second. But the indicators behaved similarly in both cases. Why is that? Am I measuring the compression algorithm incorrectly?

      Below I provide detailed information.

      Before the experiment, I filled the drive with high entropy data, leaving 13% of free space. Next, I fixed the SMART indicators, after which I wrote a large file with low entropy (7zip can compress it 10 times) with a size of 49GB. Then I again recorded SMART indicators, after which I deleted the previous file and wrote 7 files with high entropy (compressed with 7zip) with a total size of 42GB.

      In the first case, "Total LBA's written" indicator increased by 49GB, and "Total NAND writes" indicator - by 44GB. In the second case, the first indicator grew by 42GB, the second - by 36GB. That is, in the first case the ratio is 90%, in the second - 86%. Why is the first coefficient so large if data with very low entropy were written? Why is the second coefficient a little less than the first?

        • 1. Re: SSD 540s compresses incompressible data?
          Intel Corporation
          This message was posted on behalf of Intel Corporation

          Hi yurybx,

          Thank you for posting in the Intel® communities.

          This experiment is indeed interesting, and I would like to get more details about it. Is it possible for you to run again the tests with high and low entropy data? This time please get the full SMART reports using the Intel® SSD Toolbox (https://downloadcenter.intel.com/download/28309/Intel-Solid-State-Drive-Toolbox?product=80096)  and share the files here so that I can take a look at them.

          On the other hand, what kind of data/files are you using exactly as low entropy data?

          I’ll be waiting for your response.

          Have a nice day.

          Regards,
          Diego V.
          Intel Customer Support Technician
          Under Contract to Intel Corporation

          • 2. Re: SSD 540s compresses incompressible data?
            yurybx

            I'm using MS SQL 2014 database mdf-file as low entropy data. It has really low entropy because 7zip compresses it from 49GB to about 5GB.

            I repeated the experiment. Information attached to the message.

            Initial.csv - the initial state of the drive,

            low-entropy.csv - the state after writing the low entropy file;

            high-entropy.csv - after deleting the previous file and writing high entropy files.

            • 3. Re: SSD 540s compresses incompressible data?
              Intel Corporation
              This message was posted on behalf of Intel Corporation

              Hi yurybx,

               

              Thank you for the reports.

               

              After checking the results, I find that for all cases, the Write Amplification factor is around 0.26 which is fine. The Write Amplification factor should be less than 1 for the drive to be working fine. In all cases, the “Total LBAs Written” is around 10GB, so I’m not seeing the increases you mentioned in your first post either in the “Total LBAs Written” or in the “Total NAND Writes”.

               

              I have summarized the results in the table below. As you can see, when writing high entropy data, there is a little increase in the Write Amplification factor, but it's basically the same result. This makes me think about what data you are using exactly as high entropy data.

               

              ValueInitialHigh Entropy DataLow Entropy Data
              Total NAND Writes (GB)2660.002739.002704.00
              Total Host Writes (GB)10038.2510129.2510087.47
              Total LBAs Written (GB)10038.2510129.2510087.47
              Write Amplification Factor0.2650.2700.268

               

               

              Regards,
              Diego V.
              Intel Customer Support Technician
              Under Contract to Intel Corporation

              • 4. Re: SSD 540s compresses incompressible data?
                yurybx

                I apologize, apparently I did not fully explain my question. I am not interested in the total Write Amplification factor, but in the local WAF related to the writing of specific files. So in the first part of the experiment, I write the low entropy file and measure how much the SMART-indicators have grown. And in the second part, I do the same with the high entropy files. Then I compare the results. And it is logical to assume that in the first case the local WAF will be very small (as a result of the compression algorithm), and in the second case it will be close to 100% (since the compression algorithm has nothing to compress). But the results is completely inconsistent with what was expected: in the first case, the local WAF is too high (does the compression algorithm fail?), and in the second case it is suspiciously small (does compression of incompressible data occur?).

                Here are my calculations. Tell me, please, what is my mistake?

                 

                Low entropy data writtenHigh entropy data written
                BeforeAfterIncreaseBeforeAfterIncrease
                Total NAND Writes (GB)26602704442704273935
                Total LBAs Written (GB)10038,310087,4749,2210087,510129,2541,78
                local Write Amplification Factor 0,89 0,84
                • 5. Re: SSD 540s compresses incompressible data?
                  Intel Corporation
                  This message was posted on behalf of Intel Corporation

                  Hi yurybx,

                   

                  Thank you for the clarification, I do understand you now.

                   

                  Those are interesting results. I ran some tests with an Intel® SSD 540s Series that I had here (180GB though) and I got similar results to yours. See them below.

                   

                  ValueLow Entropy Data (34.3GB)
                  High Entropy Data (33.3GB)
                  BeforeAfterIncreaseBeforeAfterIncrease
                  Total NAND Writes (GB)971.001006.0035.001006.001038.0032.00
                  Total Host Writes (GB)900.44934.5034.06934.75967.8433.09
                  Total LBAs Written (GB)900.44934.5034.06934.75967.8433.09
                  Write Amplification Factor (Local)1.0280.967



                  I also noticed a checkbox when you format the drive that says “Enable file and folder compression” so I re-ran the tests with this option checked, but even though it’s noticed that the data written was less this time for the low entropy data, the overall relation between the NAND and Host writes (LBAs) is still the same as before.

                   

                  ValueLow Entropy Data (34.3GB)
                  High Entropy Data (33.3GB)
                  BeforeAfterIncreaseBeforeAfterIncrease
                  Total NAND Writes (GB)1056.001076.0020.001076.001108.0032.00
                  Total Host Writes (GB)987.751007.2219.471007.411040.6333.22
                  Total LBAs Written (GB)987.751007.2219.471007.411040.6333.22
                  Write Amplification Factor (Local)1.0270.963



                  This is the checkbox I’m talking about:

                   

                  Checkbox.JPG



                  Anyhow, please let me investigate about this because it looks like we are missing something about how to perform these tests correctly.

                   

                  I’ll share my findings as soon as possible.

                   

                  Have a nice day.

                   

                  Regards,
                  Diego V.
                  Intel Customer Support Technician
                  Under Contract to Intel Corporation