7:58 a.m.
Hi, all
Recently, we use a demo to obverse the latency.
We find that when we access the same block consecutively, the occurrence of maximum
latencies will become more frequent.
Additionally, they can reach even 2-3 ms and present a periodical change.
Why?
(1) For the same block, the latency of the first accessing is about 10-12 μs while the
second, third
and the forth accessing can reach 700-900 μs even 2-3 ms?
I want to know the reason why the operation difference between the first accessing
and the others exists.
(2) Why the maximums of 2-3 ms have a periodical change?
Best wishes,
Jiajia Chu
8:05 a.m.
What you are observing is the characteristics of how the drives firmware handles writes.
This can be from write amplification caused by the drives GC, and journaling +
checkpointing. This behavior may vary greatly between vendors and even firmware
versions.
- Danielle Costantino
________________________________
From: SPDK <spdk-bounces(a)lists.01.org> on behalf of 储 <cjj25233(a)163.com>
Sent: Tuesday, August 1, 2017 12:58:33 AM
To: spdk(a)lists.01.org
Subject: [SPDK] A issue about maximums of write latency when we access the same block
consecutively.
Hi, all
Recently, we use a demo to obverse the latency.
We find that when we access the same block consecutively, the occurrence of maximum
latencies will become more frequent.
Additionally, they can reach even 2-3 ms and present a periodical change.
Why?
(1) For the same block, the latency of the first accessing is about 10-12 μs while the
second, third
and the forth accessing can reach 700-900 μs even 2-3 ms?
I want to know the reason why the operation difference between the first accessing
and the others exists.
(2) Why the maximums of 2-3 ms have a periodical change?
Best wishes,
Jiajia Chu
8:46 a.m.
Hi, Danielle
We also did experiments to access random blocks of 500 times,
the results seem like more stable, most of the latencies are 10-30 μs and the maximum is
284 μs.
We have tested the random method many times and each experiment has similar results.
We wonder that why these same mechanisms, e.g., GC, journaling and checkpointing, cause
different influence between the random and consecutive accessing?
Thanks a lot.
At 2017-08-01 16:05:54, "Danielle Costantino" <dcostantino(a)vmem.com>
wrote:
What you are observing is the characteristics of how the drives firmware handles writes.
This can be from write amplification caused by the drives GC, and journaling +
checkpointing. This behavior may vary greatly between vendors and even firmware versions.
- Danielle Costantino
From: SPDK <spdk-bounces(a)lists.01.org> on behalf of 储 <cjj25233(a)163.com>
Sent: Tuesday, August 1, 2017 12:58:33 AM
To:[email protected]
Subject: [SPDK] A issue about maximums of write latency when we access the same block
consecutively.
Hi, all
Recently, we use a demo to obverse the latency.
We find that when we access the same block consecutively, the occurrence of maximum
latencies will become more frequent.
Additionally, they can reach even 2-3 ms and present a periodical change.
Why?
(1) For the same block, the latency of the first accessing is about 10-12 μs while the
second, third
and the forth accessing can reach 700-900 μs even 2-3 ms?
I want to know the reason why the operation difference between the first accessing
and the others exists.
(2) Why the maximums of 2-3 ms have a periodical change?
Best wishes,
Jiajia Chu
7:41 p.m.
Sounds like Danielle’s theory still holds here… the impact of background tasks in SSD
firmware is at least partially dependent on the location of the operation on the media ,
ie if your write hits an area that is being relocated as part of GC it’s going to suffer
more and as GC is done in localized areas it stands to reason that random traffic will be
less affected as you’re hitting as a smaller % of the IOs will be hitting LBA regions that
GC is affecting. I’m sure there are a host of other factors as well
-Paul
From: SPDK [mailto:[email protected]] On Behalf Of ?
Sent: Tuesday, August 1, 2017 1:47 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] A issue about maximums of write latency when we access the same block
consecutively.
Hi, Danielle
We also did experiments to access random blocks of 500 times,
the results seem like more stable, most of the latencies are 10-30 μs and the maximum is
284 μs.
We have tested the random method many times and each experiment has similar results.
We wonder that why these same mechanisms, e.g., GC, journaling and checkpointing, cause
different influence between the random and consecutive accessing?
Thanks a lot.
At 2017-08-01 16:05:54, "Danielle Costantino" <dcostantino(a)vmem.com>
wrote:
What you are observing is the characteristics of how the drives firmware handles writes.
This can be from write amplification caused by the drives GC, and journaling +
checkpointing. This behavior may vary greatly between vendors and even firmware
versions.
- Danielle Costantino
________________________________
From: SPDK <[email protected]<mailto:[email protected]>> on
behalf of 储 <[email protected]<mailto:[email protected]>>
Sent: Tuesday, August 1, 2017 12:58:33 AM
To: [email protected]<mailto:[email protected]>
Subject: [SPDK] A issue about maximums of write latency when we access the same block
consecutively.
Hi, all
Recently, we use a demo to obverse the latency.
We find that when we access the same block consecutively, the occurrence of maximum
latencies will become more frequent.
Additionally, they can reach even 2-3 ms and present a periodical change.
Why?
(1) For the same block, the latency of the first accessing is about 10-12 μs while the
second, third
and the forth accessing can reach 700-900 μs even 2-3 ms?
I want to know the reason why the operation difference between the first accessing
and the others exists.
(2) Why the maximums of 2-3 ms have a periodical change?
Best wishes,
Jiajia Chu
11:49 p.m.
Hi Jiajia,
I have a bunch of questions that will help me figure out what you are seeing.
1) When you say "access", do you mean read or write? The behavior of these two
operations is quite different.
2) Are you using a NAND based or 3D XPoint based SSD? These again work entirely
differently.
3) When you access the same block repeatedly, what's the delay between each access?
None?
4) For whatever SSD you are using, have you confirmed the firmware is up to date? This can
make a big difference.
Thanks,
Ben
-------- Original message --------
From: "Luse, Paul E" <paul.e.luse(a)intel.com>
Date: 8/1/17 12:42 PM (GMT-07:00)
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] A issue about maximums of write latency when we access the same block
consecutively.
Sounds like Danielle’s theory still holds here… the impact of background tasks in SSD
firmware is at least partially dependent on the location of the operation on the media ,
ie if your write hits an area that is being relocated as part of GC it’s going to suffer
more and as GC is done in localized areas it stands to reason that random traffic will be
less affected as you’re hitting as a smaller % of the IOs will be hitting LBA regions that
GC is affecting. I’m sure there are a host of other factors as well
-Paul
From: SPDK [mailto:[email protected]] On Behalf Of ?
Sent: Tuesday, August 1, 2017 1:47 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] A issue about maximums of write latency when we access the same block
consecutively.
Hi, Danielle
We also did experiments to access random blocks of 500 times,
the results seem like more stable, most of the latencies are 10-30 μs and the maximum is
284 μs.
We have tested the random method many times and each experiment has similar results.
We wonder that why these same mechanisms, e.g., GC, journaling and checkpointing, cause
different influence between the random and consecutive accessing?
Thanks a lot.
At 2017-08-01 16:05:54, "Danielle Costantino" <dcostantino(a)vmem.com>
wrote:
What you are observing is the characteristics of how the drives firmware handles writes.
This can be from write amplification caused by the drives GC, and journaling +
checkpointing. This behavior may vary greatly between vendors and even firmware
versions.
- Danielle Costantino
________________________________
From: SPDK <[email protected]<mailto:[email protected]>> on
behalf of 储 <[email protected]<mailto:[email protected]>>
Sent: Tuesday, August 1, 2017 12:58:33 AM
To: [email protected]<mailto:[email protected]>
Subject: [SPDK] A issue about maximums of write latency when we access the same block
consecutively.
Hi, all
Recently, we use a demo to obverse the latency.
We find that when we access the same block consecutively, the occurrence of maximum
latencies will become more frequent.
Additionally, they can reach even 2-3 ms and present a periodical change.
Why?
(1) For the same block, the latency of the first accessing is about 10-12 μs while the
second, third
and the forth accessing can reach 700-900 μs even 2-3 ms?
I want to know the reason why the operation difference between the first accessing
and the others exists.
(2) Why the maximums of 2-3 ms have a periodical change?
Best wishes,
Jiajia Chu
11:13 a.m.
Hi, Paul and Ben
We also considerate GC influences the write latencies, but have no ideas about how to
demonstrate it by experiments.
Would you like to give us some advices about experiments design?
We also believe that it exists other factors which integrate to cause the abnormal
phenomenon.
Answers:
(1) "access" = write. We experiment read and write operations respectively,
but only find the strange phenomenon in the writing experiments.
The comparison of experiments can be seen in accessories.
(2) We use a NAND based SSD, Intel P3608.
(3) The result presented in accessories is produced with no delay.
We try to set "sleep(1)" between the two operations, but it seems does not
work.
(4) We did not update the firmware...
The device is shared by many fellows, we are afraid that the update is irreversible
and affects other works.
If it could be solved by firmware update, we also want to comprehend the reason?
Thanks a lot.
At 2017-08-02 07:49:13, "Walker, Benjamin" <benjamin.walker(a)intel.com>
wrote:
Hi Jiajia,
I have a bunch of questions that will help me figure out what you are seeing.
1) When you say "access", do you mean read or write? The behavior of these two
operations is quite different.
2) Are you using a NAND based or 3D XPoint based SSD? These again work entirely
differently.
3) When you access the same block repeatedly, what's the delay between each access?
None?
4) For whatever SSD you are using, have you confirmed the firmware is up to date? This can
make a big difference.
Thanks,
Ben
-------- Original message --------
From: "Luse, Paul E" <paul.e.luse(a)intel.com>
Date: 8/1/17 12:42 PM (GMT-07:00)
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] A issue about maximums of write latency when we access the same block
consecutively.
Sounds like Danielle’s theory still holds here… the impact of background tasks in SSD
firmware is at least partially dependent on the location of the operation on the media ,
ie if your write hits an area that is being relocated as part of GC it’s going to suffer
more and as GC is done in localized areas it stands to reason that random traffic will be
less affected as you’re hitting as a smaller % of the IOs will be hitting LBA regions that
GC is affecting. I’m sure there are a host of other factors as well
-Paul
From: SPDK [mailto:[email protected]] On Behalf Of ?
Sent: Tuesday, August 1, 2017 1:47 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] A issue about maximums of write latency when we access the same block
consecutively.
Hi, Danielle
We also did experiments to access random blocks of 500 times,
the results seem like more stable, most of the latencies are 10-30 μs and the maximum is
284 μs.
We have tested the random method many times and each experiment has similar results.
We wonder that why these same mechanisms, e.g., GC, journaling and checkpointing, cause
different influence between the random and consecutive accessing?
Thanks a lot.
At 2017-08-01 16:05:54, "Danielle Costantino" <dcostantino(a)vmem.com>
wrote:
What you are observing is the characteristics of how the drives firmware handles writes.
This can be from write amplification caused by the drives GC, and journaling +
checkpointing. This behavior may vary greatly between vendors and even firmware versions.
- Danielle Costantino
From: SPDK <spdk-bounces(a)lists.01.org> on behalf of 储 <cjj25233(a)163.com>
Sent: Tuesday, August 1, 2017 12:58:33 AM
To:[email protected]
Subject: [SPDK] A issue about maximums of write latency when we access the same block
consecutively.
Hi, all
Recently, we use a demo to obverse the latency.
We find that when we access the same block consecutively, the occurrence of maximum
latencies will become more frequent.
Additionally, they can reach even 2-3 ms and present a periodical change.
Why?
(1) For the same block, the latency of the first accessing is about 10-12 μs while the
second, third
and the forth accessing can reach 700-900 μs even 2-3 ms?
I want to know the reason why the operation difference between the first accessing
and the others exists.
(2) Why the maximums of 2-3 ms have a periodical change?
Best wishes,
Jiajia Chu
11:36 p.m.
On Wed, 2017-08-02 at 19:13 +0800, 储 wrote:
Answers:
(1) "access" = write. We experiment read and write operations respectively,
but only find the strange phenomenon in the writing experiments.
The comparison of experiments can be seen in accessories.
(2) We use a NAND based SSD, Intel P3608.
(3) The result presented in accessories is produced with no delay.
We try to set "sleep(1)" between the two operations, but it seems does
not work.
At 2017-08-02 07:49:13, "Walker, Benjamin" <benjamin.walker(a)intel.com>
wrote:
> Hi Jiajia,
>
> I have a bunch of questions that will help me figure out what you are
> seeing.
>
> 1) When you say "access", do you mean read or write? The behavior of
these
> two operations is quite different.
> 2) Are you using a NAND based or 3D XPoint based SSD? These again work
> entirely differently.
> 3) When you access the same block repeatedly, what's the delay between each
> access? None?
I was able to verify the behavior you are seeing. I'm afraid I'm not going to be
able to give you an exact answer for your particular device - I don't have
insight into the specifics of how each SSD is implemented. I brainstormed with a
few of my colleagues though, so what I can do is give you some idea of what is
happening inside of the device that will make it clear why writing to the same
block over and over may cause performance problems.
A good mental model for an SSD is basically a log of (LBA, data) pairs. When you
write to any LBA, it just appends to the end of the log and updates an internal
map of the location of that LBA. It does this appending by buffering several
writes into RAM located on the SSD, then it sends that batch of data to the NAND
all at once. The other important understanding is that the SSD is composed of a
large number of physical NAND dies, with some number of entirely parallel NAND
channels that can handle writes. Writing to the log sends the batched data to
each channel more or less round-robin. The final thing to remember is that this
whole process is implemented in hardware, not software, so adding things like
coordination between parallel operations is not as simple as just adding a lock.
When you write the same LBA over and over, a few things could happen inside the
SSD (I don't know how your SSD specifically works).
One possibility is that the SSD could see that the LBA is already buffered in
memory from a previous write and it could just update that memory. However, that
doesn't actually work in general. The data in that memory buffer may be
currently in use as part of a write to actual NAND, or may even be currently
being read. So the only option is to append to the end of the log for each new
write to the LBA. This could probably be coordinated with locking in software,
but remember that the SSD controller is implemented in hardware. If handling
this case makes the design far more complex, it may not be possible given power,
latency, and other budgets.
Another possibility is that the data is appended to the log for each write just
like any other I/O. However, it is still more complicated than the case where
random LBAs are being written to. Once one buffer is filled up, a write to NAND
is issued. When that write completes, it has to update the map for the location
of the LBA. If, while that write is outstanding, another buffer fills up with
new writes to the same LBA, the device has to figure out what to do. If it
submits the second NAND write to a new channel, it's then effectively racing
against the first write. If they complete out of order, the user will end up
with stale data. This case could also probably be handled by better coordination
on the completion side, but again there is a complexity trade off when
implementing this in actual hardware.
The easiest solution is probably to just detect if a NAND write is active for an
LBA in a given buffer, and then just queue up the next write until the one
before it finishes. That adds potentially a lot of latency, but it simplifies
the hardware design considerably.
Ultimately, I have no idea what that SSD is actually doing, but you can see that
it's fairly complex to handle this case. It is certainly more complex than
handling random I/O.
I hope that helps,
Ben
6:40 a.m.
Hi, Ben
We truly appreciate your concrete analysis and help in resolving the problem.
As you say, Different devices have their different and complex mechanisms.
We do not know how the hardware processes each request actually, but we think you make
sense.
Maybe it will be relieved with the hardware upgrade and development.
We hope that we can touch other devices and verify whether this problem exists in the
future.
We also expect other fellows using other devices could share the test results about this
issue.
Thanks a lot.
At 2017-08-04 07:36:07, "Walker, Benjamin" <benjamin.walker(a)intel.com>
wrote:
On Wed, 2017-08-02 at 19:13 +0800, 储 wrote:
> Answers:
> (1) "access" = write. We experiment read and write operations
respectively,
> but only find the strange phenomenon in the writing experiments.
> The comparison of experiments can be seen in accessories.
> (2) We use a NAND based SSD, Intel P3608.
> (3) The result presented in accessories is produced with no delay.
> We try to set "sleep(1)" between the two operations, but it seems
does
> not work.
>
> At 2017-08-02 07:49:13, "Walker, Benjamin"
<benjamin.walker(a)intel.com> wrote:
> > Hi Jiajia,
> >
> > I have a bunch of questions that will help me figure out what you are
> > seeing.
> >
> > 1) When you say "access", do you mean read or write? The behavior of
these
> > two operations is quite different.
> > 2) Are you using a NAND based or 3D XPoint based SSD? These again work
> > entirely differently.
> > 3) When you access the same block repeatedly, what's the delay between each
> > access? None?
I was able to verify the behavior you are seeing. I'm afraid I'm not going to be
able to give you an exact answer for your particular device - I don't have
insight into the specifics of how each SSD is implemented. I brainstormed with a
few of my colleagues though, so what I can do is give you some idea of what is
happening inside of the device that will make it clear why writing to the same
block over and over may cause performance problems.
A good mental model for an SSD is basically a log of (LBA, data) pairs. When you
write to any LBA, it just appends to the end of the log and updates an internal
map of the location of that LBA. It does this appending by buffering several
writes into RAM located on the SSD, then it sends that batch of data to the NAND
all at once. The other important understanding is that the SSD is composed of a
large number of physical NAND dies, with some number of entirely parallel NAND
channels that can handle writes. Writing to the log sends the batched data to
each channel more or less round-robin. The final thing to remember is that this
whole process is implemented in hardware, not software, so adding things like
coordination between parallel operations is not as simple as just adding a lock.
When you write the same LBA over and over, a few things could happen inside the
SSD (I don't know how your SSD specifically works).
One possibility is that the SSD could see that the LBA is already buffered in
memory from a previous write and it could just update that memory. However, that
doesn't actually work in general. The data in that memory buffer may be
currently in use as part of a write to actual NAND, or may even be currently
being read. So the only option is to append to the end of the log for each new
write to the LBA. This could probably be coordinated with locking in software,
but remember that the SSD controller is implemented in hardware. If handling
this case makes the design far more complex, it may not be possible given power,
latency, and other budgets.
Another possibility is that the data is appended to the log for each write just
like any other I/O. However, it is still more complicated than the case where
random LBAs are being written to. Once one buffer is filled up, a write to NAND
is issued. When that write completes, it has to update the map for the location
of the LBA. If, while that write is outstanding, another buffer fills up with
new writes to the same LBA, the device has to figure out what to do. If it
submits the second NAND write to a new channel, it's then effectively racing
against the first write. If they complete out of order, the user will end up
with stale data. This case could also probably be handled by better coordination
on the completion side, but again there is a complexity trade off when
implementing this in actual hardware.
The easiest solution is probably to just detect if a NAND write is active for an
LBA in a given buffer, and then just queue up the next write until the one
before it finishes. That adds potentially a lot of latency, but it simplifies
the hardware design considerably.
Ultimately, I have no idea what that SSD is actually doing, but you can see that
it's fairly complex to handle this case. It is certainly more complex than
handling random I/O.
I hope that helps,
Ben
5:34 a.m.
Hi, Jiajia,
Here are some of my suggestion:
1. try Secure Erase before the test. It will eliminate most of the FTL
background operations, and put SSD into so-called FOB state.
2. most of the enterprise-grade SSD provides Quality of Service in
speications. For example, P3608 ensures 99.99% 4K write IO complete in 4.7
ms when QD=1. Spec also provides another write latency value like 20us. You
can consider them as worst case and best case respectively. But the
distribution between them is hard to predict in non-FOB state, due to the
diversity of firmware, host working load pattern, and NAND Flash
characteristics.
3. mapping granularity is 4K in most of the SSD firmware design. So, if
LBA size is 512B in your test, it will make scenario more complex. Try to
test with 4K aligned IO.
4.
http://codecapsule.com/2014/02/12/coding-for-ssds-part-1-introduction-and...
Thanks,
-Crane Chu
On Mon, Aug 7, 2017 at 2:40 PM, 储 <cjj25233(a)163.com> wrote:
Hi, Ben
We truly appreciate your concrete analysis and help in resolving the
problem.
As you say, Different devices have their different and complex mechanisms.
We do not know how the hardware processes each request actually, but we
think you make sense.
Maybe it will be relieved with the hardware upgrade and development.
We hope that we can touch other devices and verify whether this problem
exists in the future.
We also expect other fellows using other devices could share the test
results about this issue.
Thanks a lot.
At 2017-08-04 07:36:07, "Walker, Benjamin" <benjamin.walker(a)intel.com>
wrote:
>On Wed, 2017-08-02 at 19:13 +0800, 储 wrote:
>> Answers:
>> (1) "access" = write. We experiment read and write operations
respectively,
>> but only find the strange phenomenon in the writing experiments.
>> The comparison of experiments can be seen in accessories.
>> (2) We use a NAND based SSD, Intel P3608.
>> (3) The result presented in accessories is produced with no delay.
>> We try to set "sleep(1)" between the two operations, but it
seems does
>> not work.
>>
>> At 2017-08-02 07:49:13, "Walker, Benjamin"
<benjamin.walker(a)intel.com> wrote:
>> > Hi Jiajia,
>> >
>> > I have a bunch of questions that will help me figure out what you are
>> > seeing.
>> >
>> > 1) When you say "access", do you mean read or write? The behavior
of these
>> > two operations is quite different.
>> > 2) Are you using a NAND based or 3D XPoint based SSD? These again work
>> > entirely differently.
>> > 3) When you access the same block repeatedly, what's the delay between
each
>> > access? None?
>
>I was able to verify the behavior you are seeing. I'm afraid I'm not going to
be
>able to give you an exact answer for your particular device - I don't have
>insight into the specifics of how each SSD is implemented. I brainstormed with a
>few of my colleagues though, so what I can do is give you some idea of what is
>happening inside of the device that will make it clear why writing to the same
>block over and over may cause performance problems.
>
>A good mental model for an SSD is basically a log of (LBA, data) pairs. When you
>write to any LBA, it just appends to the end of the log and updates an internal
>map of the location of that LBA. It does this appending by buffering several
>writes into RAM located on the SSD, then it sends that batch of data to the NAND
>all at once. The other important understanding is that the SSD is composed of a
>large number of physical NAND dies, with some number of entirely parallel NAND
>channels that can handle writes. Writing to the log sends the batched data to
>each channel more or less round-robin. The final thing to remember is that this
>whole process is implemented in hardware, not software, so adding things like
>coordination between parallel operations is not as simple as just adding a lock.
>
>When you write the same LBA over and over, a few things could happen inside the
>SSD (I don't know how your SSD specifically works).
>
>One possibility is that the SSD could see that the LBA is already buffered in
>memory from a previous write and it could just update that memory. However, that
>doesn't actually work in general. The data in that memory buffer may be
>currently in use as part of a write to actual NAND, or may even be currently
>being read. So the only option is to append to the end of the log for each new
>write to the LBA. This could probably be coordinated with locking in software,
>but remember that the SSD controller is implemented in hardware. If handling
>this case makes the design far more complex, it may not be possible given power,
>latency, and other budgets.
>
>Another possibility is that the data is appended to the log for each write just
>like any other I/O. However, it is still more complicated than the case where
>random LBAs are being written to. Once one buffer is filled up, a write to NAND
>is issued. When that write completes, it has to update the map for the location
>of the LBA. If, while that write is outstanding, another buffer fills up with
>new writes to the same LBA, the device has to figure out what to do. If it
>submits the second NAND write to a new channel, it's then effectively racing
>against the first write. If they complete out of order, the user will end up
>with stale data. This case could also probably be handled by better coordination
> on the completion side, but again there is a complexity trade off when
>implementing this in actual hardware.
>
>The easiest solution is probably to just detect if a NAND write is active for an
>LBA in a given buffer, and then just queue up the next write until the one
>before it finishes. That adds potentially a lot of latency, but it simplifies
>the hardware design considerably.
>
>Ultimately, I have no idea what that SSD is actually doing, but you can see that
>it's fairly complex to handle this case. It is certainly more complex than
>handling random I/O.
>
>I hope that helps,
>Ben
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk
1:27 p.m.
Hi, Crane,
Thanks for your enlightening suggestions,
1. we are afraid that we can not do secure erasing currently...The device is shared and we
need coordinating with other fellows before erasing.
2. we did not considerate the influence of the device state deeply, and we think you make
sense.
We will provide a new test report for the FOB state to verify your suggestion.
3. our test demo is based on the spdk/examples/nvme/hello_world/hello_world.c，and the
granularity of I/O is set as 4K.
4. thanks for your knowledge sharing^_^
Best wishes,
Jiajia Chu
At 2017-08-09 13:34:17, "Crane Chu" <cranechu(a)gmail.com> wrote:
Hi, Jiajia,
Here are some of my suggestion:
try Secure Erase before the test. It will eliminate most of the FTL background operations,
and put SSD into so-called FOB state.
most of the enterprise-grade SSD provides Quality of Service in speications. For example,
P3608 ensures 99.99% 4K write IO complete in 4.7 ms when QD=1. Spec also provides another
write latency value like 20us. You can consider them as worst case and best case
respectively. But the distribution between them is hard to predict in non-FOB state, due
to the diversity of firmware, host working load pattern, and NAND Flash characteristics.
mapping granularity is 4K in most of the SSD firmware design. So, if LBA size is 512B in
your test, it will make scenario more complex. Try to test with 4K aligned IO.
http://codecapsule.com/2014/02/12/coding-for-ssds-part-1-introduction-and...
Thanks,
-Crane Chu
On Mon, Aug 7, 2017 at 2:40 PM, 储 <cjj25233(a)163.com> wrote:
Hi, Ben
We truly appreciate your concrete analysis and help in resolving the problem.
As you say, Different devices have their different and complex mechanisms.
We do not know how the hardware processes each request actually, but we think you make
sense.
Maybe it will be relieved with the hardware upgrade and development.
We hope that we can touch other devices and verify whether this problem exists in the
future.
We also expect other fellows using other devices could share the test results about this
issue.
Thanks a lot.
At 2017-08-04 07:36:07, "Walker, Benjamin" <benjamin.walker(a)intel.com>
wrote:
On Wed, 2017-08-02 at 19:13 +0800, 储 wrote:
> Answers:
> (1) "access" = write. We experiment read and write operations
respectively,
> but only find the strange phenomenon in the writing experiments.
> The comparison of experiments can be seen in accessories.
> (2) We use a NAND based SSD, Intel P3608.
> (3) The result presented in accessories is produced with no delay.
> We try to set "sleep(1)" between the two operations, but it seems
does
> not work.
>
> At 2017-08-02 07:49:13, "Walker, Benjamin"
<benjamin.walker(a)intel.com> wrote:
> > Hi Jiajia,
> >
> > I have a bunch of questions that will help me figure out what you are
> > seeing.
> >
> > 1) When you say "access", do you mean read or write? The behavior of
these
> > two operations is quite different.
> > 2) Are you using a NAND based or 3D XPoint based SSD? These again work
> > entirely differently.
> > 3) When you access the same block repeatedly, what's the delay between each
> > access? None?
I was able to verify the behavior you are seeing. I'm afraid I'm not going to be
able to give you an exact answer for your particular device - I don't have
insight into the specifics of how each SSD is implemented. I brainstormed with a
few of my colleagues though, so what I can do is give you some idea of what is
happening inside of the device that will make it clear why writing to the same
block over and over may cause performance problems.
A good mental model for an SSD is basically a log of (LBA, data) pairs. When you
write to any LBA, it just appends to the end of the log and updates an internal
map of the location of that LBA. It does this appending by buffering several
writes into RAM located on the SSD, then it sends that batch of data to the NAND
all at once. The other important understanding is that the SSD is composed of a
large number of physical NAND dies, with some number of entirely parallel NAND
channels that can handle writes. Writing to the log sends the batched data to
each channel more or less round-robin. The final thing to remember is that this
whole process is implemented in hardware, not software, so adding things like
coordination between parallel operations is not as simple as just adding a lock.
When you write the same LBA over and over, a few things could happen inside the
SSD (I don't know how your SSD specifically works).
One possibility is that the SSD could see that the LBA is already buffered in
memory from a previous write and it could just update that memory. However, that
doesn't actually work in general. The data in that memory buffer may be
currently in use as part of a write to actual NAND, or may even be currently
being read. So the only option is to append to the end of the log for each new
write to the LBA. This could probably be coordinated with locking in software,
but remember that the SSD controller is implemented in hardware. If handling
this case makes the design far more complex, it may not be possible given power,
latency, and other budgets.
Another possibility is that the data is appended to the log for each write just
like any other I/O. However, it is still more complicated than the case where
random LBAs are being written to. Once one buffer is filled up, a write to NAND
is issued. When that write completes, it has to update the map for the location
of the LBA. If, while that write is outstanding, another buffer fills up with
new writes to the same LBA, the device has to figure out what to do. If it
submits the second NAND write to a new channel, it's then effectively racing
against the first write. If they complete out of order, the user will end up
with stale data. This case could also probably be handled by better coordination
on the completion side, but again there is a complexity trade off when
implementing this in actual hardware.
The easiest solution is probably to just detect if a NAND write is active for an
LBA in a given buffer, and then just queue up the next write until the one
before it finishes. That adds potentially a lot of latency, but it simplifies
the hardware design considerably.
Ultimately, I have no idea what that SSD is actually doing, but you can see that
it's fairly complex to handle this case. It is certainly more complex than
handling random I/O.
I hope that helps,
Ben
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk
1830
days inactive
1840
days old
9 comments
5 participants
participants (5)
-
Crane Chu -
Danielle Costantino -
Luse, Paul E -
Walker, Benjamin -
储